Develop Reference

How to create a test for DeadLetter Kafka

In my little microservice, I created a Producer Kafka to send the messages with errors (messages having errors in the JSON format) inside the DeadLetter in this way :
#Component
public class KafkaProducer {
#Autowired
private KafkaTemplate<String, String> kafkaTemplate;
public void sendDeadLetter(String message) {
kafkaTemplate.send("DeadLetter", message);
}
}
I would like to create a JUnitTest for the completeness of the project, but I have no idea how to create the eventuality of a possible JSON error in order to create the test. I thank everyone for any possible help and advice

To create a JUnitTest consistent with your code. I should recreate the case where you pass it a warped or invalid JSON. In your case, I would opt to configure a MockConsumer from which to read any message that the logic of your code will be invited to the dead letter.
To have a usable test structure, I recommend something like this:
#KafkaListener(topics = "yourTopic")
public void listen(String message) {
messages.add(message);
}
For testing a basic structure could be
#Test
public void testDeadLetter(){
//Set up a mockConsumer
MockConsumer<String,String> yourMockConsumer = new MockConsumer<String,String> (OffsetResetStrategy.EARLIEST);
yourMockConsumer.subscribe(Collections.singletonList("yourTopic"));
//Sending message on embedded Kafka broker
String error = "ERRORE";
kafkaTemplate.send("yourTopic", error);
//Reading the message may take a second
Thread.sleep(1000);
//Create an Assert that checks you that the message is equal to the error specified above
}
I hope it will be useful to you!

You can create Kafka topic using testcontainers and write your tests on top of that.
Sharing an example on how to use testcontainers https://github.com/0001vrn/testcontainers-example

Spring dataflow Processor to process one payload and write multiple rows in data table

I would like to ask a simple question,
I have implemented a processor, which process one payload and return an array of entities, such like:
#EnableBinding(Processor.class)
public class SimpleProcessor {
...
public SimpleProcessor () {
...
}
#Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)
public OutgoingEntity[] processData(IncomingEntity payload) {
// business logic here
return outgoingEntity;
}
I have my stream in SCDF and middleware as kafka like this:
some source | SimpleProcessor | JDBC sink
to validate the messages, I have replaced log sink to relpace JDBC sink and it logs arrays of json. When I use JDBC sink, it throws Exception and says JDBC sink can not access the properties in the object, which makes sense, that it is array of objects instead object...
My question is:
Can I use modify my processor, so it can process a payload once and give out message multiple time, like this,
#Transformer(inputChannel = Processor.INPUT)
public void processData(IncomingEntity payload) {
...
for(OutGoingEntity o: OutgoingEntity[]){
outputMethode();
}
}
#Transformer(outputChannel = Processor.OUTPUT)
private OutGoingEntity outputMethode() {
.....
return outGoingEntity;
}
So it can pass multiple json object to jdbc sink and write in the datatable.
Can I use JDBC sink to deal with arrays? How?
Can I use some other processors or sink to finish this task?

Like Matthias J. Sax suggested in comment, I have used flatMapValue method of KStream to deal with array from input. I put this scdf processor after the one forwards array. That works fine.
#EnableBinding(KafkaStreamsProcessor.class)
public class ArrayProcessor {
#StreamListener("input") #SendTo("output")
public KStream<?, String> process(KStream<?, String> payload) {
return payload.flatMapValues( //impl )
...} }

Is there a way to stream a log to a Spring Boot application?

I have a Spring Boot app that is used as an event logger. Each client sends different events via a REST api, which are then saved in a database. But apart from simple events, I need the clients to also send their execution logs to Spring Boot.
Now, uploading a log after a client finishes executing is easy, and there are plenty examples for it out there. What I need is to stream the log as the client is executing, line by line, and not wait until the client has finished.
I've spent quite some time googling for a possible answer and I couldn't find anything that fits my needs. Any advice how to do this using Spring Boot (future releases included)? Is it feasible?

I see a couple of possibilities here. First, consider using a logback (the default Spring Boot logging implementation) SocketAppender or ServerSocketAppender in your client. See: https://logback.qos.ch/manual/appenders.html. This would let you send log messages to any logging service.
But I might suggest that you not log to your Spring Boot Event App as I suspect that will add complexity to your app unnecessarily, and I can see a situation where there is some bug in the Event App that then causes clients to log a bunch of errors which in turn all go back to the event app making it difficult to determine the initial error.
What I would respectfully suggest is that you instead log to a logging server - logstash: https://www.elastic.co/products/logstash for example, or if you already have a db that you are saving the event to, then maybe use the logbook DBAppender and write the logs directly to a db.

I wrote here an example on how to stream file updates in a spring boot endpoint. The only difference is that the code uses the Java WatchService API to trigger file updates on a given file.
However, in your situation, I would also choose the log appender to directly send messages to the connected clients (with sse - call template.broadcast from there) instead of watching for changes like I described.
The endpoint:
#GetMapping(path = "/logs", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public SseEmitter streamSseMvc() {
return sseService.newSseEmitter();
}
The service:
public class LogsSseService {
private static final Logger log = LoggerFactory.getLogger(LogsSseService.class);
private static final String TOPIC = "logs";
private final SseTemplate template;
private static final AtomicLong COUNTER = new AtomicLong(0);
public LogsSseService(SseTemplate template, MonitoringFileService monitoringFileService) {
this.template = template;
monitoringFileService.listen(file -> {
try {
Files.lines(file)
.skip(COUNTER.get())
.forEach(line ->
template.broadcast(TOPIC, SseEmitter.event()
.id(String.valueOf(COUNTER.incrementAndGet()))
.data(line)));
} catch (IOException e) {
e.printStackTrace();
}
});
}
public SseEmitter newSseEmitter() {
return template.newSseEmitter(TOPIC);
}
}
The custom appender (which you have to add to your logger - check here):
public class StreamAppender extends UnsynchronizedAppenderBase<ILoggingEvent> implements SmartLifecycle {
public static final String TOPIC = "logs";
private final SseTemplate template;
public StreamAppender(SseTemplate template) {
this.template = template;
}
#Override
protected void append(ILoggingEvent event) {
template.broadcast(TOPIC, SseEmitter.event()
.id(event.getThreadName())
.name("log")
.data(event.getFormattedMessage()));
}
#Override
public boolean isRunning() {
return isStarted();
}
}

Custom messages using RabbitMQ/ActiveMQ?

I am coming from Sidekiq and am now moving to a Java solution for distributed jobs. I came across RabbitMQ and ActiveMQ, but it seems those brokers use plaintext or raw byte[] messages. I was wondering if it's possible to send custom messages using these frameworks?
Ideally I would just define a Java class for each specific message type and use that in both worker and producer. Is such a thing possible? Or should I look at other types of middleware?
MyOwnMessageFormat message = new MyOwnMessageFormat(content)
channel.send(message)
Message message = channel.receive()
if (message.class == MyOwnMessageFormat)
{
doSomething();
}

Exchanging messages through message brokers in Java is much easier when done through a service bus like Camel. You dont loose the flexibility of configuring your broker endpoint and still your code is isolated from the particular transport used or the message format. E.g. you can deploy ActiveMQ and later switch to RabbitMQ without having to update your code - just the service bus configuration. Or you can switch from plain Java serialization to JSON when sending messages to the broker by adding a message transformer. Again your business layer does not have to be modified.
Here's a sample that uses POJO producing/consuming where the producer calls a regular Java interface and the consumer implements the interface. The sample assumes that the sender/receiver are instantiated with Spring in order for the Camel endpoints to be injected
Message sender:
interface MyService {
MyResult addTask(MyTask task);
}
class Sender {
#Produce(uri="activemq:queue:myservice")
MyService service;
public void run() {
MyTask task = new MyTask();
MyResult result = service.addTask(task);
}
Message receiver:
class Receiver {
#Consume(uri="activemq:queue:myservice")
public MyResult addTask(MyTask task) {
return new MyResult();
}
}
MyTask & MyResult need to be serializable.
I think learning the Camel framework is not very hard while it can be very rewarding.

In JMS,as long as your custom message object is Serializable, you can use an ObjectMessage,Stream Message or Map Message according to your requirement like below to send(Object Message):
MessageProducer producer = session.createProducer( destination );
ObjectMessage message = session.createObjectMessage( getMyObject() );
producer.send( message );
For receiving:
Message message = consumer.receive();
if (message instanceof ObjectMessage) {
Object object = ((ObjectMessage) message).getObject();
Hope this helps you!

Yes, Before publishing the message to RabbitMQ, convert your message to a Json String.
Now on the receiving end, when the message is received, Parse it to convert into the same format.
In ruby this can be done with
while sending => Message.to_json
on Receiving => message = JSON.parse(received_msg)
To get the class of data, you can send a variable specifying the class of the data along with the data.

This is what worked for me with the latest version of ActiveMQ
...
MessageConsumer consumer=session.createConsumer(destination);
while(true) {
javax.jms.Message message=consumer.receive();
ActiveMQObjectMessage queueMessage=(ActiveMQObjectMessage)message;
Object payload=queueMessage.getObject();
if(payload instanceof NotificationMessage) {
this.sendMessage((NotificationMessage)payload);
}
}
The NotificationMessage object extends Serializable and looks like this
public static class NotificationMessage implements Serializable {
/**
*
*/
private static final long serialVersionUID = 1631373969001850200L;
public String to;
public String data;
}

How to mock a javax.mail.Session

i need to mock a javax.mail.Session object in my unit tests.
The class javax.mail.Session is marked final so Mockito is not able to create a mock. Does anyone have an idea how to fix this?
Edit:
My test is an Arquillian test and has already an annotation #RunWith(Arquillian.class). Therefore powermock is not an option.

You may refactor your code a little bit. In the "Working with Legacy Code" book by Martin Fowler, he describes a technique of separating the external API (think Java Mail API) from your own application code. The technique is called "Wrap and Skin" and is pretty simple.
What you should do is simply:
Create an interface MySession (which is your own stuff) by extracting methods from the javax.mail.Session class.
Implement that interface by creating a concrete class (looking a bit like the original javax.mail.Session)
In each method DELEGATE the call to equivalent javax.mail.Session method
Create your mock class which implements MySession :-)
Update your production code to use MySession instead of javax.mail.Session
Happy testing!
EDIT: Also take a look at this blog post: http://www.mhaller.de/archives/18-How-to-mock-a-thirdparty-final-class.html

Use PowerMockito to mock it.
#RunWith(PowerMockRunner.class)
// We prepare PartialMockClass for test because it's final or we need to mock private or static methods
#PrepareForTest(javax.mail.Session.class)
public class YourTestCase {
#Test
public void test() throws Exception {
PowerMockito.doReturn(value).when(classUnderTest, "methodToMock", "parameter1");
}
}

You can use the Mock JavaMail project. I first found it from Kohsuke Kawaguchi. Alan Franzoni also has a primer for it on his blog.
When you put this jar file in your classpath, it substitutes any sending of mail to in memory mailboxes that can be checked immediately. It's super easy to use.
Adding this to your classpath is admittedly a pretty heavy handed way to mock something, but you rarely want to send real emails in your automated tests anyway.

If you want to mock a final classes you can use the JDave unfinalizer which can be found here : http://jdave.org/documentation.html#mocking
It uses CGLib to alter the bytecode dynamically when the JVM is loaded to transform the class as a non final class.
This library can then be used with JMock2 ( http://www.jmock.org/mocking-classes.html ) to make your tests because as far as I know, Mockito is not compatible with JDave.

Use Java 8 Functions!
public class SendEmailGood {
private final Supplier<Message> messageSupplier;
private final Consumer<Message> messageSender;
public SendEmailGood(Supplier<Message> messageSupplier,
Consumer<Message> messageSender) {
this.messageSupplier = messageSupplier;
this.messageSender = messageSender;
}
public void send(String[] addresses, String from,
String subject, String body)
throws MessagingException {
Message message = messageSupplier.get();
for (String address : addresses) {
message.addRecipient
(Message.RecipientType.TO, new InternetAddress(address));
}
message.addFrom(new InternetAddress[]{new InternetAddress(from)});
message.setSubject(subject);
message.setText(body);
messageSender.accept(message);
}
}
Then your test code will look something like the following:
#Test
public void sendBasicEmail() throws MessagingException {
final boolean[] messageCalled = {false};
Consumer<Message> consumer = message -> {
messageCalled[0] = true;
};
Message message = mock(Message.class);
Supplier<Message> supplier = () -> message;
SendEmailGood sendEmailGood = new SendEmailGood(supplier, consumer);
String[] addresses = new String[2];
addresses[0] = "foo#foo.com";
addresses[1] = "boo#boo.com";
String from = "baz#baz.com";
String subject = "Test Email";
String body = "This is a sample email from us!";
sendEmailGood.send(addresses, from, subject, body);
verify(message).addRecipient(Message.RecipientType.TO, new InternetAddress("foo#foo.com"));
verify(message).addRecipient(Message.RecipientType.TO, new InternetAddress("boo#boo.com"));
verify(message).addFrom(new InternetAddress[]{new InternetAddress("baz#baz.com")});
verify(message).setSubject(subject);
verify(message).setText(body);
assertThat(messageCalled[0]).isTrue();
}
To create an integration test, plugin the real Session, and Transport.
Consumer<Message> consumer = message -> {
try {
Transport.send(message);
} catch (MessagingException e) {
e.printStackTrace();
}
};
Supplier<Message> supplier = () -> {
Properties properties = new Properties();
return new MimeMessage(Session.getDefaultInstance(properties));
};

See the PowerMock docs for running under JUnit 3, since it did not have runners, or use a byte-code manipulation tool.

If you can introduce Spring into your project you can use the JavaMailSender and mock that. I don't know how complicated your requirements are.
import org.springframework.mail.SimpleMailMessage;
import org.springframework.mail.javamail.JavaMailSender;
#Test
public void createAndSendBookChangesMail() {
// Your custom service layer object to test
MailServiceImpl service = new MailServiceImpl();
// MOCK BEHAVIOUR
JavaMailSender mailSender = mock(JavaMailSender.class);
service.setMailSender(mailSender);
// PERFORM TEST
service.createAndSendMyMail("some mail message content");
// ASSERT
verify(mailSender).send(any(SimpleMailMessage.class));
}

This is a pretty old question, but you could always implement your own Transport using the JavaMail API. With your own transport, you could just configure it according to this documentation. One benefit of this approach is you could do anything you want with these messages. Perhaps you would store them in a hash/set that you could then ensure they actually got sent in your unit tests. This way, you don't have to mock the final object, you just implement your own.

I use the mock-javamail library. It just replace the origin javamail implementation in the classpath. You can send mails normally, it just sends to in-memory MailBox, not a real mailbox.
Finally, you can use the MailBox object to assert anything you want to check.