Develop Reference

How to create a test for DeadLetter Kafka

In my little microservice, I created a Producer Kafka to send the messages with errors (messages having errors in the JSON format) inside the DeadLetter in this way :
#Component
public class KafkaProducer {
#Autowired
private KafkaTemplate<String, String> kafkaTemplate;
public void sendDeadLetter(String message) {
kafkaTemplate.send("DeadLetter", message);
}
}
I would like to create a JUnitTest for the completeness of the project, but I have no idea how to create the eventuality of a possible JSON error in order to create the test. I thank everyone for any possible help and advice

To create a JUnitTest consistent with your code. I should recreate the case where you pass it a warped or invalid JSON. In your case, I would opt to configure a MockConsumer from which to read any message that the logic of your code will be invited to the dead letter.
To have a usable test structure, I recommend something like this:
#KafkaListener(topics = "yourTopic")
public void listen(String message) {
messages.add(message);
}
For testing a basic structure could be
#Test
public void testDeadLetter(){
//Set up a mockConsumer
MockConsumer<String,String> yourMockConsumer = new MockConsumer<String,String> (OffsetResetStrategy.EARLIEST);
yourMockConsumer.subscribe(Collections.singletonList("yourTopic"));
//Sending message on embedded Kafka broker
String error = "ERRORE";
kafkaTemplate.send("yourTopic", error);
//Reading the message may take a second
Thread.sleep(1000);
//Create an Assert that checks you that the message is equal to the error specified above
}
I hope it will be useful to you!

You can create Kafka topic using testcontainers and write your tests on top of that.
Sharing an example on how to use testcontainers https://github.com/0001vrn/testcontainers-example

Send message in specific partition of Topic

Is there any way producer could send messages to specific partiton of topic in broker ?
As of now, am able to send topic having 2 partitions, but dont have control to send in specific partition.
Component
#EnableBinding(Source.class)
public class RsvpsKafkaProducer {
private static final int SENDING_MESSAGE_TIMEOUT_MS = 10000;
private final Source source;
public RsvpsKafkaProducer(Source source) {
this.source = source;
}
public void sendRsvpMessage(WebSocketMessage<?> message) {
System.out.println("sendRsvpMessage");
source.output()
.send(MessageBuilder.withPayload(message.getPayload())
.build(),
SENDING_MESSAGE_TIMEOUT_MS);
}
}
application.properties
spring.cloud.stream.kafka.binder.zkNodes=localhost:2181
spring.cloud.stream.kafka.binder.brokers=localhost:9093
spring.cloud.stream.bindings.output.destination=meetupTopic
spring.cloud.stream.bindings.output.producer.partitionCount=2
spring.cloud.stream.bindings.output.content-type=text/plain
spring.cloud.stream.bindings.output.producer.headerMode=raw
Is there any way I could achieve it using spring cloud stream ? I want some messages to go in P1 partition and some to P2 partition within meetupTopic.

MessageBuilder.withPayload(message.getPayload())
.setHeader(KafkaHeaders.PARTITION_ID, 23)
.build()

I haven't tried this but from the docs, it looks like it might work.
spring.cloud.stream.bindings.output.producer.partitionSelectorExpression=headers['partitionKey']
And then you add the header partitionKey when you send the message.

Testing listener with #EmbeddedKafka from spring-kafka-test

I'm trying to test listener in springboot created using #KafkaListener
But listener always listens on localhost:9092 instead of using this embededKafka
My listener looks like this:
#Component
#Slf4j
class SomeListener {
private final List<String> receivedMessages = new ArrayList<>();
#KafkaListener(topics = "some-ultra-cool-topic")
public void onKafkaMessage(String theMessage) {
log.info("Message received {}", theMessage);
receivedMessages.add(theMessage);
}
Collection<String> getAll() {
return unmodifiableCollection(receivedMessages);
}
}
And spock test like this:
#SpringBootTest
#EmbeddedKafka
#TestPropertySource(properties = ['spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}', 'spring.kafka.consumer.auto-offset-reset=earliest'])
class SomeListenerTest extends Specification {
#Autowired
EmbeddedKafkaBroker embeddedKafkaBroker
#Autowired
SomeListener someListener
void 'should receive message'() {
given:
def sender = new KafkaTemplate<>(new DefaultKafkaProducerFactory<String, String>(KafkaTestUtils.producerProps(embeddedKafkaBroker)))
when:
sender.send('some-ultra-cool-topic', 'first message content')
then:
someListener.all.size() == 1
}
}
My application.yaml doesn't have bootstraps servers configured - so it is purly default from spring-boot.
I can see in logs that producer is sending message to broker (it starts every time on different random port).
But listener always try to connect to broker on localhost:9092
How can I configure it to use this embedded one?

Thanks #sawim for tips
Actual problem was in test. I ended up doing this test with lib org.awaitility:awaitility
then:
waitAtMost(5, SECONDS)
.untilAsserted({ ->
assertThat(personFacade.findAll(), hasSize(1))
})
Configuration from first example works, however during startup I can see kafka-logs trying to connect to localhost:9200 - seems we can Ignore it

How to implement FlinkKafkaProducer serializer for Kafka 2.2

I've been working on updating a Flink processor (Flink version 1.9) that reads from Kafka and then writes to Kafka. We have written this processor to run towards a Kafka 0.10.2 cluster and now we have deployed a new Kafka cluster running version 2.2. Therefore I set out to update the processor to use the latest FlinkKafkaConsumer and FlinkKafkaProducer (as suggested by the Flink docs). However I've run into some problems with the Kafka producer. I'm unable to get it to Serialize data using deprecated constructors (not surprising) and I've been unable to find any implementations or examples online about how to implement a Serializer (all the examples are using older Kafka Connectors)
The current implementation (for Kafka 0.10.2) is as follows
FlinkKafkaProducer010<String> eventBatchFlinkKafkaProducer = new FlinkKafkaProducer010<String>(
"playerSessions",
new SimpleStringSchema(),
producerProps,
(FlinkKafkaPartitioner) null
);
When trying to implement the following FlinkKafkaProducer
FlinkKafkaProducer<String> eventBatchFlinkKafkaProducer = new FlinkKafkaProducer<String>(
"playerSessions",
new SimpleStringSchema(),
producerProps,
null
);
I get the following error:
Exception in thread "main" java.lang.NullPointerException
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:525)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:483)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:357)
at com.ebs.flink.sessionprocessor.SessionProcessor.main(SessionProcessor.java:122)
and I haven't been able to figure out why.
The constructor for FlinkKafkaProducer is also deprecated and when I try implementing the non-deprecated constructor I can't figure out how to serialize the data.
The following is how it would look:
FlinkKafkaProducer<String> eventBatchFlinkKafkaProducer = new FlinkKafkaProducer<String>(
"playerSessions",
new KafkaSerializationSchema<String>() {
#Override
public ProducerRecord<byte[], byte[]> serialize(String s, #Nullable Long aLong) {
return null;
}
},
producerProps,
FlinkKafkaProducer.Semantic.EXACTLY_ONCE
);
But I don't understand how to implement the KafkaSerializationSchema and I find no examples of this online or in the Flink docs.
Does anyone have any experience implementing this or any tips on why the FlinkProducer gets NullPointerException in the step?

If you are just sending String to Kafka:
public class ProducerStringSerializationSchema implements KafkaSerializationSchema<String>{
private String topic;
public ProducerStringSerializationSchema(String topic) {
super();
this.topic = topic;
}
#Override
public ProducerRecord<byte[], byte[]> serialize(String element, Long timestamp) {
return new ProducerRecord<byte[], byte[]>(topic, element.getBytes(StandardCharsets.UTF_8));
}
}
For sending a Java Object:
import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.flink.streaming.connectors.kafka.KafkaSerializationSchema;
import org.apache.kafka.clients.producer.ProducerRecord;
public class ObjSerializationSchema implements KafkaSerializationSchema<MyPojo>{
private String topic;
private ObjectMapper mapper;
public ObjSerializationSchema(String topic) {
super();
this.topic = topic;
}
#Override
public ProducerRecord<byte[], byte[]> serialize(MyPojo obj, Long timestamp) {
byte[] b = null;
if (mapper == null) {
mapper = new ObjectMapper();
}
try {
b= mapper.writeValueAsBytes(obj);
} catch (JsonProcessingException e) {
// TODO
}
return new ProducerRecord<byte[], byte[]>(topic, b);
}
}
In your code
.addSink(new FlinkKafkaProducer<>(producerTopic, new ObjSerializationSchema(producerTopic),
params.getProperties(), FlinkKafkaProducer.Semantic.EXACTLY_ONCE));

To the deal with the timeout in the case of FlinkKafkaProducer.Semantic.EXACTLY_ONCE you should read https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-011-and-newer, particularly this part:
Semantic.EXACTLY_ONCE mode relies on the ability to commit transactions that were started before taking a checkpoint, after recovering from the said checkpoint. If the time between Flink application crash and completed restart is larger than Kafka’s transaction timeout there will be data loss (Kafka will automatically abort transactions that exceeded timeout time). Having this in mind, please configure your transaction timeout appropriately to your expected down times.
Kafka brokers by default have transaction.max.timeout.ms set to 15 minutes. This property will not allow to set transaction timeouts for the producers larger than it’s value. FlinkKafkaProducer011 by default sets the transaction.timeout.ms property in producer config to 1 hour, thus transaction.max.timeout.ms should be increased before using the Semantic.EXACTLY_ONCE mode.

Is there a way to stream a log to a Spring Boot application?

I have a Spring Boot app that is used as an event logger. Each client sends different events via a REST api, which are then saved in a database. But apart from simple events, I need the clients to also send their execution logs to Spring Boot.
Now, uploading a log after a client finishes executing is easy, and there are plenty examples for it out there. What I need is to stream the log as the client is executing, line by line, and not wait until the client has finished.
I've spent quite some time googling for a possible answer and I couldn't find anything that fits my needs. Any advice how to do this using Spring Boot (future releases included)? Is it feasible?

I see a couple of possibilities here. First, consider using a logback (the default Spring Boot logging implementation) SocketAppender or ServerSocketAppender in your client. See: https://logback.qos.ch/manual/appenders.html. This would let you send log messages to any logging service.
But I might suggest that you not log to your Spring Boot Event App as I suspect that will add complexity to your app unnecessarily, and I can see a situation where there is some bug in the Event App that then causes clients to log a bunch of errors which in turn all go back to the event app making it difficult to determine the initial error.
What I would respectfully suggest is that you instead log to a logging server - logstash: https://www.elastic.co/products/logstash for example, or if you already have a db that you are saving the event to, then maybe use the logbook DBAppender and write the logs directly to a db.

I wrote here an example on how to stream file updates in a spring boot endpoint. The only difference is that the code uses the Java WatchService API to trigger file updates on a given file.
However, in your situation, I would also choose the log appender to directly send messages to the connected clients (with sse - call template.broadcast from there) instead of watching for changes like I described.
The endpoint:
#GetMapping(path = "/logs", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public SseEmitter streamSseMvc() {
return sseService.newSseEmitter();
}
The service:
public class LogsSseService {
private static final Logger log = LoggerFactory.getLogger(LogsSseService.class);
private static final String TOPIC = "logs";
private final SseTemplate template;
private static final AtomicLong COUNTER = new AtomicLong(0);
public LogsSseService(SseTemplate template, MonitoringFileService monitoringFileService) {
this.template = template;
monitoringFileService.listen(file -> {
try {
Files.lines(file)
.skip(COUNTER.get())
.forEach(line ->
template.broadcast(TOPIC, SseEmitter.event()
.id(String.valueOf(COUNTER.incrementAndGet()))
.data(line)));
} catch (IOException e) {
e.printStackTrace();
}
});
}
public SseEmitter newSseEmitter() {
return template.newSseEmitter(TOPIC);
}
}
The custom appender (which you have to add to your logger - check here):
public class StreamAppender extends UnsynchronizedAppenderBase<ILoggingEvent> implements SmartLifecycle {
public static final String TOPIC = "logs";
private final SseTemplate template;
public StreamAppender(SseTemplate template) {
this.template = template;
}
#Override
protected void append(ILoggingEvent event) {
template.broadcast(TOPIC, SseEmitter.event()
.id(event.getThreadName())
.name("log")
.data(event.getFormattedMessage()));
}
#Override
public boolean isRunning() {
return isStarted();
}
}