Develop Reference

How to get the processing kafka topic name dynamically in Flink Kafka Consumer?

Currently, I have one Flink Cluster which wants to consume Kafka Topic by one Pattern, By using this way, we don't need to maintain one hard code Kafka topic list.
import java.util.regex.Pattern;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010;
...
private static final Pattern topicPattern = Pattern.compile("(DC_TEST_([A-Z0-9_]+)");
...
FlinkKafkaConsumer010<KafkaMessage> kafkaConsumer = new FlinkKafkaConsumer010<>(
topicPattern, deserializerClazz.newInstance(), kafkaConsumerProps);
DataStream<KafkaMessage> input = env.addSource(kafkaConsumer);
I just want to know by using the above way, How can I get to know the real Kafka topic name during the processing?
Thanks.
--Update--
The reason why I need to know the topic information is we need this topic name as the parameter to be used in the coming Flink sink part.

You can implement your own custom KafkaDeserializationSchema, like this:
public class CustomKafkaDeserializationSchema implements KafkaDeserializationSchema<Tuple2<String, String>> {
#Override
public boolean isEndOfStream(Tuple2<String, String> nextElement) {
return false;
}
#Override
public Tuple2<String, String> deserialize(ConsumerRecord<byte[], byte[]> record) throws Exception {
return new Tuple2<>(record.topic(), new String(record.value(), "UTF-8"));
}
#Override
public TypeInformation<Tuple2<String, String>> getProducedType() {
return new TupleTypeInfo<>(BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.STRING_TYPE_INFO);
}
}
With the custom KafkaDeserializationSchema, you can create DataStream of which the element contains topic infos. In my demo case the element type is Tuple2<String, String>, so you can access the topic name by Tuple2#f0.
FlinkKafkaConsumer010<Tuple2<String, String>> kafkaConsumer = new FlinkKafkaConsumer010<>(
topicPattern, new CustomKafkaDeserializationSchema, kafkaConsumerProps);
DataStream<Tuple2<String, String>> input = env.addSource(kafkaConsumer);
input.process(new ProcessFunction<Tuple2<String,String>, String>() {
#Override
public void processElement(Tuple2<String, String> value, Context ctx, Collector<String> out) throws Exception {
String topicName = value.f0;
// your processing logic here.
out.collect(value.f1);
}
});

There are two ways to do that.
Option 1 :
You can use Kafka-clients library to access the Kafka metadata, get topic lists. Add maven dependency or equivalent.
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>2.3.0</version>
</dependency>
You can fetch topics from Kafka cluster and filter using regex as given below
private static final Pattern topicPattern = Pattern.compile("(DC_TEST_([A-Z0-9_]+)");
Properties properties = new Properties();
properties.put("bootstrap.servers","localhost:9092");
properties.put("client.id","java-admin-client");
try (AdminClient client = AdminClient.create(properties)) {
ListTopicsOptions options = new ListTopicsOptions();
options.listInternal(false);
Collection<TopicListing> listing = client.listTopics(options).listings().get();
List<String> allTopicsList = listings.stream().map(TopicListing::name)
.collect(Collectors.toList());
List<String> matchedTopics = allTopicsList.stream()
.filter(topicPattern.asPredicate())
.collect(Collectors.toList());
}catch (Exception e) {
e.printStackTrace();
}
}
Once you have matchedTopics list, you can pass that to FlinkKafkaConsumer.
Option 2 :
FlinkKafkaConsumer011 in Flink release 1.8 supports Topic & partition discovery dynamically based on pattern. Below is the example :
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
private static final Pattern topicPattern = Pattern.compile("(DC_TEST_([A-Z0-9_]+)");
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:9092");
properties.setProperty("group.id", "test");
FlinkKafkaConsumer011<String> myConsumer = new FlinkKafkaConsumer011<>(
topicPattern ,
new SimpleStringSchema(),
properties);
Link : https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-consumers-topic-and-partition-discovery
In your case, option 2 suits best.
Since you want to access topic metadata as part of KafkaMessage, you need to implement KafkaDeserializationSchema interface as given below :
public class CustomKafkaDeserializationSchema extends KafkaDeserializationSchema<KafkaMessage> {
/**
* Deserializes the byte message.
*
* #param messageKey the key as a byte array (null if no key has been set).
* #param message The message, as a byte array (null if the message was empty or deleted).
* #param partition The partition the message has originated from.
* #param offset the offset of the message in the original source (for example the Kafka offset).
*
* #return The deserialized message as an object (null if the message cannot be deserialized).
*/
#Override
public KafkaMessage deserialize(ConsumerRecord<byte[], byte[]> record) throws IOException {
//You can access record.key(), record.value(), record.topic(), record.partition(), record.offset() to get topic information.
KafkaMessage kafkaMessage = new KafkaMessage();
kafkaMessage.setTopic(record.topic());
// Make your kafka message here and assign the values like above.
return kafkaMessage ;
}
#Override
public boolean isEndOfStream(Long nextElement) {
return false;
}
}
And then call :
FlinkKafkaConsumer010<Tuple2<String, String>> kafkaConsumer = new FlinkKafkaConsumer010<>(
topicPattern, new CustomKafkaDeserializationSchema, kafkaConsumerProps);

Consumer unit test with poll() never receives anything

Consider the following code:
#Test(singleThreaded = true)
public class KafkaConsumerTest
{
private KafkaTemplate<String, byte[]> template;
private DefaultKafkaConsumerFactory<String, byte[]> consumerFactory;
private static final KafkaEmbedded EMBEDDED_KAFKA;
static {
EMBEDDED_KAFKA = new KafkaEmbedded(1, true, "topic");
try { EMBEDDED_KAFKA.before(); } catch (final Exception e) { e.printStackTrace(); }
}
#BeforeMethod
public void setUp() throws Exception {
final Map<String, Object> senderProps = KafkaTestUtils.senderProps(EMBEDDED_KAFKA.getBrokersAsString());
senderProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
senderProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
final ProducerFactory<String, byte[]> pf = new DefaultKafkaProducerFactory<>(senderProps);
this.template = new KafkaTemplate<>(pf);
this.template.setDefaultTopic("topic");
final Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("sender", "false", EMBEDDED_KAFKA);
this.consumerFactory = new DefaultKafkaConsumerFactory<>(consumerProps);
this.consumerFactory.setValueDeserializer(new ByteArrayDeserializer());
this.consumerFactory.setKeyDeserializer(new StringDeserializer());
}
#Test
public void testSendToKafka() throws InterruptedException, ExecutionException, TimeoutException {
final String message = "42";
final Message<byte[]> msg = MessageBuilder.withPayload(message.getBytes(StandardCharsets.UTF_8)).setHeader(KafkaHeaders.TOPIC, "topic").build();
this.template.send(msg).get(10, TimeUnit.SECONDS);
final Consumer<String, byte[]> consumer = this.consumerFactory.createConsumer();
consumer.subscribe(Collections.singleton("topic"));
final ConsumerRecords<String, byte[]> records = consumer.poll(10000);
Assert.assertTrue(records.count() > 0);
Assert.assertEquals(new String(records.iterator().next().value(), StandardCharsets.UTF_8), message);
consumer.commitSync();
}
}
I am trying to send a message to a KafkaTemplate and read it again using Consumer.poll(). The test framework I am using is TestNG.
Sending works, I have verified that using the "usual" code I found in the net (register a message listener on a KafkaMessageListenerContainer).
Only, I never receive anything in the consumer. I have tried the same sequence (create Consumer, poll()) against a "real" Kafka installation, and it works.
Hence it looks like there is something wrong with the way I set up my ConsumerFactory? Any help would be greatly appreciated!

You need to use
EMBEDDED_KAFKA.consumeFromAnEmbeddedTopic(consumer, "topic");
before publishing records via KafkaTemplate.
And then in the end of test for verification you need to use something like this:
ConsumerRecord<String, String> record = KafkaTestUtils.getSingleRecord(consumer, "topic");
You can also use it the way you do, only what you are missing is a ConsumerConfig.AUTO_OFFSET_RESET_CONFIG as an earliest, because the default one is latest. That way a consumer added to the topic later won't see any records published before.

KafkaUtils.createDirectStream does not take right parameters - Spark Streaming + Kafka

I have an application which sends serialized Twitter data to a Kafka topic. All good so far.
The consumer application should read data and deserialize it. Now, when I call KafkaUtils.createDirectStream, I think I put the right parameters (as you will see inside the thrown error), so I can't understand why it is not working.
The method createDirectStream(JavaStreamingContext, Class -K-,
Class -V-, Class -KD-, Class -VD-, Map -String,String-, Set -String-) in
the type KafkaUtils is not applicable for the arguments
(JavaStreamingContext, Class-String-, Class-Status-,
Class -StringDeserializer-, Class -StatusDeserializer-,
Map-String,String-, Set-String-)
Checking the Spark Javadoc, my params still seem right to me.
My code is:
Set<String> topics = new HashSet<>();
topics.add("twitter-test");
JavaStreamingContext jssc = new JavaStreamingContext(jsc, new Duration(duration));
Map<String, String> props = new HashMap<>();
//some properties...
JavaPairInputDStream messages = KafkaUtils.createDirectStream(jssc, String.class, Status.class, org.apache.kafka.common.serialization.StringDeserializer.class, stream_data.StatusDeserializer.class, props, topics);
Status serializer code:
public class StatusSerializer implements Serializer<Status> {
#Override public byte[] serialize(String s, Status o) {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(o);
oos.close();
byte[] b = baos.toByteArray();
return b;
} catch (IOException e) {
return new byte[0];
}
}
#Override public void close() {
}
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
}
}

Looks like the issue is with "stream_data.StatusDeserializer.class". Can you please the code of this custom deserializer class. Also, can you please look into this Kafka Consumer for Spark written in Scala for Kafka API 0.10: custom AVRO deserializer .
Include the below in the KafkaParam arguments.
key.deserializer -> classOf[StringDeserializer]
value.deserializer -> classOf[StatusDeserializer]

Mocking a Vertx.io async handler

when I was sync I wrote unit tests mocking the persistence part and check the caller's behavior. Here is an example about what I usually did:
#Mock
private OfferPersistenceServiceImpl persistenceService;
#Inject
#InjectMocks
private OfferServiceImpl offerService;
...
#Test
public void createInvalidOffer() {
offer = new Offer(null, null, null, null, null, 4, 200D, 90D);
String expectedMessage = Offer.class.getName() + " is not valid: " + offer.toString();
Mockito.when(persistenceService.create(offer)).thenThrow(new IllegalArgumentException(expectedMessage));
Response response = offerService.create(offer);
Mockito.verify(persistenceService, Mockito.times(1)).create(offer);
Assert.assertEquals(INVALID_INPUT, response.getStatus());
String actualMessage = response.getEntity().toString();
Assert.assertEquals(expectedMessage, actualMessage);
}
But now I fell in love with Vertx.io (to which I am pretty new) and I want to be async. Nice. But Vertx has handlers, so the new persistence component to mock looks like this:
...
mongoClient.insert(COLLECTION, offer, h-> {
...
});
So I am guessing how to mock handler h to tests class who's using that mongoClient or even if it is the right way to test with Vertx.io. I am using vertx.io 3.5.0, junit 4.12 and mockito 2.13.0. Thanks.
Update
I tried to follow tsegimond suggestion but I can't get how Mockito's Answer and ArgumentCaptor can help me. Here is what I tried so far.
Using ArgumentCaptor:
JsonObject offer = Mockito.mock(JsonObject.class);
Mockito.when(msg.body()).thenReturn(offer);
Mockito.doNothing().when(offerMongo).validate(offer);
RuntimeException rex = new RuntimeException("some message");
...
ArgumentCaptor<Handler<AsyncResult<String>>> handlerCaptor =
ArgumentCaptor.forClass(Handler.class);
ArgumentCaptor<AsyncResult<String>> asyncResultCaptor =
ArgumentCaptor.forClass(AsyncResult.class);
offerMongo.create(msg);
Mockito.verify(mongoClient,
Mockito.times(1)).insert(Mockito.anyString(), Mockito.any(), handlerCaptor.capture());
Mockito.verify(handlerCaptor.getValue(),
Mockito.times(1)).handle(asyncResultCaptor.capture());
Mockito.when(asyncResultCaptor.getValue().succeeded()).thenReturn(false);
Mockito.when(asyncResultCaptor.getValue().cause()).thenReturn(rex);
Assert.assertEquals(Json.encode(rex), msg.body().encode());
and using Answer:
ArgumentCaptor<AsyncResult<String>> handlerCaptor =
ArgumentCaptor.forClass(AsyncResult.class);
AsyncResult<String> result = Mockito.mock(AsyncResult.class);
Mockito.when(result.succeeded()).thenReturn(true);
Mockito.when(result.cause()).thenReturn(rex);
Mockito.doAnswer(new Answer<MongoClient>() {
#Override
public MongoClient answer(InvocationOnMock invocation) throws Throwable {
((Handler<AsyncResult<String>>)
invocation.getArguments()[2]).handle(handlerCaptor.capture());
return null;
}
}).when(mongoClient).insert(Mockito.anyString(), Mockito.any(),
Mockito.any());
userMongo.create(msg);
Assert.assertEquals(Json.encode(rex), msg.body().encode());
And now I got confused. Is there a way to mock an AsyncResult to let it return false on succeed()?

Finally I got some times to investigate and I made it. Here is my solution.
#RunWith(PowerMockRunner.class)
#PowerMockRunnerDelegate(VertxUnitRunner.class)
#PrepareForTest({ MongoClient.class })
public class PersistenceTest {
private MongoClient mongo;
private Vertx vertx;
#Before
public void initSingleTest(TestContext ctx) throws Exception {
vertx = Vertx.vertx();
mongo = Mockito.mock(MongoClient.class);
PowerMockito.mockStatic(MongoClient.class);
PowerMockito.when(MongoClient.createShared(Mockito.any(), Mockito.any())).thenReturn(mongo);
vertx.deployVerticle(Persistence.class, new DeploymentOptions(), ctx.asyncAssertSuccess());
}
#SuppressWarnings("unchecked")
#Test
public void loadSomeDocs(TestContext ctx) {
Doc expected = new Doc();
expected.setName("report");
expected.setPreview("loremipsum");
Message<JsonObject> msg = Mockito.mock(Message.class);
Mockito.when(msg.body()).thenReturn(JsonObject.mapFrom(expected));
JsonObject result = new JsonObject().put("name", "report").put("preview", "loremipsum");
AsyncResult<JsonObject> asyncResult = Mockito.mock(AsyncResult.class);
Mockito.when(asyncResult.succeeded()).thenReturn(true);
Mockito.when(asyncResult.result()).thenReturn(result);
Mockito.doAnswer(new Answer<AsyncResult<JsonObject>>() {
#Override
public AsyncResult<JsonObject> answer(InvocationOnMock arg0) throws Throwable {
((Handler<AsyncResult<JsonObject>>) arg0.getArgument(3)).handle(asyncResult);
return null;
}
}).when(mongo).findOne(Mockito.any(), Mockito.any(), Mockito.any(), Mockito.any());
Async async = ctx.async();
vertx.eventBus().send("persistence", new JsonObject(), msgh -> {
if (msgh.failed()) {
System.out.println(msgh.cause().getMessage());
}
ctx.assertTrue(msgh.succeeded());
ctx.assertEquals(expected, Json.decodeValue(msgh.result().body().toString(), Doc.class));
async.complete();
});
async.await();
}
}
Use Powemockito to mock the MongoClient.createShared static method, so you'll have your mock when verticle starts. Mocking async handler is a bit of code to write. As you can see mocking start at Message<JsonObject> msg = Mockito.mock(Message.class); and ends at Mockito.doAnswer(new Answer.... In the Answer's method pick the handler param and force it to handle your async result then you're done.

Normally, I'd use a comment to post this, but formatting gets lost. The accepted solution is works great, just note that it can be simplified a bit using Java 8+, and you can use your actual objects instead of JSON.
doAnswer((Answer<AsyncResult<List<Sample>>>) arguments -> {
((Handler<AsyncResult<List<Sample>>>) arguments.getArgument(1)).handle(asyncResult);
return null;
}).when(sampleService).findSamplesBySampleFilter(any(), any());
getArgument(1), refers to the index of the handler argument in a method such as:
#Fluent
#Nonnull
SampleService findSamplesBySampleFilter(#Nonnull final SampleFilter sampleFilter,
#Nonnull final Handler<AsyncResult<List<Sample>>> resultHandler);

PowerMockito.spy breaks test when used for verify

I have a unit test (simplified version below) that tests a Netty handler.
I create an EmbeddedChannel with an instance of the handler.
The caller writes a string to the channel
The handler receives the string, reverses and writes it back.
The caller reads the return values from the channel and verifies it is the reverse of the sent string.
This works perfectly. However, I need to verify the number of invocations on the channel, so I created a spy of the channel but mocked no methods, since I don't want to change the behavior of the class, just count invocations.
Now the test fails. 2 of the assertions succeed. They are a test to make sure the handler was called, and a test to verify the number of times a method of the channel was called. However, the final read response is always null when the spy is used.
I was under the impression that a solitary spy with no other mocking would not affect the behavior of the spied object, but obviously it does. The [nonPower] Mockito docs indicate the objects are copied which might cause this issue, but the PowerMockito docs are not as specific.
I am using Netty 4.1.6.Final and Powermock 1.5.6.
UPDATE: I managed to get the test working but it's a bit of wonky workaround. See the new method testSpiedEmbeddedChannel2. The workaround is that I create a non-spied channel (ecx), then a the spied channel (ec) using ecx. I issued the write on ec, and the read using ecx. This means if I try to verify methods used in the read, they will not be counted.
Here's the code with the successful and failing tests.
#RunWith(PowerMockRunner.class)
#PowerMockIgnore({"javax.management.*"})
#PrepareForTest(EmbeddedChannel.class)
public class TestEmbeddedChannel {
class EchoHandler extends ChannelDuplexHandler {
final AtomicInteger reads = new AtomicInteger(0);
#Override
public void channelRead(final ChannelHandlerContext ctx, final Object msg) throws Exception {
reads.incrementAndGet();
final String value = (String)msg;
final String response = new StringBuilder(value).reverse().toString();
ctx.channel().writeAndFlush(response);
}
}
#Test
public void testEmbeddedChannel() { // PASSES
final EchoHandler handler = new EchoHandler();
final EmbeddedChannel ec = new EmbeddedChannel(handler);
ec.writeInbound("Hello World");
final String response = ec.readOutbound();
Assert.assertEquals(1, handler.reads.get());
Assert.assertEquals("dlroW olleH", response);
}
#Test
public void testSpiedEmbeddedChannel() { // FAILS
final EchoHandler handler = new EchoHandler();
final EmbeddedChannel ec = spy(new EmbeddedChannel(handler));
ec.writeInbound("Hello World");
final String response = ec.readOutbound();
verify(ec, times(2)).isOpen(); // OK
Assert.assertEquals(1, handler.reads.get()); // OK
Assert.assertEquals("dlroW olleH", response); // FAILS
}
#Test
public void testSpiedEmbeddedChannel2() { // PASSES
final EchoHandler handler = new EchoHandler();
final EmbeddedChannel ecx = new EmbeddedChannel(handler);
final EmbeddedChannel ec = spy(ecx);
ec.writeInbound("Hello World");
final String response = ecx.readOutbound(); // Read using non-spied channel
verify(ec, times(2)).isOpen();
Assert.assertEquals(1, handler.reads.get());
Assert.assertEquals("dlroW olleH", response);
}
}
Thanks for any guidance here.