Develop Reference

Kafka apache camel sink task fails with weird ClassCastException

I am trying to sink kafka topic records into s3 bucket using kafka-connect + camel-kafka-connector-0.9.
The connector loads up fine, I can see it connected to kafka as the consumer can be seen in AKHQ. But it fails immediately just after trying to commit offsets with the following exception from the kafka-connect pod:
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:614)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:189)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:238)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassCastException: class java.math.BigDecimal cannot be cast to class [B (java.math.BigDecimal and [B are in module java.base of loader 'bootstrap')
at org.apache.camel.kafkaconnector.CamelSinkTask.mapHeader(CamelSinkTask.java:233)
at org.apache.camel.kafkaconnector.CamelSinkTask.put(CamelSinkTask.java:184)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:586)
... 10 more
I don't understand whats going on under the hood and there is little room for debugging...
Kafka-connector config:
{
"name": "connector-name",
"config": {
"connector.class": "org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SinkConnector",
"topics": "topic-v1-0",
"camel.sink.endpoint.region": "region",
"camel.sink.path.bucketNameOrArn": "bucket-name",
"camel.sink.endpoint.keyName": "incoming-v1-0/${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}",
"camel.sink.endpoint.useDefaultCredentialsProvider": "true",
"camel.beans.aggregate": "#class:org.apache.camel.kafkaconnector.aggregator.StringAggregator",
"camel.aggregation.size": "1000",
"camel.aggregation.timeout": "5000",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable": "false",
"value.converter.schemas.enable": "false"
}
}

twilio phoneNUmber pricing api throw exception

https://pricing.twilio.com/v1/PhoneNumbers/Countries/{countryCode}
API is fetching country and give a response like
expected response =>
{
"url": "https://pricing.twilio.com/v1/PhoneNumbers/Countries/US",
"country": "United States",
"price_unit": "USD",
"phone_number_prices": [
{
"number_type": "local",
"base_price": "1.00",
"current_price": "1.00"
},
{
"number_type": "toll free",
"base_price": "2.00",
"current_price": "2.00"
}
],
"iso_country": "US"
}
but I am facing a problem while fetching country, it will give exception
Country country= Country.fetcher(countryCode).fetch();
exception =>
Unrecognized field "number_type" (class com.twilio.type.PhoneNumberPrice), not marked as ignorable (5 known properties: "basePrice", "type", "base_price", "currentPrice", "current_price"])
at [Source: (org.apache.http.conn.EofSensorInputStream); line: 1, column: 212] (through reference chain: com.twilio.rest.pricing.v1.phonenumber.Country["phone_number_prices"]-
>java.util.ArrayList[0]->com.twilio.type.PhoneNumberPrice["number_type"])
how to resolve this problem

Twilio developer evangelist here.
What version of the Twilio Java library are you using? This looks like an issue that was fixed back in March 2017.
I recommend you upgrade the version of the Twilio Java library you are using and if that doesn't solve the issue, raise an issue on the library repo.

SerializationException: Error deserializing Avro message (StringIndexOutOfBoundsException)

I've run into this error when KafkaStream tries to deserialise the Arvo message:
[filtering-app-6adef284-11eb-48f8-8ca0-cde7da5224ab-StreamThread-1] ERROR org.apache.kafka.streams.KafkaStreams - stream-client [filtering-app-6adef284-11eb-48f8-8ca0-cde7da5224ab] All stream threads have died. The instance will be in error state and should be closed.
[filtering-app-6adef284-11eb-48f8-8ca0-cde7da5224ab-StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [filtering-app-6adef284-11eb-48f8-8ca0-cde7da5224ab-StreamThread-1] Shutdown complete
Exception in thread "filtering-app-6adef284-11eb-48f8-8ca0-cde7da5224ab-StreamThread-1" org.apache.kafka.streams.errors.StreamsException: Deserialization exception handler is set to fail upon a deserialization error. If you would rather have the streaming pipeline continue after a deserialization error, please set the default.deserialization.exception.handler appropriately.
at org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:80)
at org.apache.kafka.streams.processor.internals.RecordQueue.maybeUpdateTimestamp(RecordQueue.java:160)
The cause exception was:
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 1
Caused by: java.lang.RuntimeException: java.lang.StringIndexOutOfBoundsException: begin 1, end 0, length 1
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1529)
and
Caused by: java.lang.StringIndexOutOfBoundsException: begin 1, end 0, length 1
at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
at java.base/java.lang.String.substring(String.java:1874)
The avro configuration is straigthforward:
{
"namespace": "io.confluent.developer.avro",
"type": "record",
"name": "Publication",
"fields": [
{"name": "name", "type": "string"},
{"name": "title", "type": "string"}
]
}
which is from this tutorial: https://kafka-tutorials.confluent.io/filter-a-stream-of-events/kstreams.html. The producer serialises the input string "{"name": "George R. R. Martin", "title": "A Dream of Spring"}" with no problem, but then the KafkaStream which basically tries to filter the event failed to deserialise the object to perform the Java filtering logic...
Has anyone encountered this problem before ? Appreciate any suggestions!

Found the issue: a proxy gets in the way.
The root cause was that the app can't connect to schema-registry. Just note it here in case someone runs into the same problem later.

Pushing mocks to remote wiremock server fails with "JSON Parsing" Error

I am trying to post mappings to a remote server from a spring application. What I found while debugging is that my JSON gets converted to "StubMapping" and this is the place where the code is failing with the following error.
Error creating bean with name 'wiremockConfig' defined in file [C:\Users\Addy\school-impl-api\target\classes\com\test\school\project\wiremock\WiremockConfig.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.test.order.implementation.product.wiremock.WiremockConfig$$EnhancerBySpringCGLIB$$b100848d]: Constructor threw exception; nested exception is com.github.tomakehurst.wiremock.common.JsonException: {
"errors" : [ {
"code" : 10,
"source" : {
"pointer" : "/mappings"
},
"title" : "Error parsing JSON",
"detail" : "Unrecognized field \"mappings\" (class com.github.tomakehurst.wiremock.stubbing.StubMapping), not marked as ignorable"
} ]
}
I got details for posting to a remote standalone server from the following issue (last comment).
https://github.com/tomakehurst/wiremock/issues/1138
My code for posting to the remote server is like this:
WireMock wm = new WireMock("https", "wiremock-poc.apps.pcf.sample.int", 443);
wm.loadMappingsFrom("src/main/resources"); // Root dir contains mappings and __files
This gets loaded when I run the profile local.
Please provide your guidance on how to solve this and move further.
Regards
Update: Sample mapping file.
{
"mappings": [
{
"request": {
"method": "GET",
"urlPathPattern": "/school/admin/rest/users/([0-9]*)?([a-zA-Z0-9_\\-\\=\\?\\.]*)"
},
"response": {
"status": 200,
"headers": {
"Content-Type": "application/json"
},
"bodyFileName": "./mockResponses/School-getUser.json"
}
}
]
}

After a discussion in chat found out that it's supported to keep each mapping in a separate file.
Here's the source code that is responsible for that: RemoteMappingsLoader#load

How to ignore Avro schema by different Confluent Registry source?

I produce the same Avro schema to one topic use different Confluent Registry sources. I get the error when I consume this topic:
org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition XXXXX_XXXX_XXX-0 at offset 0. If needed, please seek past the record to continue consumption.
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 7
Caused by: org.apache.kafka.common.errors.SerializationException: Could not find class XXXXX_XXXX_XXX specified in writer's schema whilst finding reader's schema for a SpecificRecord.
How to ignore differently Avro message-id?
Schema:
{
"type": "record",
"name": "XXXXX_XXXX_XXX",
"namespace": "aa.bb.cc.dd",
"fields": [
{
"name": "ACTION",
"type": [
"null",
"string"
],
"default":null,
"doc":"action"
},
{
"name": "EMAIL",
"type": [
"null",
"string"
],
"default":null,
"doc":"email address"
}
]
}
Produced command
{"Action": "A", "EMAIL": "xxxx#xxx.com"}

It's not possible to use different Registry urls in a producer and be able to consume them consistently.
The reason is that a different ID will be placed in the topic.
The Schema ID lookup cannot be skipped
If you had the used same registry, the same schema payload would always generate the same ID, which the consumer would then be able to use consistently to read messages