Develop Reference

How to integrate Camel Spring boot with Kafka in Confluent Cloud?

I have an application which uses camel Spring Boot with debeezium to listen to a mysql database and to publish in a Kafka topic.
It was all working fine since I changed the kafka from local to use confluent cloud. I have some other applications (normal producers and consumers) that connect to confluent cloud and all works fine.
This is my application.yml. I removed the debeezium-mysql part because it works fine. So I left only the config part of Kafka/Confluent.
routes:
debezium:
allow-public-key-retrieval: true
bootstrap-servers: ${application.kafka.brokers}
offset-storage:
topic-cleanup-policy: compact
camel:
component:
debezium-mysql:
ALL CONFIG with mysql, it is not here because it is working fine
kafka:
brokers: ${application.kafka.brokers}
schema-registry-u-r-l: ${application.schema-registry.base-urls}
value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
key-serializer: org.apache.kafka.common.serialization.StringSerializer
additional-properties:
#CCloud Schema Registry Connection parameter
schema.registry.basic.auth.credentials.source: USER_INFO
schema.registry.basic.auth.user.info: ${SCHEMA_REGISTRY_ACCESS_KEY}:${SCHEMA_REGISTRY_SECRET_KEY}
ssl.endpoint.identification.algorithm: https
client.dns.lookup: use_all_dns_ips
sasl-jaas-config: org.apache.kafka.common.security.plain.PlainLoginModule required username="${CONFLUENT_CLOUD_USERNAME}" password="${CONFLUENT_CLOUD_PASSWORD}";
security-protocol: SASL_SSL
retry-backoff-ms: 500
request-timeout-ms: 20000
sasl-mechanism: PLAIN
With this config, it keeps giving me an error when I try to star the app:
[AdminClient clientId=adminclient-1] Node -1 disconnected.
2023-02-02 16:57:24.853 INFO 9644 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Cancelled in-flight API_VERSIONS request with correlation id 0 due to node -1 being disconnected (elapsed time since creation: 146ms, elapsed time since send: 146ms, request timeout: 3600000ms)
I could check that the problem is in this AdminConfig, which doesn't get the right properties. An example, the security.protocol should be SASL_SSL, but it gets PLAINTEXT. But when creating the producer and consumer it gets the right values.
Really, I have been struggling with this two days. I would be really happy with any help. Thank you.

Failed to connect Azure servicebus topic using JMS - Java

I followed steps as mentioned in Azure ServiceBus JMS Sample with below properties
spring.jms.servicebus.connection-string=Endpoint=sb://test-dt.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=key
spring.jms.servicebus.topic-client-id=12345
spring.jms.servicebus.idle-timeout=18000
spring.jms.servicebus.pricing-tier=Standard
however I get below error
ERROR 43904 --- [ntContainer#0-1] org.apache.qpid.jms.JmsConnection : Failed to connect to remote at: amqps://test-dt.servicebus.windows.net:-1
ERROR 43904 --- [ntContainer#0-1] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'test-topic' - retrying using FixedBackOff{interval=5000, currentAttempts=6, maxAttempts=unlimited}. Cause: handshake timed out after 10000ms
On the other hand, I followed steps as mentioned in ServiceBus without JMS and added transportType as AmqpTransportType.AMQP_WEB_SOCKETS then I am able to connect.
We want to implement using spring boot starter and listener method, instead of calling from (public static void main) method.
Please guide on what am I missing when following first link

ERROR 43904 --- [ntContainer#0-1] org.apache.qpid.jms.JmsConnection : Failed to connect to remote at: amqps://test-dt.servicebus.windows.net:-1
To resolve above error, try as suggested by Anand Sowmithiran:
Check if port 5671 is blocked:
telnet <yournamespacename>.servicebus.windows.net 5671
Note: Clients that use AMQP connections over TCP require ports 5671 and 5672 to be opened in the firewall. Along with these ports, it might be necessary to open additional ports if the EnableLinkRedirect feature is enabled.
You can refer to Troubleshooting guide for Azure Service Bus, AMQP outbound port requirements and Port 5671 Blocked :(. What are other options?

Log4j2 Syslog appender is not writing the 1st message to syslog after the syslog service is restarted

We have configured our application to write some specific log messages to System's Syslog file using the Syslog appender of Log4j2. No issue in writing the Syslog to the file. But when the syslog service is restarted, the first log message is not written to the syslog. The subsequent messages are written.
Enabled debug logs of Log4j, no exception is seen while writing 1st message to syslog after the restart. But for the subsequent request, the following messages were captured in the Log4j2 log.
2022-01-27 18:07:40,120 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Reconnecting localhost/127.0.0.1:514
2022-01-27 18:07:40,121 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Creating socket localhost/127.0.0.1:514
2022-01-27 18:07:40,122 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Closing SocketOutputStream java.net.SocketOutputStream#1a769d7
2022-01-27 18:07:40,122 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Connection to localhost:514 reestablished: Socket[addr=localhost/127.0.0.1,port=514,localport=57852]
I took threaddump and checked whether the Reconnector thread is running but no such exists in the threaddump. I am clueless here, any help on finding the reason for missing the message would be helpful.
Environment details:
CentOS 7.9 + RSyslog Service,
Application deployed in Tomcat and running on Java 11,
Log4j2 version is 2.17.1

This is due to the way plain text TCP syslog works. Check out this post for further information.
This "bug" exists, since version 8.1901 and newer.
The only way you can fix this - as far as i know - is to send the messages over the RELP protocol. See omrelp module.

Apache Kafka: Failed to Update Metadata/java.nio.channels.ClosedChannelException

I'm just getting started with Apache Kafka/Zookeeper and have been running into issues trying to set up a cluster on AWS. Currently I have three servers:
One running Zookeeper and two running Kafka.
I can start the Kafka servers without issue and can create topics on both of them. However, the trouble comes when I try to start a producer on one machine and a consumer on the other:
on the Kafka producer:
kafka-console-producer.sh --broker-list <kafka server 1 aws public dns>:9092,<kafka server 2 aws public dns>:9092 --topic samsa
on the Kafka consumer:
kafka-console-consumer.sh --zookeeper <zookeeper server ip>:2181 --topic samsa
I type in a message on the producer ("hi") and nothing happens for a while. Then I get this message:
ERROR Error when sending message to topic samsa with key: null, value: 2 bytes
with error: Failed to update metadata after 60000 ms.
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
On the consumer side I get this message, which repeats periodically:
WARN Fetching topic metadata with correlation id # for topics [Set(samsa)] from broker [BrokerEndPoint(<broker.id>,<producer's advertised.host.name>,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
After a while, the producer will then start rapidly throwing this error message with # increasing incrementally:
WARN Error while fetching metadata with correlation id # : {samsa=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
Not sure where to go from here. Let me know if more details about my configuration files are needed

This was a configuration issue.
In order to get it running several changes to config files had to happen:
In config/server.properties on each Kafka server:
host.name: <Public IP>
advertised.host.name: <AWS Public DNS Address>
In config/producer.properties on each Kafka server:
metadata.broker.list: <Producer Server advertised.host.name>:<Producer Server port>,<Consumer Server advertised.host.name>:<Consumer Server port>
In /etc/hosts on each Kafka server, change 127.0.0.1 localhost localhost.localdomain to:
<Public IP> localhost localhost.localdomain

kafka + zookeeper remote = error

I am trying to install a kafka & zookeeper instance on a remote server. I only need 1 node of each actually because i only want to provide remote kafka for test purposes.
Kafka and Zookeeper are running from the Apache Kafka tarball you can find there (v0.0.9), inside a Docker image.
Trying to consume / produce using the provided scripts. And trying to produce using own java application. Everythinf is working fine if Kafka & ZK are installed on the local server.
Here is the error I get while trying to produce :
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
No partition metadata for topic RSS due to kafka.common.LeaderNotAvailableException}] for topic [RSS]: class kafka.common.LeaderNotAvailableException
Kafka properties tested
First :
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=localhost:<PORT>
Second:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
Third:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Last:
borker.id=0
port=9092
host.name=</etc/host name>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Here is my "/etc/hosts"
127.0.0.1 kafka kafka
127.0.0.1 localhost
I followed the Getting Started, which if I understood is a localhost / signle server configurations. I cannot understand what I have to do to get this work with remote calls...
Thanks for your help !
EDIT 1
host.name=localhost
advertised.host.name=politik.cm-cloud.fr
Seems to allow a local consumer (on the server) and producer. But if we want to do the same from a remote server we get
[2015-12-09 12:44:10,826] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host

The error does not look like connectivity problem with Zookeeper / Kafka.
Just follow the instruction in "quickstart" from http://kafka.apache.org/
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
Additionally the error indicates there is no partition info i.e topic not yet created . Try creating topics first and then try to produce/consume because when producing to a non existent topic kafka will create the topic based on auto.create.topics.enable in server.properties but remotely it is better to create topics rathen than relying on auto create