Iam using spring kafka and avro,
The schema registry is secured with plain sasl protocol.
I must develop a kafka consumer and producer, i don't find a way with spring kafka to configure the security properties for schema registry.
Does spring kafka library already integrate this configuration ?
I need to configure kafkastore.. In a schema registry properties file?
I didn't find any examples with spring kafka
there is docs like this related to your question and demo examples in github.
note that with spring you can have common properties for kafka, just make sure to use spring kafka dependency, maybe first try to establish connection between both consumer and producer and then put sasl plain security.
Related
I`m migrating and upgrading the project, from Spring Cloud Netflix to New Spring Cloud(Spring loadbalancer CircuitBreak).
https://spring.io/blog/2018/12/12/spring-cloud-greenwich-rc1-available-now#spring-cloud-netflix-projects-entering-maintenance-mode
I have some configurations where I can't find the equivalent config in Spring loadbalancer And CircuitBreak. In my limited experience with Spring ,I searching for a long time on net. But no use. --Here is my questions:
old-ribbon-config migration to loadbalancer:
serverName.ribbon.MaxAutoRetries=0
serverName.ribbon.MaxAutoRetriesNextServer=2
serverName.ribbon.retryableStatusCodes=503
serverName.ribbon.NFLoadBalancerRuleClassName=com.netflix.loadbalancer.RoundRobinRule
serverName.ribbon.ReadTimeout=5000
serverName.ribbon.ConnectTimeout=5000
serverName2.ribbon.ReadTimeout=6000
serverName2.ribbon.ConnectTimeout=7000
The closest configuration as below:
But I need to configure it for each server-name particularly the 'MaxAutoRetriesNextServer' and enabled retry for per server-name.
spring.cloud.loadbalancer.retry.enabled=false
spring.cloud.loadbalancer.retry.max-retries-on-same-service-instance=0
spring.cloud.loadbalancer.retry.max-retries-on-next-service-instance=2
The other configuration is hystrix timeout, In resilience4j it needs to return a CompletedFuture type, which is not compatible with older code.
old-hystrix-config migration to resilience4j:
hystrix.command.FeignClientName#method(String).execution.isolation.thread.timeoutInMilliseconds=2000
resilience4j.timelimiter.instances.name.timeout-duration=6s
What would be the best way to do that? Thanks in advance!
Is there a configuration I can use to instruct Spring to continue on startup and initialize the Beans even if Kafka connection failed?
I am using Spring Framework 5.2.3 and Spring Kafka 2.5.3.RELEASE.
If you need kafka beans for your application to work in every use case then continue with startup if there is no kafka connection makes no sense. Your application will not be able to do anything without kafka.
But if some parts of your application do not need kafka and you would like to use only those parts then you can either mark kafka related beans as lazy or make all beans lazy by default. In this case spring will create beans only when they are actually needed. And even if there is no kafka connection available parts of your app that do not need kafka will work.
How Do I connect to multiple bootstrap servers(DEV, STAGE and PROD) from a microservice ( Admin MS) with security in place?
I want to connect to all the kafka servers and create/manage topics, create ACLS etc.
I am using spring kafka adminclient , and configuring properties from application.yml using spring boot to connect to Dev right now. But now I want to connect to all environments.
Is there an easier and better approach other that wring a properties hashmap and putting config values in it. Does Spring cloud stream help?
Is this something similar to connecting multiple databases to a micro service ?
You can do it by creating multiple child boot applications, each with its own environment containing the properties.
But it's probably easier to bypass Boot's auto configuration and wire up your own AdminClients with their own properties.
The Kafka guide from Quarkus works nicely when running Kafka locally in Docker. I'm trying to change this sample by replacing the local Kafka service with a hosted Kafka service in the cloud which requires TLS.
Does anyone know how I can configure this? In the Quarkus documentation and the Smallrye documentation I don't see any properties for this.
I'd like to use the Kafka service in the IBM Cloud. Based on the documentation I've tried the following configuration in application.properties:
kafka.bootstrap.servers=broker-0-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-4-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-3-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-5-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-2-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-1-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093
kafka.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="...";
kafka.sasl.mechanism=PLAIN
kafka.security.protocol=SASL_SSL
kafka.ssl.protocol=TLSv1.2
Update:
I've also tried Gunnar's suggestion below, but it doesn't work. When I use the following application.properties ...
mp.messaging.outgoing.generated-price.connector=smallrye-kafka
mp.messaging.outgoing.generated-price.topic=prices
mp.messaging.outgoing.generated-price.value.serializer=org.apache.kafka.common.serialization.IntegerSerializer
mp.messaging.outgoing.generated-price.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="...";
mp.messaging.outgoing.generated-price.sasl.mechanism=PLAIN
mp.messaging.outgoing.generated-price.security.protocol=SASL_SSL
mp.messaging.outgoing.generated-price.ssl.protocol=TLSv1.2
mp.messaging.incoming.prices.connector=smallrye-kafka
mp.messaging.incoming.prices.topic=prices
mp.messaging.incoming.prices.value.deserializer=org.apache.kafka.common.serialization.IntegerDeserializer
mp.messaging.outgoing.prices.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="...";
mp.messaging.outgoing.prices.sasl.mechanism=PLAIN
mp.messaging.outgoing.prices.security.protocol=SASL_SSL
mp.messaging.outgoing.prices.ssl.protocol=TLSv1.2
kafka.bootstrap.servers=broker-0-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-4-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-3-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-5-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-2-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093,broker-1-8c8cph49mx2p2wqy.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093
... I get an error:
javax.enterprise.inject.spi.DeploymentException: java.lang.IllegalArgumentException: Invalid channel configuration - the connector attribute must be set for channel prices
at io.quarkus.smallrye.reactivemessaging.runtime.SmallRyeReactiveMessagingLifecycle.onApplicationStart(SmallRyeReactiveMessagingLifecycle.java:22)
Is TLS currently possible for Kafka in Quarkus?
Thanks
Have you tried specifying the relevant properties at the channel level? E.g.
mp.messaging.outgoing.generated-price.connector=smallrye-kafka
mp.messaging.outgoing.generated-price.topic=mytopic
mp.messaging.outgoing.generated-price.ssl.protocol=...
mp.messaging.outgoing.generated-price.ssl.keystore.location=...
mp.messaging.outgoing.generated-price.ssl.keystore.password=...
You also could refer to variables when requiring the same values for multiple topics.
One property is incorrect in the accepted answer by #Gunnar. It should be "security" instead of "ssl" in the property name.
mp.messaging.outgoing.generated-price.security.protocol=SSL
I am trying to connect to kafka server via spring integration module with SASL config and get error
java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
but when I construct simple consumer and poll messages everything works fine. Can someone tell me how to turn off JAAS authorization or connect via it properly.
Here is my KafkaConfig.java
and SaslConfiguration.java. Thanks for answer!
There is a KafkaConsumerFactory which simply can accept the same set of properties you are mentioning in you gist: https://docs.spring.io/spring-kafka/docs/2.1.10.RELEASE/reference/html/_reference.html#_receiving_messages