I am trying to implement a custom Kafka Partitioner using spring cloud stream bindings. I would like to just custom Partition the user topic and not do anything with company topic(Kafka will use DefaultPartitioner in this case).
My bindings configuration:
spring:
cloud:
stream:
bindings:
comp-out:
destination: company
contentType: application/json
user-out:
destination: user
contentType: application/json
As per reference document: https://cloud.spring.io/spring-cloud-static/spring-cloud-stream-binder-kafka/2.1.0.RC4/single/spring-cloud-stream-binder-kafka.html#_partitioning_with_the_kafka_binder
I modified the configuration to this:
spring:
cloud:
stream:
bindings:
comp-out:
destination: company
contentType: application/json
user-out:
destination: user
contentType: application/json
producer:
partitioned: true
partitionSelectorClass: config.UserPartitioner
I Post the message into Stream using this:
public void postUserStream(User user) throws ServiceException {
try {
LOG.info("Posting User {} into Kafka stream...", user);
MessageChannel messageChannel = messageStreams.outboundUser();
messageChannel
.send(MessageBuilder.withPayload(user)
.setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON).build());
} catch (Exception ex) {
LOG.error("Error while populating User stream into Kafka.. ", ex);
throw ex;
}
}
My UserPartitioner Class:
public class UserPartitioner extends DefaultPartitioner {
#Override
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes,
Cluster cluster) {
String partitionKey = null;
if (Objects.nonNull(value)) {
User user = (User) value;
partitionKey = String.valueOf(user.getCompanyId()) + "_" + String.valueOf(user.getId());
keyBytes = partitionKey.getBytes();
}
return super.partition(topic, partitionKey, keyBytes, value, valueBytes, cluster);
}
}
I end up receiving following exception:
Description:
Failed to bind properties under 'spring.cloud.stream.bindings.user-out.producer' to org.springframework.cloud.stream.binder.ProducerProperties:
Property: spring.cloud.stream.bindings.user-out.producer.partitioned
Value: true
Origin: "spring.cloud.stream.bindings.user-out.producer.partitioned" from property source "bootstrapProperties"
Reason: No setter found for property: partitioned
Action:
Update your application's configuration
Any reference link on how to set up Custom Partition using message binders will be helpful.
Edit: Based on the documentation Tried the below steps as well:
user-out:
destination: user
contentType: application/json
producer:
partitionKeyExtractorClass: config.SimpleUserPartitioner
#Component
public class SimpleUserPartitioner implements PartitionKeyExtractorStrategy {
#Override
public Object extractKey(Message<?> message) {
if(message.getPayload() instanceof BaseUser) {
BaseUser user = (BaseUser) message.getPayload();
return user.getId();
}
return 10;
}
}
update 2: Solution that worked for me add partitioncount to bindings and autoaddpartitions to true in binder:
spring:
logging:
level: info
cloud:
stream:
bindings:
user-out:
destination: user
contentType: application/json
producer:
partition-key-expression: headers['partitionKey']
partition-count: 4
spring:
cloud:
stream:
kafka:
binder:
brokers: localhost:9092
autoAddPartitions: true
There is no property partitioned; the getter depends on other properties...
public boolean isPartitioned() {
return this.partitionKeyExpression != null
|| this.partitionKeyExtractorName != null;
}
partitionSelectorClass: config.UserPartitioner
The UserPartitioner is a Kafka Partitioner - it determines which consumers get which partitions (on the consumer side)
The partitionSelectorClass has to be a PartitionSelectorStrategy - it determines which partition a record is sent to (on the producer side).
These are completely different objects.
If you really want to customize the way partitions are distributed across consumer instances, that is a Kafka concern and has nothing to do with Spring.
Furthermore, all consumer bindings in the same binder will use the same Partitioner. You would have to configure multiple binders to have different Partitioners.
Given your question, I think you are simply confusing Partitioner with PartitionSelectorStrategy and you need the latter.
Also, note; The partitionSelectorClass . has been deprecated for a while now and have been removed in the current master (won't be available in 3.0.0) in favor of partitionSelectorName - https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/3.0.0.M1/spring-cloud-stream.html#spring-cloud-stream-overview-partitioning
Related
I'm exploring Spring Kafka API (spring-boot-starter-parent version 2.7.4) and I found strange behavior in the consumer with standard #KafkaListener annotation.
I produce messages with KafkaTemplate and add custom header prop __ProducerApp__, but I have standard header prop __TypeId__ too because it is automatically added by Spring starter implementation.
Properties:
spring:
kafka:
bootstrap-servers: localhost:9092
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
consumer:
group-id: consumer-localhost
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
properties.spring.json.trusted.packages: '*'
Producer class:
#Component
public class KafkaExampleProducer {
private final KafkaTemplate<String, KafkaPayload> kafkaTemplate;
public KafkaExampleProducer(KafkaTemplate<String, KafkaPayload> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void sendPayload(KafkaPayload payload) {
ProducerRecord<String, KafkaPayload> record = new ProducerRecord<>(
KafkaExampleTopicConfig.EXAMPLE_TOPIC_NAME, UUID.randomUUID().toString(), payload
);
record.headers().add("__ProducerApp__", "ExampleApp-localhost".getBytes(StandardCharsets.UTF_8));
kafkaTemplate.send(record);
}
}
I can see fulfilled headers in the web UI for Apache Kafka:
But in the consumer, after receiving a message from a topic I see only the __ProducerApp__ header prop.
Listener class:
#Component
public class KafkaExampleListener {
private final Logger logger = LoggerFactory.getLogger(KafkaExampleListener.class);
#KafkaListener(topics = KafkaExampleTopicConfig.EXAMPLE_TOPIC_NAME)
public void listenMessage(
ConsumerRecord<String, KafkaPayload> consumerRecord
) {
logger.info("Received message:\nKey: {}, type: {}, producer: {}",
consumerRecord.key(),
extractHeaderValue(consumerRecord.headers(), "__TypeId__"),
extractHeaderValue(consumerRecord.headers(), "__ProducerApp__")
);
}
private String extractHeaderValue(Headers headers, String headerId) {
return StreamSupport.stream(headers.spliterator(), false)
.filter(header -> header.key().equals(headerId))
.findFirst()
.map(header -> new String(header.value()))
.orElse("N/A");
}
}
The console result presents that headers are received without __TypeId__ prop:
Received message:
Key: 3e8ee64e-b691-48e1-98b1-614291cc0451, type: N/A, producer: ExampleApp-localhost
You did not add your beans configs, but my guess is that you are missing the correct deserializer props.
Add:
#Bean
RecordMessageConverter messageConverter() { return new
StringJsonMessageConverter(); }
Also, instead of a JsonDeserializer use a StringDeserializer in your consumer value-deserializer
The JsonDeserializer strips the type headers by default to avoid polluting the application with internals.
/**
* Set to false to retain type information headers after deserialization.
* Default true.
* #param removeTypeHeaders true to remove headers.
* #since 2.2
*/
public void setRemoveTypeHeaders(boolean removeTypeHeaders) {
this.removeTypeHeaders = removeTypeHeaders;
this.setterCalled = true;
}
Or set spring.json.remove.type.headers: false.
I have several microservices in my architecture. I want to implement an API Gateway to route request to services. To achieve that, I implement spring-cloud-gateway and this is my application.yml
server:
port: 9090
spring:
application:
name: "API-GATEWAY"
cloud:
gateway:
routes:
- id: task-service
uri: 'http://localhost:8083'
predicates:
- Path=/task/**
So far everything works as expected. a request localhost:9090/task/123 is to localhost:8083/task/123. Here comes to second part.
I want some users access to only some endpoints. In my JWT token, I have role field.
{
"accountName": "erdem.ontas",
"surname": "Öntaş",
"roles": [
"ADMIN",
"USER"
],
}
I don't want specify authorization in every service separately, is there any way to specify role based access in spring-cloud-gateway? For example I want USER role to be able to access to GET http://localhost:9090/task/ but not to GET http://localhost:9090/dashboard/
If you do not want and need to create full OAuth 2 Server/Client infrastructure and want to keep it simple just create a custom GatewayFilter in which just check if the JWT token extracted from the header has the preconfigured roles.
So start with a simple GatewayFilter
#Component
public class RoleAuthGatewayFilterFactory extends
AbstractGatewayFilterFactory<RoleAuthGatewayFilterFactory.Config> {
public RoleAuthGatewayFilterFactory() {
super(Config.class);
}
#Override
public GatewayFilter apply(Config config) {
return (exchange, chain) -> {
var request = exchange.getRequest();
// JWTUtil can extract the token from the request, parse it and verify if the given role is available
if(!JWTUtil.hasRole(request, config.getRole())){
// seems we miss the auth token
var response = exchange.getResponse();
response.setStatusCode(HttpStatus.UNAUTHORIZED);
return response.setComplete();
}
return chain.filter(exchange);
};
}
#Data
public static class Config {
private String role;
}
#Override
public List<String> shortcutFieldOrder() {
// we need this to use shortcuts in the application.yml
return Arrays.asList("role");
}
}
Here we just create a simple filter which receives the required role from the config (application.yml) and checks if the request is authorized to continue.
To use the filter just add filters into you route config.
server:
port: 9090
spring:
application:
name: "API-GATEWAY"
cloud:
gateway:
routes:
- id: task-service
uri: 'http://localhost:8083'
filters:
- RoleAuth=ADMIN
predicates:
- Path=/task/**
So this way the RoleAuth filter can be reused over the several routes.
I'm trying to make a custom spring cloud stream binder but it just wont register itself:
Binder Implementation:
public class DPSBinder implements Binder<SubscribableChannel, ConsumerProperties, ProducerProperties> {
private DecisionPersistenceServiceClient dpsClient;
private MessageHandler dpsClientConsumerMessageHandler = null;
public DPSBinder(DecisionPersistenceServiceClient dpsClient) {
this.dpsClient = dpsClient;
}
#Override
public Binding<SubscribableChannel> bindConsumer(String name, String group, SubscribableChannel inboundBindTarget,
ConsumerProperties consumerProperties) {
return null;
}
#Override
public Binding<SubscribableChannel> bindProducer(String name, SubscribableChannel outboundBindTarget,
ProducerProperties producerProperties) {
switch (name) {
case "PERSIST_POST":
this.dpsClientConsumerMessageHandler = message -> dpsClient.persist((DPAPayload) message.getPayload());
break;
default:
this.dpsClientConsumerMessageHandler = null;
}
if (this.dpsClientConsumerMessageHandler != null)
this.subscribe(outboundBindTarget);
return () -> this.dpsClientConsumerMessageHandler = null;
}
public void subscribe(SubscribableChannel outboundBindTarget) {
outboundBindTarget.subscribe(this.dpsClientConsumerMessageHandler);
}}
configuration class:
#Configuration
public class DPSBinderConfiguration {
#Bean
public DPSBinder dpsBinder(DecisionPersistenceServiceClient dpsClient) {
return new DPSBinder(dpsClient);
}}
spring.binders file:
dps:something.something.DPSBinderConfiguration
application.yml
application.yml
spring:
cloud:
stream:
bindings:
input:
destination: DPP_EVENTS
group: dpp-local
binder: kafka
output:
destination: PERSIST_POST
binder: dps
binders:
kafka:
type: kafka
dps:
type: dps
I've followed the spring cloud stream guidelines for creating a custome binder but this is not working. Moreover, using the #Configuration for creating binder beans disables the kafka binder which i've added on classpath.
I found the issue. Actually #Configuration should not be used where the binding bean is being declared.
Also, there were some logical issues in My binder implementation which i fixed.
I'm developing an application in the event-driven architecture.
I'm trying to model the following flow of events:
UserAccountCreated (user-management-events) -> sending an e-mail -> MailNotificationSent (notification-service-events)
The notification-service application executes the whole flow. It waits for the UserAccountCreated event by listening to user-management-events topic. When the event is received, the application sends the email and publishes a new event - MailNotificationSent to the notification-service-events topic.
I have no problems with listening to the first event (UserAccountCreated) - application receives it and performs the rest of the flow. I also have no problem with publishing the MailNotificationSent event. Unfortunately, for development purposes, I want to listen to the MailNotificationSent event in the notification service, so the application has to listen to both UserAccountCreated and MailNotificationSent. Here I'm not able to make it works.
Let's take a look at the implementation:
NotificationStreams:
public interface NotificationStreams {
String INPUT = "notification-service-events-in";
String OUTPUT = "notification-service-events-out";
#Input(INPUT)
SubscribableChannel inboundEvents();
#Output(OUTPUT)
MessageChannel outboundEvents();
}
NotificationsEventsListener:
#Slf4j
#Component
#RequiredArgsConstructor
public class NotificationEventsListener {
#StreamListener(NotificationStreams.INPUT)
public void notificationServiceEventsIn(Flux<ActivationLinkSent> input) {
input.subscribe(event -> {
log.info("Received event ActivationLinkSent: " + event.toString());
});
}
}
UserManagementEvents:
public interface UserManagementEvents {
String INPUT = "user-management-events";
#Input(INPUT)
SubscribableChannel inboundEvents();
}
UserManagementEventsListener:
#Slf4j
#Component
#RequiredArgsConstructor
public class UserManagementEventsListener {
private final Gate gate;
#StreamListener(UserManagementEvents.INPUT)
public void userManagementEvents(Flux<UserAccountCreated> input) {
input.subscribe(event -> {
log.info("Received event UserAccountCreated: " + event.toString());
gate.dispatch(SendActivationLink.builder()
.email(event.getEmail())
.username(event.getUsername())
.build()
);
});
}
}
KafkaStreamsConfig:
#EnableBinding(value = {NotificationStreams.class, UserManagementEvents.class})
public class KafkaStreamsConfig {
}
EventPublisher:
#Slf4j
#RequiredArgsConstructor
#Component
public class EventPublisher {
private final NotificationStreams eventsStreams;
private final AvroMessageBuilder messageBuilder;
public void publish(Event event) {
MessageChannel messageChannel = eventsStreams.outboundEvents();
AvroActivationLinkSent activationLinkSent = new AvroActivationLinkSent(); activationLinkSent.setEmail(((ActivationLinkSent)event).getEmail());
activationLinkSent.setUsername(((ActivationLinkSent)event).getUsername() + "-domain");
activationLinkSent.setTimestamp(System.currentTimeMillis());
messageChannel.send(messageBuilder.buildMessage(activationLinkSent));
}
}
application config:
spring:
devtools:
restart:
enabled: true
cloud:
stream:
default:
contentType: application/*+avro
kafka:
binder:
brokers: localhost:9092
schemaRegistryClient:
endpoint: http://localhost:8990
kafka:
consumer:
group-id: notification-group
auto-offset-reset: earliest
kafka:
bootstrap:
servers: localhost:9092
The application seems to ignore the notification-service-events listener. It works when listening to only one stream.
I'm almost 100% sure that this is not an issue with publishing the event, because I've connected manually to Kafka and verified that messages are published properly:
kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic notification-service-events-out --from-beginning
Do you have any ideas what else I should check? Is there any additional configuration on the Spring side?
I've found where the problem was.
I was missing bindings configuration. In the application properties, I should have added the following lines:
cloud:
stream:
bindings:
notification-service-events-in:
destination: notification-service-events
notification-service-events-out:
destination: notification-service-events
user-management-events-in:
destination: user-management-events
In the user-management-service I didn't have such a problem because I used a different property:
cloud:
stream:
default:
contentType: application/*+avro
destination: user-management-events
My application is getting below error when consuming a service that performs queries in SQL Server using FeignClient.
ERROR:
Exception in thread "pool-10-thread-14" feign.RetryableException: Read timed out executing GET
http://127.0.0.1:8876/processoData/search/buscaProcessoPorCliente?cliente=ELEKTRO+-+TRABALHISTA&estado=SP
My Consumer Service:
#FeignClient(url="http://127.0.0.1:8876")
public interface ProcessoConsumer {
#RequestMapping(method = RequestMethod.GET, value = "/processoData/search/buscaProcessoPorCliente?cliente={cliente}&estado={estado}")
public PagedResources<ProcessoDTO> buscaProcessoClienteEstado(#PathVariable("cliente") String cliente, #PathVariable("estado") String estado);
}
My YML:
server:
port: 8874
endpoints:
restart:
enabled: true
shutdown:
enabled: true
health:
sensitive: false
eureka:
client:
serviceUrl:
defaultZone: ${vcap.services.eureka-service.credentials.uri:http://xxx.xx.xxx.xx:8764}/eureka/
instance:
preferIpAddress: true
ribbon:
eureka:
enabled: true
spring:
application:
name: MyApplication
data:
mongodb:
host: xxx.xx.xxx.xx
port: 27017
uri: mongodb://xxx.xx.xxx.xx/recortesExtrator
repositories.enabled: true
solr:
host: http://xxx.xx.xxx.xx:8983/solr
repositories.enabled: true
Anyone know how to solve this?
Thanks.
Add the following properties into application.properties file, in milliseconds.
feign.client.config.default.connectTimeout=160000000
feign.client.config.default.readTimeout=160000000
I'm using Feign.builder() to instantiate my Feign clients.
In order to set connectTimeout and readTimeout, I use the following :
Feign.builder()
...
.options(new Request.Options(connectTimeout, readTimeout))
.target(MyApiInterface.class, url);
Using this I can configure different timeout for different APIs.
just ran into this issue as well. As suggested by #spencergibb here is the workaround I'm using. See the link
Add these in the application.properties.
# Disable Hystrix timeout globally (for all services)
hystrix.command.default.execution.timeout.enabled: false
# Increase the Hystrix timeout to 60s (globally)
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds: 60000
Add this in the Java configuration class.
import feign.Request;
#Configuration
#EnableDiscoveryClient
#EnableFeignClients(basePackageClasses = { ServiceFeignClient.class })
#ComponentScan(basePackageClasses = { ServiceFeignClient.class })
public class FeignConfig {
/**
* Method to create a bean to increase the timeout value,
* It is used to overcome the Retryable exception while invoking the feign client.
* #param env,
* An {#link ConfigurableEnvironment}
* #return A {#link Request}
*/
#Bean
public static Request.Options requestOptions(ConfigurableEnvironment env) {
int ribbonReadTimeout = env.getProperty("ribbon.ReadTimeout", int.class, 70000);
int ribbonConnectionTimeout = env.getProperty("ribbon.ConnectTimeout", int.class, 60000);
return new Request.Options(ribbonConnectionTimeout, ribbonReadTimeout);
}
}
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=6000
ribbon.ReadTimeout=60000
ribbon.ConnectTimeout=60000
make sure ribbon's timeout is bigger than hystrix
You can add 'Options' argument to you methods and control timeouts dynamically.
#FeignClient(url="http://127.0.0.1:8876")
public interface ProcessoConsumer {
#RequestMapping(method = RequestMethod.GET, value = "/processoData/search/buscaProcessoPorCliente?cliente={cliente}&estado={estado}")
PagedResources<ProcessoDTO> buscaProcessoClienteEstado(#PathVariable("cliente") String cliente, #PathVariable("estado") String estado,
Request.Options options);
}
Use like next:
processoConsumer.buscaProcessoClienteEstado(..., new Request.Options(100, TimeUnit.MILLISECONDS,
100, TimeUnit.MILLISECONDS, true));
Add the below properties to the application.properties file
value 5000 is in milliseconds
feign.client.config.default.connectTimeout: 5000
feign.client.config.default.readTimeout: 5000
Look at this answer. It did the trick for me. I also did a bit of research and I've found the properties documentation here:
https://github.com/Netflix/Hystrix/wiki/Configuration#intro
eureka:
client:
eureka-server-read-timeout-seconds: 30
Add these in the application.properties
feign.hystrix.enabled=false
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=5000