I'm working on an application which uses Kafka to consume messages from multiple topics, persisting data as it goes.
To that end I use a #Service class, with a couple of methods annotated with #kafkaListener. Consider this:
#Transactional
#KafkaListener(topics = MyFirstMessage.TOPIC, autoStartup = "false", containerFactory = "myFirstKafkaListenerContainerFactory")
public void handleMyFirstMessage(ConsumerRecord<String, MyFirstMessage> record, Acknowledgment acknowledgment) throws Exception {
MyFirstMessage message = consume(record, acknowledgment);
try {
doHandle(record.key(), message);
} catch (Exception e) {
TransactionInterceptor.currentTransactionStatus().setRollbackOnly();
} finally {
acknowledgment.acknowledge();
}
}
#Transactional
#KafkaListener(topics = MySecondMessage.TOPIC, autoStartup = "false", containerFactory = "mySecondKafkaListenerContainerFactory")
public void handleMySecondMessage(ConsumerRecord<String, MySecondMessage> record, Acknowledgment acknowledgment) throws Exception {
MySecondMessage message = consume(record, acknowledgment);
try {
doHandle(record.key(), message);
} catch (Exception e) {
TransactionInterceptor.currentTransactionStatus().setRollbackOnly();
} finally {
acknowledgment.acknowledge();
}
}
Please disregard the stuff about setRollbackOnly, it's not relevant to this question.
What IS relevant is that the doHandle() methods in each listener perform inserts in a table, which occasionally fail because autogenerated keys turn out to be non-unique once the final commit is done.
What happens is that each doHandle() method will increment the key column in their own little transactions, and only one of them will "win" that race. The other will fail during commit, with a non-unique constraint violation.
What is best practice to handle this? How do I "synchronize" transactions to execute like pearls on a string in stead of all at once?
I'm thinking of using some kind of semaphor or lock, to serialize things but that smells like a solution with many pitfalls. If there was a general pattern or framework to help with this problem I would be much more comfortable implementing it.
See the documentation.
Using #Transactional for the DB and a KafkaTransactionManager in the listener container is similar to using a ChainedKafkaTransactionManager (configured with both TMs) in the container. The DB tx is committed, followed by Kafka, when the listener exits normall.
When the listener throws an exception, both transactions are rolled back in the same order.
The setRollbackOnly is definitely relevant to this question since you are not rolling back the kafka transaction when you do that.
Related
Spring managed KafkaTemplate provides
template.send(record).addCallback(...
template.executeInTransaction(...
Now let's say I have a method doWork() which is triggered on a event (say a TCP/IP message).
#Autowired
KafkaTemplate template;
// This method is triggered on a event
doWork(EventType event){
switch(event){
case Events.Type1 :
template.send(record); break;
case Events.Type2 :
// Question : How do I achieve a commit of all my previous sends here?
default : break;
}
}
Basically, I need to achieve a transaction by adding #Transaction over doWork() or a
template.executeInTransaction(...
in code. But I want to batch a couple of [template.send()]s and do a commit after a couple of calls to the doWork() method, how do I achieve that?
My producer configurations has transactions enabled and a KafkaTransactionManager wired to the producer factory.
kafkaTemplate.executeInTransaction(t -> {
boolean stayIntransaction = true;
while (stayInTransaction) {
Event event = readTcp()
doWork(event);
stayInTransaction = transactionDone(event);
}
}
As long as the doWork() method uses the same template, and it runs within the scope of the callback, the work will run in the transaction.
Or
#Transactional
public void doIt() {
boolean stayIntransaction = true;
while (stayInTransaction) {
Event event = readTcp()
doWork(event);
stayInTransaction = transactionDone(event);
}
}
When using declarative transactions.
If the TCP events are async, you will somehow need to hand them off to the thread running the transaction, such as using a BlockingQueue<?>.
I am trying to use the new feature of KCL library in Java for AWS Kinesis to do a graceful shutdown by registering with shutdown hook to stop all the record processors and then the worker gracefully. The new library provides a new interface which record processors needs to be implemented. But how does it get invoked?
Tried invoking first the worker.requestShutdown() then worker.shutdown() and it works. But is it any intended way to use it. What is the use then to use both, and its benefit?
Starting a consumer
As you might know that when you create a Worker, it
1) creates the consumer offset table in dynamodb
2) create leases, schedule lease taker and lease renewer at configured interval of time
If you have two partitions, then there will be two records in your same dynamodb table, meaning partition needs a lease.
eg.
{
"checkpoint": "TRIM_HORIZON",
"checkpointSubSequenceNumber": 0,
"leaseCounter": 38,
"leaseKey": "shardId-000000000000",
"leaseOwner": "ComponentTest_Consumer_With_Two_Partitions_Consumer_192.168.1.83",
"ownerSwitchesSinceCheckpoint": 0
}
{
"checkpoint": "49570828493343584144205257440727957974505808096533676050",
"checkpointSubSequenceNumber": 0,
"leaseCounter": 40,
"leaseKey": "shardId-000000000001",
"leaseOwner": "ComponentTest_Consumer_With_Two_Partitions_Consumer_192.168.1.83",
"ownerSwitchesSinceCheckpoint": 0
}
schedule for taking and renewing lease is taken care by Lease Coordinator ScheduledExecutorService (called leaseCoordinatorThreadPool)
3) Then for each partition in the stream, Worker creates an internal PartitionConsumer, which actually fetches the events, and dispatches to your RecordProcessor#processRecords. see ProcessTask#call
4) on your question, you have to register your IRecordProcessorFactory impl to the worker, which will give one ProcessorFactoryImpl to each PartitionConsumer.
eg. see example here, which might be helpful
KinesisClientLibConfiguration streamConfig = new KinesisClientLibConfiguration(
"consumerName", "streamName", getAuthProfileCredentials(), "consumerName-" + "consumerInstanceId")
.withKinesisClientConfig(getHttpConfiguration())
.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); // "TRIM_HORIZON" = from the tip of the stream
Worker consumerWorker = new Worker.Builder()
.recordProcessorFactory(new DavidsEventProcessorFactory())
.config(streamConfig)
.dynamoDBClient(new DynamoDB(new AmazonDynamoDBClient(getAuthProfileCredentials(), getHttpConfiguration())))
.build();
public class DavidsEventProcessorFactory implements IRecordProcessorFactory {
private Logger logger = LogManager.getLogger(DavidsEventProcessorFactory.class);
#Override
public IRecordProcessor createProcessor() {
logger.info("Creating an EventProcessor.");
return new DavidsEventPartitionProcessor();
}
}
class DavidsEventPartitionProcessor implements IRecordProcessor {
private Logger logger = LogManager.getLogger(DavidsEventPartitionProcessor.class);
//TODO add consumername ?
private String partitionId;
private ShutdownReason RE_PARTITIONING = ShutdownReason.TERMINATE;
public KinesisEventPartitionProcessor() {
}
#Override
public void initialize(InitializationInput initializationInput) {
this.partitionId = initializationInput.getShardId();
logger.info("Initialised partition {} for streaming.", partitionId);
}
#Override
public void processRecords(ProcessRecordsInput recordsInput) {
recordsInput.getRecords().forEach(nativeEvent -> {
String eventPayload = new String(nativeEvent.getData().array());
logger.info("Processing an event {} : {}" , nativeEvent.getSequenceNumber(), eventPayload);
//update offset after configured amount of retries
try {
recordsInput.getCheckpointer().checkpoint();
logger.debug("Persisted the consumer offset to {} for partition {}",
nativeEvent.getSequenceNumber(), partitionId);
} catch (InvalidStateException e) {
logger.error("Cannot update consumer offset to the DynamoDB table.", e);
e.printStackTrace();
} catch (ShutdownException e) {
logger.error("Consumer Shutting down", e);
e.printStackTrace();
}
});
}
#Override
public void shutdown(ShutdownInput shutdownReason) {
logger.debug("Shutting down event processor for {}", partitionId);
if(shutdownReason.getShutdownReason() == RE_PARTITIONING) {
try {
shutdownReason.getCheckpointer().checkpoint();
} catch (InvalidStateException e) {
logger.error("Cannot update consumer offset to the DynamoDB table.", e);
e.printStackTrace();
} catch (ShutdownException e) {
logger.error("Consumer Shutting down", e);
e.printStackTrace();
}
}
}
}
// then start a consumer
consumerWorker.run();
Stopping a consumer
Now, when you want to stop your Consumer instance(Worker), you don't need to deal much with each PartitionConsumer, which will be taken care by Worker once you ask it to shut down.
with shutdown, it asks the leaseCoordinatorThreadPool to stop, which was responsible for renewing and taking leases, and awaits for termination.
requestShutdown on the other hand cancels the lease taker, AND notifies the PartitionConsumers about the shutdown.
And more important thing with requestShutdown is if you want to get notified on your RecordProcessor then you can implement IShutdownNotificationAware as well. That way in case of race condition when your RecordProcessor is processing an event but worker is about to shut down, you should still be able to commit your offset and then shutdown.
requestShutdown returns a ShutdownFuture, which then calls back worker.shutdown
You will have to implement following method on your RecordProcessor to get notified on requestShutdown,
class DavidsEventPartitionProcessor implements IRecordProcessor, IShutdownNotificationAware {
private String partitionId;
// few implementations
#Override
public void shutdownRequested(IRecordProcessorCheckpointer checkpointer) {
logger.debug("Shutdown requested for {}", partitionId);
}
}
But but if you loose the lease before notifying then it might not be called.
Summary to your questions
The new library provides a new interface which record processors needs
to be implemented. But how does it get invoked?
implement a IRecordProcessorFactory and IRecordProcessor.
then wire your RecordProcessorFactory to your Worker.
Tried invoking first the worker.requestShutdown() then
worker.shutdown() and it works. But is it any intended way to use it?
You should use requestShutdown() for graceful shutdown, which will take care of race-condition. It was introduced in kinesis-client-1.7.1
I am writing an integration test in JUnit for a Message Driven Pojo (MDP):
#JmsListener(destination = "jms/Queue", containerFactory = "cf")
public void processMessage(TextMessage message) throws JMSException {
repo.save(new Entity("ID"));
}
where repo is a spring-data repository
my unit test:
#Test
public void test() {
//sendMsg
sendJMSMessage();
//verify DB state
Entity e = repo.findOne("ID");
assertThat(e, is(notNullValue()) );
}
Now, the thing is that the processMessage() method is executed in a different thread than the test() method, so I figured out that I need to somehow wait for the processMessage() method to complete before verifying the state of the DB. The best solution I could find was based on CountDownLatch. so now the methods look like this:
#JmsListener(destination = "jms/Queue", containerFactory = "cf")
public void processMessage(TextMessage message) throws JMSException {
repo.save(new Entity("ID"));
latch.countDown();
}
and the test
#Test
public void test() {
//set the countdownlatch
CountDownLatch latch = new CountDownLatch(1);
JMSProcessor.setLatch(latch);
//sendMsg
sendJMSMessage();
try {
countDownLatch.await();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
//verify DB state
Entity e = repo.findOne("ID");
assertThat(e, is(notNullValue()) );
}
So I was very proud of myself and then I run the test and it failed. The repo.findOne("ID") returned null. In the first reaction I set up a breakpoint at that line and proceed with debugging. During the debugging session the repo.findOne("ID") actually returned the entity inserted by the #JMSListenerlistener method.
After scratching my head for a while here's the current theory: Since the spring-data repository is accessed in two different threads, it gets two different instances of EntityManager and therefore the two threads are in a differen't transaction. Eventhough there's some sort of synchronization using the CountDownLatch, the transaction bound to the thread executing the #JMSListener annotated method has not committed yet when the JUnit #Test annotated method starts a new transaction and tries to retrieve the entity.
So my question is:
Is there a way for one thread to wait for the commit of the other.
Can two threads share one transaction in such a synchronized context (ie, the two threads would not access the EntityManager simultaneously)
Is my testing approach a nonsense and there is a better way of doing this
I am using Spring's message-driven POJO framework (and DefaultMessageListenerContainer in particular) to listen to several queues and topics.
In the case of one particularly queue, there is a need to slow the rate at which I drain the queue, on the order of one message every five minutes. The actual processing of the messages is a sub-second operation, but I would like the listener to sit idle for some time in between messages.
I have created a bit of a hack, but it is decidedly sub-optimal: What I've done is to set the max concurrency to 1 and add a Thread.sleep(..) after processing each message. I would like to find a way instead to use the DefaultMessageListenerContainer to wait between attempts to receive, rather than causing the handler to do the waiting during the would-be processing of a message.
I had considered if there was a ScheduledExecutor that would help, but I realize that the throttling would need to be done where the tasks are produced. Is there perhaps some method from DefaultMessageListenerContainer that I could override to accomplish what I'm after?
Depending on the provider of the queue, you may be able to set a max rate for consumers that consume it's queues.
For example in hornetQ you set this in the connection factory using consumer-max-rate.
An alternative to modifying the behavior of your consumer would be to make use of Apache Camel to delay the messages on that one specific queue.
http://camel.apache.org/delayer.html describes the functionality of the Camel Delayer pattern. So for example:
<route>
<from uri="jms:YOURQUEUE"/>
<delay>
<constant>1000</constant>
</delay>
<to uri="jms:DELAYEDQUEUE"/>
</route>
Where you would then consume the DELAYEDQUEUE and all messages would be delayed by 1 second.
I'm not sure for 100%, but believe that receiveTimeout is what you want.
<bean id="blahContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
....
<!-- 300000 = 5 * 60 * 1000 == 5 min -->
<property name="receiveTimeout" value="300000"/>
</bean>
receiveTimeout accepts timeout in milliseconds, you can read more about it in javadocs
Here's a solution that extends DefaultMessageListenerContainer to provide the throttling functionality. The advantage of this approach is that Thread.sleep() is not being called within onMessage(). This would hold a Transaction open for longer than necessary if a Transaction is in play (as configured in this example below). The call to Thread.sleep() occurs after the transaction has been committed. A limitation to implementing this throttling feature is that we can only support one consumer thread, hence the name ThrottlingSingleConsumerMessageListenerContainer.
#Configuration
#EnableJms
#EnableTransactionManagement
public class Config
{
private static final long THROTTLE_FIVE_SECONDS = 5_000;
#Bean
public DefaultMessageListenerContainer defaultMessageListenerContainer(
ConnectionFactory connectionFactory,
PlatformTransactionManager transactionManager,
MyJmsListener myJmsListner)
{
DefaultMessageListenerContainer dmlc = new ThrottlingSingleConsumerMessageListenerContainer(THROTTLE_FIVE_SECONDS);
dmlc.setConnectionFactory(connectionFactory);
dmlc.setSessionAcknowledgeMode(Session.SESSION_TRANSACTED);
dmlc.setSessionTransacted(true);
dmlc.setTransactionManager(transactionManager);
dmlc.setDestinationName("QUEUE.IN");
dmlc.setMessageListener(myJmsListner);
return dmlc;
}
}
#Component
public class MyJmsListener implements MessageListener
{
#Override
public void onMessage(Message message)
{
// process the message
}
}
public class ThrottlingSingleConsumerMessageListenerContainer extends DefaultMessageListenerContainer
{
private static final Logger log = LoggerFactory.getLogger(ThrottlingSingleConsumerMessageListenerContainer.class);
private final long delayMillis;
public ThrottlingSingleConsumerMessageListenerContainer(long delayMillis)
{
this.delayMillis = delayMillis;
super.setMaxConcurrentConsumers(1);
}
#Override
protected boolean receiveAndExecute(Object invoker, #Nullable Session session, #Nullable MessageConsumer consumer) throws JMSException
{
boolean messageReceived = super.receiveAndExecute(invoker, session, consumer);
if (messageReceived) {
log.info("Sleeping for {} millis", delayMillis);
try {
Thread.sleep(delayMillis);
} catch (InterruptedException e) {
log.warn("Sleeping thread has been interrupted");
Thread.currentThread().interrupt();
}
}
return messageReceived;
}
#Override
public void setMaxConcurrentConsumers(int maxConcurrentConsumers)
{
super.setMaxConcurrentConsumers(maxConcurrentConsumers);
Assert.isTrue(getMaxConcurrentConsumers() <= 1, "Throttling does not support maxConcurrentConsumers > 1");
}
#Override
public void setConcurrency(String concurrency)
{
super.setConcurrency(concurrency);
Assert.isTrue(getMaxConcurrentConsumers() <= 1, "Throttling does not support maxConcurrentConsumers > 1");
}
}
This has been tested on org.springframework 5.x but should run on earlier versions also.
I have a onMessage method where I'm reciving an ObjectMessage from the Queue and using that information to populate and persist a JPA entity object. But when something goes wrong while persisting the entity object it is re-executing the onMessage(). My guess is it is pushing the ObjectMessage back the queue and hence the onmessage is getting executed again. This way I'm entering an infinite loop. How can stop onMessage() to get execute again or control the no of times it gets executed. Here is the code I have.
Error is happening at saveAuditData(auditInfo).
public void onMessage(Message inMessage) {
log.debug("Entering onMessage() Method.");
AuditInfo auditInfo = null;
try {
ObjectMessage om = (ObjectMessage) inMessage;
auditInfo = (AuditInfo) om.getObject();
log.debug("Message received : " + auditInfo.getApiUsed());
log.debug("Calling saveAuditData().");
saveAuditData(auditInfo);
log.debug("Leaving onMessage() Method.");
}
catch (Exception e) {
e.printStackTrace();
log.debug("Error persisting Audit Info.",e);
log.debug("Printing Audit Info:");
log.debug(auditInfo.toString());
}
}
private void saveAuditData(AuditInfo auditInfo) {
log.debug("Entering saveAuditData() Method.");
log.debug("Populating Audit Object.");
IdmAudit idmAudit = new IdmAudit();
idmAudit.setApiUsed("API");
idmAudit.setAppClientIpAddress("localhost");
idmAudit.setAuditActivity("activity1");
idmAudit.setAuditData(auditInfo.getAuditData());
idmAudit.setAuditGroup(AUDIT_GROUP);
idmAudit.setAuditType("Type");
idmAudit.setIdmAuditCreationDate(new Date());
idmAudit.setLocationCd("Location");
idmAudit.setPurgeDate(null);
idmAudit.setSubscriberId(new BigDecimal(0));
idmAudit.setSuccessInd("Y");
idmAudit.setUserId(new BigDecimal(0));
idmAudit.setAuditSource("Source");
idmAudit.setVersionNumber(new BigDecimal(0));
log.debug("Saving Audit.");
entityManager.persist(idmAudit);
entityManager.flush();
log.debug("Leaving saveAuditData() Method.");
}
When a container-managed transaction is started by the container to process a JMS message, any failure in JDBC connections or exception thrown in the thread will result into a rollback of the global XA transaction. So the message goes back to the queue and will be retry later according to the queue configuration: period between retries, maximum number of retry before moving the message to a dead-letter queue.
So you have the following options:
Choose "Bean managed" transaction mode in your MDB deployment descriptor and use UserTransaction from lookup to java:comp/UserTransaction to call begin, commit or rollback manually, so care your exception handling.
Keep "Container managed" transaction but query the redelivery count property on the JMS message to decide what to do next: either try again something that can fail or either skip this step and save your data in database. You can get redelivery info on your message from Message.getJMSRedelivered() or Message.getLongProperty("JMSXDeliveryCount") if your JMS provider delivers it.
Or else, move your saveAuditData method to a EJB StatelessBean with transaction support RequiresNew in deployment descriptor so that a new transaction is created and your data is saved whatever happens to your MDB transaction. This option can be combined with the previous one.
You can simply mark the onMessage method with the TransactionType annotation:
#TransactionAttribute(value=TransactionAttributeType.REQUIRES_NEW)
public void onMessage(Message message) {
.....
}