Develop Reference

How to clear topics in tests with Spring Kafka

I'm writing a unit test with Spring Kafka 2.4 to prove that my Spring Boot setup is correct. I'm validating that SeekToCurrentBatchErrorHandler works as expected which requires sending an incorrect message that should be retried. Unfortunately this incorrect messages breaks other tests because this message will be retried forever.
Because of above I'd like to ensure that each test is correctly isolated. I either need to:
Delete and recreate the Kafka topic with AdminClient
Seek to the end of the existing Kafka topic and commit new offsets
I was trying option 2 with Consumer.seekToEnd() method however Spring Kafka hides the created consumers behind few layers of internal framework classes. I'm also not 100% sure if this method can be called in test thread which is different from listener thread.
What is the recommended way to clear topics in tests with Spring Kafka?

Best practice is to use unique topic names in each test to provide complete isolation; you could also stop the container(s), create a new Consumer with the same group.id and perform the seeks there.

how to make persistent JMS messages with java spring boot application?

I am trying to make a queue with activemq and spring boot using this link and it looks fine. What I am unable to do is to make this queue persistent after application goes down. I think that SimpleJmsListenerContainerFactory should be durable to achieve that but when I set factory.setSubscriptionDurable(true) and factory.setClientId("someid") I am unable to receive messages any more. I would be greatfull for any suggestions.

I guess you are embedding the broker in your application. While this is ok for integration tests and proof of concepts, you should consider having a broker somewhere in your infrastructure and connect to it. If you choose that, refer to the ActiveMQ documentation and you should be fine.
If you insist on embedding it, you need to provide a brokerUrl that enables message persistence.
Having said that, it looks like you misunderstand durable subscriber and message persistence. The latter can be achieved by having a broker that actually stores the content of the queue somewhere so that if the broker is stopped and restarted, it can restore the content of its queue. The former is to be able to receive a message even if the listener is not active at a period of time.

you can enable persistence of messages using ActiveMQConnectionFactory.
as mentioned in the spring boot link you provided, this ActiveMQConnectionFactory gets created automatically by spring boot.so you can have this bean in your application configuration created manually and you can set various property as well.
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=true");
Here is the link http://activemq.apache.org/how-do-i-embed-a-broker-inside-a-connection.html

Is Apache Kafka able to handle transactions?

we plan to use Kafka as a central component in our data warehouse given that the producer is able to handle transactions (in short: rollbacks and commits).
When googling Kafka + Transactions I find a lot of theoretical thoughts about the possibility of how Kafka could handle transactions but at the moment I do not see any function in the java API that supports commits and rollbacks for the producer.
Has anybody made some experiences with transactions and Kafka and can give me an hint?

I think what you are looking for is basically called transactional messaging in Kafka where producers are capable of creating session (aka transactional session) and send messages within the sessions. Hence it can choose to either commit / abort the transaction.
[Source]: Please read the wiki for details

Actually, from the last version 0.11.0.0 transactions are supported. See Guarantee unique global transaction for Kafka Producers

No; Kafka does not support transactions.
You can get certainty that a message has been produced to a partition, but once produced you are not able to rollback that message.
Since version 0.11.0 Apache Kafka supports transactions: https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka

Spring - JMS - DB

My questions is as follows:
I have a service, which queries the DB and retrieves some records. Then it updates an external system using that information and in the end in updates back the DB.
Using spring trascations and weblogic jta transaction manager i was able with sample code below not to loose any messages in case
No records are retrived. (these are mandatory for the External System)
External System Error
Failed to update the DB
So in all aboce cases the JMS Listener puts the errror message back to the queue.
My question, is there any better way, using Spring with all its goodies, to manage that? The sample code below throws explicilty a RuntimeException which i dont think is a good design...
Please for your comments.
EDIT:
The queue is being polled by the submissionListener MDP that its configuration is shown below. After the message is consumed it invokes the registerDocument() of the service. (another Spring bean). That service invokes 2 times the DAO and the external system.

Check out Spring's documentation on JmsTemplate and Message Driven POJOs for patterns in the core Spring framework.
Spring Integration models higher-level abstractions related to message-orientated patterns.

Spring synchronising Hibernate and JMS transactions

I am working on a stand-alone application that uses both JMS and Hibernate.
The documentation suggests JTA has to be used if I want to have transactions across both resources.
However, right now with a #Transaction annotated DAO method (and HibernateTransactionManager), this already seems to work. When I call send() on the JmsTemplate, the message is not immediately sent, but rather the JMS session is committed with the Hibernate session as the method returns.
I didn't know how this is possible without the JtaTransactionManager, so I checked the source code. It turns out both the wrapper for Hibernate and JmsTemplate registers the sessions with TransactionSynchronizationManager and the JMS session will be committed when the Hibernate session commits.
What's the different between this and a JTA transaction. Can I use this to replace the latter??

In short no, you can't get support for 2-phase commit without a JTATransactionManager and XA aware datasources.
What you are witnessing is a co-ordination of two Local Transactions supporting 1-phase commit only. Roughly performing this sequence of events...
Start JMS Transaction
Read JMS message
Start JDBC Transaction
Write to database
Commit JDBC Transaction
Commit/Acknowledge JMS
The JMS transaction will be started first wrapping the nested JDBC transaction, so that the JMS queue will rollback if the Hibernate/JDBC commit fails. Your JMS Listener Container should be setup not to acknowledge="auto" and instead wait for the Hibernate transaction to complete before sending the acknowledgement.
If you only have these two resources then the issue you will have to consider is when Hibernate succeeds in persiting then you get an Exception before you can acknowledge the JMS server. Not a big issue as the JMS message is not lost and you will read it again.
However
You must write your MessageListener to handle duplicate messages from the server
You must also handle a message that cannot be processed due to bad data and ending up in an infinite loop of trying to comsume it. In this case the server may be configured to move the message to a "dead message queue", or you deal with this yourself in the MessageListener
Other options and further reading
If your JMS server does not support XA (global) transactions this is pretty much your only solution.
If JMS server does support XA transactions but JDBC doesn't then you can use a JTATransactionManager and use the LastResourceCommitOptimisation. There are open source JTATransactionManagers you can use like JOTM
This JavaWorld article goes into more detail on your problem space.

Although this has been answered in detail by Brad, i would like to address a very specific part of your query:-
I didn't know how this is possible without the JtaTransactionManager
From the spring documentation:-
When a JTA environment is detected, Spring’s JtaTransactionManager will be used to manage transactions
https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-jta.html