I have a requirement to send a message to a RabbitMQ instance with a JPA entity after persisting/flushing it, which lead me to configure the rabbitTemplate as channelTransacted.
The consumer is external, but, to create an integration test, I have added an embedded broker (Apache QPid) and a listener to be able to check that the messages are sent.
As indicated by the documentation, I seem to have ran into a deadlock:
If we have producers and consumers in the same application, we may end up with a deadlock when producers are blocking the connection (because there are no resources on the Broker any more) and consumers cannot free them (because the connection is blocked). [...]
A separate CachingConnectionFactory is not possible for transactional producers that execute on a consumer thread, since they should reuse the Channel associated with the consumer transactions.
If I set rabbitTemplate.channelTransacted = false, the listener gets invoked fine, otherwise harness.getNextInvocationDataFor just waits until it times out.
What I'm hoping is that there's still a way to do this kind of integration test and that perhaps I configured something wrong.
I've tried with both the simple and the direct listener types, didn't make any difference:
queues:
transactions: 'transactions'
spring:
rabbitmq:
host: rabbitmq
username: guest
password: guest
dynamic: true # automatically create queues and exchanges on the RabbitMQ server
template:
routing-key: ${queues.transactions}
retry.enabled: true
# mandatory: true # interesting only for cases where a reply is expected
# publisher-confirms: true # does not work in transacted mode
publisher-returns: true # required to get notifications in case of send problems
# used for integration tests
listener:
type: direct
direct:
retry:
enabled: true
stateless: false # needed when transacted mode is enabled
max-attempts: 1 # since this is used just for integration tests, we don't want more
I'm using Spring Boot 2.1.3 with the spring-boot-starter-amqp, which pulls in spring-rabbit-2.1.4 and Apache Qpid 7.1.1 as the embedded broker for the test:
#RunWith(SpringRunner.class)
#SpringBootTest(properties = "spring.main.allow-bean-definition-overriding=true")
#AutoConfigureTestDatabase
#Transactional
#ActiveProfiles("test")
public class SalesTransactionGatewayTest {
private static final String TRANSACTIONS_LISTENER = "transactions";
#TestConfiguration
#RabbitListenerTest(spy = false, capture = true)
public static class Config {
#Bean
public SystemLauncher broker() throws Exception {
SystemLauncher broker = new SystemLauncher();
Map<String, Object> attributes = new HashMap<>();
attributes.put(SystemConfig.TYPE, "Memory");
attributes.put(SystemConfig.INITIAL_CONFIGURATION_LOCATION, "classpath:qpid-config.json");
attributes.put(SystemConfig.STARTUP_LOGGED_TO_SYSTEM_OUT, false);
broker.startup(attributes);
return broker;
}
#Bean
public Listener listener() {
return new Listener();
}
}
public static class Listener {
#RabbitListener(id = TRANSACTIONS_LISTENER, queues = "${queues.transactions}")
public void receive(SalesTransaction transaction) {
Logger.getLogger(Listener.class.getName()).log(Level.INFO, "Received tx: {0}", transaction);
}
}
#Before
public void setUp() {
// this makes the test work, setting it to `true` makes it deadlock
rabbitTemplate.setChannelTransacted(false);
}
#Test
public void shouldBeSentToGateway() throws Exception {
SalesTransaction savedTransaction = service.saveTransaction(salesTransaction);
InvocationData invocationData = this.harness.getNextInvocationDataFor(TRANSACTIONS_LISTENER, 10, TimeUnit.SECONDS);
assertNotNull(invocationData);
assertEquals(salesTransaction, invocationData.getArguments()[0]);
}
}
11:02:56.844 [SimpleAsyncTaskExecutor-1] INFO org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer - SimpleConsumer [queue=transactions, consumerTag=sgen_1 identity=16ef3497] started
Mar 25, 2019 11:02:57 AM AmqpSalesTransactionGateway send
INFO: Sending transaction: 01a92a56-c93b-4d02-af66-66ef007c2817 w/ status COMPLETED
11:02:57.961 [main] INFO org.springframework.amqp.rabbit.connection.CachingConnectionFactory - Attempting to connect to: [localhost:5672]
11:02:57.972 [main] INFO org.springframework.amqp.rabbit.connection.CachingConnectionFactory - Created new connection: rabbitConnectionFactory.publisher#6d543ec2:0/SimpleConnection#79dd79eb [delegate=amqp://guest#127.0.0.1:5672/, localPort= 56501]
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:712)
at org.junit.Assert.assertNotNull(Assert.java:722)
Related
I am trying to develop a small Springboot application (using JHipster) that listens to a RabbitMQ queue, acknowledges the messages and then runs some operations on a hibernate repository. For now I can consume messages with no problem. But when I try to use the hibernate repository when receiving a message, it appears the TransactionManagers get mixed up and I get the Exception:
Caused by: org.hibernate.TransactionException: Unable to commit against JDBC Connection
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.commit(AbstractLogicalConnectionImplementor.java:92)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:282)
...
Caused by: org.postgresql.util.PSQLException: Cannot commit when autoCommit is enabled.
at org.postgresql.jdbc.PgConnection.commit(PgConnection.java:872)
My Rabbit configuration is simplistic:
#Configuration
public class RabbitExchangeConfiguration {
#Bean
Queue queue() {
return new Queue("myQueue", false);
}
}
Then there is my TestService, with a RabbitListener (listen) and a method that uses my database transaction manager to access the repository (test)
#Service
public class TestService {
public TestService(
MyRepository myRepository
) {
this.myRepository= myRepository;
}
// My Listener to receive the messages
#RabbitListener(queues = "datafactory")
public void listen(Message in) {
String messageBody = new String(in.getBody());
test(messageBody);
}
// the method that should do stuff with the repository
#Transactional(transactionManager = "myDatabaseTransactionManager")
public void test(String content) {
myRepository.findById(content).orElse(null);
..
}
}
Right now I don't care what the message contains, nor is it important if the entity can be found.
When I comment out the call to test(content) then the message is received (I can see it in the logs). But when I try to call the function, then I get the Exception.
Is there any way to end the RabbitMQ-transaction before entering the hibernate transaction context? Why is the listener trying to use the hibernate transaction?
solved
Thanks to the answer of p3consulting I found that jpa.properties.hibernate.connection.provider_disables_autocommit=true was set by default. It was configuered automatically by JHipster for me. I just had to make sure that autocommit was disabled for good:
spring.datasource.hikari.auto-commit=false // disables auto commit
spring.jpa.properties.hibernate.connection.provider_disables_autocommit=false // makes sure it stays that way
So nothing to do with RabbitMQ...
The message is clear: you left autocommit to TRUE then you can't commit explicitly because it's done for you after each DML statement, you could turn it off with hibernate.connection.autocommit=false in your properties or calling connection.setAutoCommit(false);
I'm trying to write a basic controller test in a micronaut (3.2.7) application. When I run it, it fails to start as it wants to create DB related beans too. micronaut-hibernate-jpa, flyway, etc. are in the pom.xml.
Can I configure the context somehow so it doesn't pick up hikaripool,flyway, and jpa realted beans?
11:46:23.820 [main] INFO i.m.context.env.DefaultEnvironment - Established active environments: [test]
11:46:24.112 [main] WARN i.m.c.h.j.JpaConfiguration$EntityScanConfiguration - Runtime classpath scanning is no longer supported. Use #Introspected to declare the packages you want to index at build time. Example #Introspected(packages="foo.bar", includedAnnotations=Entity.class)
11:46:24.133 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
11:46:25.197 [main] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Exception during pool initialization.
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "postgres"
The code:
class HelloTest {
private static EmbeddedServer server;
private static HttpClient client;
#BeforeAll
public static void setupServer() {
server = ApplicationContext.run(EmbeddedServer.class);
client = server
.getApplicationContext()
.createBean(HttpClient.class, server.getURL());
}
#AfterAll
public static void stopServer() {
if (server != null) {
server.stop();
}
if (client != null) {
client.stop();
}
}
#Test
void testHelloWorldResponse() {
...
}
}
I tried to exclude configurations like this, but with no luck:
server = ApplicationContext.builder("test")
.exclude("io.micronaut.configuration.hibernate.jpa","io.micronaut.configuration.jdbc.hikari")
.run(EmbeddedServer.class);
Note: If I remove everything from application.yml then the test works. It looks like that in tests the default properties are resolved which turns on jpa,metrics, etc. So I guess the test needs to ignore the default settings too somehow.
You can override all of your (default) application.yml with (test-)environment specific property files: https://docs.micronaut.io/latest/guide/index.html#_included_propertysource_loaders
So you can just provide a dedicated application-mycustomtest.yml as part of your test resources, in which you override all default settings.
Then you can specify as part of the test, which environments shall be active:
#MicronautTest(environments={"mycustomtest"})
Asked the micronaut team on gitter and currenlty the only option is not having a default configuration and having multiple configuration files for controller, repo and e2e testing.
If you prefer using configuration properties to configured all #FeignClient, you can create configuration properties with default feign name. It's also possible to set these timeouts per specific client by naming the client. And, we could, of course, list a global setting and also per-client overrides together without a problem.
My client:
#FeignClient( contextId = "fooFeignClient", name = "foo-client", url = "${foo.url}",
fallbackFactory = FooFallBackFactory.class,
configuration = FooFeignConfiguration.class)
I'm trying to do this and I want foo-client.readTimeout to override default.readTimeout when I'm using foo-client:
feign:
hystrix:
enabled: true
client:
config:
default:
connectTimeout: 3000
readTimeout: 3000
loggerLevel: full
foo-client:
connectTimeout: 3000
readTimeout: 5000
hystrix:
command:
default:
execution:
timeout:
enabled: "false"
isolation:
strategy: "THREAD"
thread:
timeoutInMilliseconds: "3000"
But this is not happening. I'm not sure if Hystrix's timeoutInMilliseconds could be impacting, but from my tests, it's not interfering. I want the feign.readTimeout to be 5000 only for the foo-client, not for the other clients. But it looks like it is ignoring this foo-client configuration and using only the default configuration.
I know it's not a #Configuration class problem because the Feign documentation says that if we create both #Configuration bean and configuration properties, configuration properties will win.
As you have seen in the documentation, you can set timeouts per specific client. But you have only configured it for feign client, not for hystrix command. Remember that they have independent timeouts.
I would recommend to use a configuration class instead of editing your application.yml in order to handle multiples feign clients with hystrix. The following example sets up a Feign.Builder where timeouts are being injected for Feign client and Hystrix command.
# application.yml
microservices:
foo:
feign:
url: xxxx
connect-timeout: 5s
read-timeout: 5s
hystrix:
enabled: true
timeout: 5s
#FeignClient(name = "foo",
url = "${microservice.foo.feign.url}",
configuration = FooFeignConfiguration.class)
public interface FooFeignRepository {
}
#Configuration
#RequiredArgsConstructor
public class FooFeignConfiguration {
#Value("${microservice.foo.hystrix.enabled:true}")
private final Boolean hystrixEnabled;
#DurationUnit(value = ChronoUnit.MILLIS)
#Value("${microservice.foo.feign.connect-timeout:5000}")
private final Duration feignConnectTimeout;
#DurationUnit(value = ChronoUnit.MILLIS)
#Value("${microservice.foo.feign.read-timeout:5000}")
private final Duration feignReadTimeout;
#DurationUnit(value = ChronoUnit.MILLIS)
#Value("${microservice.foo.hystrix.timeout:5000}")
private final Duration hystrixTimeout;
#Bean
#Scope("prototype")
public Feign.Builder feignBuilder() {
return (hystrixEnabled ?
HystrixFeign.builder()
.options(new Request.Options(
(int) feignConnectTimeout.toMillis(),
(int) feignReadTimeout.toMillis()))
.setterFactory((target, method) -> {
return new SetterFactory.Default()
.create(target, method)
.andCommandPropertiesDefaults(HystrixCommandProperties.Setter()
.withExecutionTimeoutInMilliseconds((int) hystrixTimeout.toMillis()));
}):
Feign.builder())
.retryer(Retryer.NEVER_RETRY);
}
}
I wrote simple sample to read text from console and send it to the rabbitMq server:
#Configuration
#EnableIntegration
#IntegrationComponentScan
public class Config {
#Autowired
private AmqpTemplate amqpTemplate;
#Bean
public IntegrationFlow fromConsoleToRabbitFlow() {
return IntegrationFlows.from(consoleSource(), c -> c.id("consoleInput")
.poller(Pollers.fixedRate(1000))
.autoStartup(true)
).channel("consoleOutputChannel")
.handle(Amqp.outboundAdapter(amqpTemplate).routingKey("my_spring_integration_queue"))
.get();
}
public MessageSource<String> consoleSource() {
return CharacterStreamReadingMessageSource.stdin();
}
}
It looks like almost working solution but I can't find my_spring_integration_queue in rabbitmq admin console:
But I can't find anything related to 'my_spring_integration_queue' on other tab. Where can I find it ?
I expect that application will create queue if it doesn't exist. I was not able to find a method for send into the queur so I used .routingKey method. I also tried .exchangeName method but it lead to the:
32019-08-27 13:26:15.972 ERROR 16372 --- [ 127.0.0.1:5672] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'my_spring_integration_queue' in vhost '/', class-id=60, method-id=40)
P.S.
the Queue tab looks like this:
You either need to add the queue manually or use a RabbitAdmin #Bean to automatically declare it for you - the admin will find all beans of type Queue and declare them.
If you are using Spring Boot, it will automatically configure an admin bean for you so you just need the Queue #Bean.
See Configuring the Broker.
I am new to spring boot and am trying to write a consumer using spring boot
Below is my snippet:
application.properties
spring.activemq.broker-url=tcp://localhost:8161
spring.activemq.user=admin
spring.activemq.password=admin
SampleActiveMQApplication.java
#SpringBootApplication
#EnableJms
public class SampleActiveMQApplication {
public static void main(String[] args) {
SpringApplication.run(MicroserviceAddPayeeApplication.class, args);
}
}
Consumer.java
#Component
public class Consumer {
#JmsListener(destination = "queue/msgQueue")
public void receiveQueue(String text) {
System.out.println("inside consumer");
System.out.println(text);
}
}
I am getting the below error
2018-04-23 07:08:08.277 WARN 9196 --- [enerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'queue/msgQueue' - trying to recover. Cause: Disposed due to prior exception
2018-04-23 07:08:08.292 ERROR 9196 --- [enerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'queue/msgQueue' - retrying using FixedBackOff{interval=5000, currentAttempts=0, maxAttempts=unlimited}. Cause: Cannot send, channel has already failed: tcp://127.0.0.1:8161
Please assist:
Are you sure that your broker listens on your specified port 8161? By default, Active-mq admin management consoles listens on port 8161 which is accessible over the browser, and the broker listens on 61616. You may need to confirm that. You can check out for more information here