I've been trying to get pubsub to work within a spring application. To get up and running I've been reading through tutorials and documentation like this
I can get things to build and start but if I go through cloud console to send a message to the test subscription it never arrives.
This is what my code looks like right now:
#Configuration
#Import({GcpPubSubAutoConfiguration.class})
public class PubSubConfigurator {
#Bean
public GcpProjectIdProvider projectIdProvider(){
return () -> "project-id";
}
#Bean
public CredentialsProvider credentialsProvider(){
return GoogleCredentials::getApplicationDefault;
}
#Bean
public MessageChannel inputMessageChannel() {
return new PublishSubscribeChannel();
}
#Bean
#InboundChannelAdapter(channel = "inputMessageChannel", poller = #Poller(fixedDelay = "5"))
public MessageSource<Object> pubsubAdapter(PubSubTemplate pubSubTemplate) {
PubSubMessageSource messageSource = new PubSubMessageSource(pubSubTemplate, "tst-sandbox");
messageSource.setAckMode(AckMode.MANUAL);
messageSource.setPayloadType(String.class);
messageSource.setBlockOnPull(false);
messageSource.setMaxFetchSize(10);
//pubSubTemplate.pull("tst-sandbox", 10, true);
return messageSource;
}
// Define what happens to the messages arriving in the message channel.
#ServiceActivator(inputChannel = "inputMessageChannel")
public void messageReceiver(
String payload,
#Header(GcpPubSubHeaders.ORIGINAL_MESSAGE) BasicAcknowledgeablePubsubMessage message) {
System.out.println("Message arrived via an inbound channel adapter from sub-one! Payload: " + payload);
message.ack();
}
}
My thinking was that the poller annotation would start a poller to run every so often to check for messages and send them to the method annotated with service activator but this is clearly not the case as it is never hit.
Interestingly enough if I put a breakpoint right before "return messageSource" and check the result of the template.pull call the messages ARE returned so it is seemingly not an issue with the connection itself.
What am I missing here? Tutorials and documentation aren't helping much at this point as they all use pretty much the same bit of tutorial code like above...
I have tried variations of the above code like creating the adapter instead of the messagesource like so:
#Bean
public PubSubInboundChannelAdapter inboundChannelAdapter(
#Qualifier("inputMessageChannel") MessageChannel messageChannel,
PubSubTemplate pubSubTemplate) {
PubSubInboundChannelAdapter adapter =
new PubSubInboundChannelAdapter(pubSubTemplate, "tst-sandbox");
adapter.setOutputChannel(messageChannel);
adapter.setAckMode(AckMode.MANUAL);
adapter.setPayloadType(String.class);
return adapter;
}
to no avail. Any suggestions are welcome.
Found the problem after creating a spring boot project from scratch (main project is normal spring). Noticed in the debug output that it was auto starting the service activator bean and some other things like actually subscribing to the channels which it wasn't doing in the main project.
After a quick google the solution was simple, had to add
#EnableIntegration
annotation at class level and the messages started coming in.
Related
I have found out that an IntegrationFlow I have written using Java DSL wasn't very testable so I have followed Configuring with Java Configuration and split it into #Bean configuration.
In my unit test I have used a 3rd party SFTP in memory server and I tried triggering InboundChannelAdaper and then calling receive() on the channel.
I had a problem with finding out the type of Channel to use, as Channel usage was not mentioned anywhere in the SFTP Adapters documentation, but ultimately I found what I think is correct (QueueChannel) in the testing examples repository .
My problem is that the unit test I wrote is hanging on the channel's receive() method. Through debugging I determined that session factory's getSession() never gets called.
What am I doing wrong?
#Bean
public PollableChannel sftpChannel() {
return new QueueChannel();
}
#Bean
#EndpointId("sftpInboundAdapter")
#InboundChannelAdapter(channel = "sftpChannel", poller = #Poller(fixedDelay = "1000"))
public SftpInboundFileSynchronizingMessageSource sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source =
new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer);
source.setLocalDirectory(new File("/local"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
source.setMaxFetchSize(6);
return source;
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(testSftpSessionFactory);
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setPreserveTimestamp(true);
fileSynchronizer.setRemoteDirectory("/remote");
List<String> filterFileNameList = List.of("1.txt");
fileSynchronizer.setFilter(new FilenameListFilter(filterFileNameList));
return fileSynchronizer;
}
#Bean
private DefaultSftpSessionFactory testSftpSessionFactory(String username, String password, int port, String host) {
DefaultSftpSessionFactory defaultSftpSessionFactory = new DefaultSftpSessionFactory();
defaultSftpSessionFactory.setPassword("password");
defaultSftpSessionFactory.setUser("username");
defaultSftpSessionFactory.setHost("localhost");
defaultSftpSessionFactory.setPort(777);
defaultSftpSessionFactory.setAllowUnknownKeys(true);
Properties config = new java.util.Properties();
config.put( "StrictHostKeyChecking", "no" );
defaultSftpSessionFactory.setSessionConfig(config);
return defaultSftpSessionFactory;
}
#ExtendWith(SpringExtension.class)
#ContextConfiguration(classes = {IntegrationFlowTestSupport.class, Synchronizer.class, Channel.class, Activator.class})
public class IntegrationFlowConfigTest {
private static final String CONTENTS = "abcdef 1234567890";
#Autowired
PollableChannel sftpChannel;
#Autowired
DefaultSftpSessionFactory testSftpSessionFactory;
#Autowired
SftpInboundFileSynchronizer sftpInboundFileSynchronizer;
#Autowired
SftpInboundFileSynchronizingMessageSource sftpMessageSource;
#Autowired
SourcePollingChannelAdapter sftpInboundAdapter;
#Test
public void test() throws Exception {
FileEntry f1 = new FileEntry("/remote/1.txt", CONTENTS);
FileEntry f2 = new FileEntry("/remote/2.txt", CONTENTS);
FileEntry f3 = new FileEntry("/remote/3.txt", CONTENTS);
withSftpServer(server -> {
server.setPort(777);
server.addUser("username", "password");
server.putFile(f1.getPath(), f1.createInputStream());
server.putFile(f2.getPath(), f2.createInputStream());
sftpInboundAdapter.start();
Message<?> message = sftpChannel.receive();
});
}
}
First of all it is wrong to rewrite your code to satisfy unit test expectations. We spend not one hour thinking about dividing concerns from production code to testing.
See respective documentation: https://docs.spring.io/spring-integration/docs/current/reference/html/testing.html#test-context.
For your use-case it might be better to do a mock on that #ServiceActivator instead of QueueChannel and competing consumer in your test. What I mean that you already have a consumer in your configuration with that #ServiceActivator. So, there is no guarantee that your manual sftpChannel.receive() would give you a message from the queue since this one could be consumed by your #ServiceActivator subscriber.
The fixedDelay = "0" looks suspicious. Isn't that too often to ask SFTP server for new files? How do you expect your system would be stable enough if you give it so much stress with such a short delay?
We don't know what is withSftpServer(server -> {, and it is also not clear what is testSftpSessionFactory. So, not sure yet how you start an SFTP server and connect to it from your code.
I also see sftpMessageSource.start();, but there is nowhere in your that it is stopped somehow. Plus I guess you really meant to start an endpoint, not source. The endpoint in your case is a SourcePollingChannelAdapter created for that #InboundChannelAdapter. You can use an #EndpointId, if it is not autowired automatically by type.
In our tests we use Apache MINA SSH library: https://github.com/spring-projects/spring-integration/blob/main/spring-integration-sftp/src/test/java/org/springframework/integration/sftp/SftpTestSupport.java#L64-L76
In my little microservice, I created a Producer Kafka to send the messages with errors (messages having errors in the JSON format) inside the DeadLetter in this way :
#Component
public class KafkaProducer {
#Autowired
private KafkaTemplate<String, String> kafkaTemplate;
public void sendDeadLetter(String message) {
kafkaTemplate.send("DeadLetter", message);
}
}
I would like to create a JUnitTest for the completeness of the project, but I have no idea how to create the eventuality of a possible JSON error in order to create the test. I thank everyone for any possible help and advice
To create a JUnitTest consistent with your code. I should recreate the case where you pass it a warped or invalid JSON. In your case, I would opt to configure a MockConsumer from which to read any message that the logic of your code will be invited to the dead letter.
To have a usable test structure, I recommend something like this:
#KafkaListener(topics = "yourTopic")
public void listen(String message) {
messages.add(message);
}
For testing a basic structure could be
#Test
public void testDeadLetter(){
//Set up a mockConsumer
MockConsumer<String,String> yourMockConsumer = new MockConsumer<String,String> (OffsetResetStrategy.EARLIEST);
yourMockConsumer.subscribe(Collections.singletonList("yourTopic"));
//Sending message on embedded Kafka broker
String error = "ERRORE";
kafkaTemplate.send("yourTopic", error);
//Reading the message may take a second
Thread.sleep(1000);
//Create an Assert that checks you that the message is equal to the error specified above
}
I hope it will be useful to you!
You can create Kafka topic using testcontainers and write your tests on top of that.
Sharing an example on how to use testcontainers https://github.com/0001vrn/testcontainers-example
I am having one config file as below,
#Slf4j
#Configuration
#EnableJms
public class MyQpidConfig {
#Bean("myQpidConn")
public AMQConnectionFactory amqConnectionFactory() throws URLSyntaxException {
AMQConnectionFactory amqConnectionFactory = new AMQConnectionFactory("URL");
return amqConnectionFactory;
}
#Bean("myJmsContainer")
public DefaultMessageListenerContainer defaultMessageListenerContainer(#Qualifier("myQpidAdaptor") MessageListenerAdapter messageListenerAdapter ,#Qualifier("myQpidConn")AMQConnectionFactory connectionFactory) {
DefaultMessageListenerContainer messageListenerContainer = new DefaultMessageListenerContainer();
messageListenerContainer.setConnectionFactory(connectionFactory);
messageListenerContainer.setDestinationName(destinationName);
messageListenerContainer.setMessageListener(messageListenerAdapter);
messageListenerContainer.setPubSubDomain(false);
messageListenerContainer.setSessionTransacted(true);
messageListenerContainer.setDurableSubscriptionName("MYSUBSCRIBER");
messageListenerContainer.setSubscriptionDurable(true);
messageListenerContainer.setAutoStartup(true);
messageListenerContainer.setDestination((Queue) () -> "MYQUEUE");
return messageListenerContainer;
}
#Bean("myQpidAdaptor")
public MessageListenerAdapter messageListenerAdapter() {
Preconditions.checkNotNull(qpidMessageListner, "Consumer listener not set");
MessageListenerAdapter messageListenerAdapter = new MessageListenerAdapter(new MyQpidMessageListner());
messageListenerAdapter.setDefaultListenerMethod("onMessage");
messageListenerAdapter.setDefaultResponseQueueName("MYQUEUE");
return messageListenerAdapter;
}
}
where destinationName is equal to
ADDR:MYSUBSCRIBER:MYQUEUE; {"create":"receiver",
"node":{"durable":true,"x-declare":
{"exclusive":true,"auto-delete":false,
"x-bindings":[{"exchange":"amq.topic","key":"MYQUEUE"}]}}}
and my Listener class is as below,
#Slf4j
#Component
public class MyQpidMessageListner implements MessageListener{
#Override
public void onMessage(Message jmsBytesMessage) {
log.info("Control Inside");
//to do
}
}
as a async consumer when i am running the my application at that time getting exception related to destination,
DefaultMessageListenerContainer -
Setup of JMS message listener invoker failed for destination 'com.my.MyQpidConfig$$Lambda$637/429515253#3aa1975' - trying to recover. Cause: com.my.MyQpidConfig$$Lambda$637/429515253 cannot be cast to org.apache.qpid.client.AMQDestination
If its related to AMQDestination then how can i create the AMQDestination in java code.
I am not getting any resource for this QPID issue.
Any suggestion help must be appreciated.
For some reason you're passing a lambda to org.springframework.jms.listener.DefaultMessageListenerContainer.setDestination(Destination), i.e.:
messageListenerContainer.setDestination((Queue) () -> "MYQUEUE");
This method needs a javax.jms.Destination implementation (e.g. org.apache.qpid.client.AMQDestination), not a lambda.
You're already setting the destination name, i.e.:
messageListenerContainer.setDestinationName(destinationName);
So it's not clear why you're also trying to set the destination as well. These are two ways to configure the same thing.
I recommend you simply remove the code attempting to set the destination and leave the code setting the destination name.
I'm trying to test listener in springboot created using #KafkaListener
But listener always listens on localhost:9092 instead of using this embededKafka
My listener looks like this:
#Component
#Slf4j
class SomeListener {
private final List<String> receivedMessages = new ArrayList<>();
#KafkaListener(topics = "some-ultra-cool-topic")
public void onKafkaMessage(String theMessage) {
log.info("Message received {}", theMessage);
receivedMessages.add(theMessage);
}
Collection<String> getAll() {
return unmodifiableCollection(receivedMessages);
}
}
And spock test like this:
#SpringBootTest
#EmbeddedKafka
#TestPropertySource(properties = ['spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}', 'spring.kafka.consumer.auto-offset-reset=earliest'])
class SomeListenerTest extends Specification {
#Autowired
EmbeddedKafkaBroker embeddedKafkaBroker
#Autowired
SomeListener someListener
void 'should receive message'() {
given:
def sender = new KafkaTemplate<>(new DefaultKafkaProducerFactory<String, String>(KafkaTestUtils.producerProps(embeddedKafkaBroker)))
when:
sender.send('some-ultra-cool-topic', 'first message content')
then:
someListener.all.size() == 1
}
}
My application.yaml doesn't have bootstraps servers configured - so it is purly default from spring-boot.
I can see in logs that producer is sending message to broker (it starts every time on different random port).
But listener always try to connect to broker on localhost:9092
How can I configure it to use this embedded one?
Thanks #sawim for tips
Actual problem was in test. I ended up doing this test with lib org.awaitility:awaitility
then:
waitAtMost(5, SECONDS)
.untilAsserted({ ->
assertThat(personFacade.findAll(), hasSize(1))
})
Configuration from first example works, however during startup I can see kafka-logs trying to connect to localhost:9200 - seems we can Ignore it
There's a Job which has a list of tasks.
Each task has id, name, status.
I've created service activators for each tasks, as follows:
#ServiceActivator
public Message<Task> execute(Message<Task> message){
//do stuff
}
I've created a gateway for Job
and in the Integration flow, starting from the gateway:
#Bean
public IntegrationFlow startJob() {
return f -> f
.handle("jobService", "execute")
.channel("TaskRoutingChannel");
}
#Bean
public IntegrationFlow startJobTask() {
return IntegrationFlows.from("TaskRoutingChannel")
.handle("jobService", "executeTasks")
.route("headers['Destination-Channel']")
.get();
}
#Bean
public IntegrationFlow TaskFlow() {
return IntegrationFlows.from("testTaskChannel")
.handle("aTaskService", "execute")
.channel("TaskRoutingChannel")
.get();
}
#Bean
public IntegrationFlow TaskFlow2() {
return IntegrationFlows.from("test2TaskChannel")
.handle("bTaskService", "execute")
.channel("TaskRoutingChannel")
.get();
}
I've got the tasks to execute sequentially, using routers as above.
However, I need to start the job, execute all of it's tasks in parallel.
I couldn't figure out how to get that going.. I tried using #Async on the service activator methods and making it return void. but in that case, how do i chain it back to the routing channel and make it start next task?
Please help. Thanks.
EDIT:
I used the RecepientListRouter along with ExecutorChannel to get the parallel execution:
#Bean
public IntegrationFlow startJobTask() {
return IntegrationFlows.from("TaskRoutingChannel")
.handle("jobService", "executeTasks")
.routeToRecipients(r -> r
.recipient("testTaskChannel")
.recipient("test2TaskChannel"))
.get();
}
#Bean ExecutorChannel testTaskChannel(){
return new ExecutorChannel(this.getAsyncExecutor());
}
#Bean ExecutorChannel test2TaskChannel(){
return new ExecutorChannel(this.getAsyncExecutor());
}
#Bean
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(5);
executor.setMaxPoolSize(10);
executor.setQueueCapacity(10);
executor.initialize();
return executor;
}
Now, 3 questions:
1) If this is a good approach, how do i send specific parts of the payload to each recipient channel. Assume the payload is a List<>, and i want to send each list item to each channel.
2) How do I dynamically set the recipient channel? say from header? or a list?
3) Is this really a good approach? Is there a preferred way to do this?
Thanks in Advance.
Your TaskRoutingChannel must be an instance of ExecutorChannel. For example:
return f -> f
.handle("jobService", "execute")
.channel(c -> c.executor("TaskRoutingChannel", threadPoolTaskExecutor()));
Otherwise, yes: everything is invoked with the single Thread and it isn't good for your task.
UPDATE
Let me try to answer to your questions one by one, although it sounds like each of them must as separate SO one :-).
If you really need to send the same message to several services, you can use routeToRecipients, or can back to the publishSubscribe. Or even can do dynamic routing based on the header, for example.
To send the part of message to each channel there is just enough place .split() before your .routeToRecipients()
To answer to your last question I need to know the business requirements for the task.