try to use spring cloud gcp binder library polling messages from GCP pubsub topic.
ref from streaming-vs-polled-input. use spring.cloud.stream.gcp.pubsub.default.consumer.maxFetchSize poll get N message (maxFetchSize's value) at a time, but even i set up the property and set maxFetchSize to 2 or others. i alwasy got 1 message from topic, even the topic has other messages. Does anyone here has other idea?
spring cloud version: 2021.0.3
google spring-cloud-gcp-dependencies version: 3.3.0
java: openjdk 11
# application.properties
spring.cloud.gcp.pubsub.project-id=
spring.cloud.gcp.credentials.location=
spring.cloud.stream.default-binder=pubsub
spring.cloud.stream.bindings.input.destination=iamatopic
spring.cloud.stream.bindings.input.content-type=text/plain;charset=UTF-8
spring.cloud.stream.gcp.pubsub.default.consumer.maxFetchSize=3
import org.springframework.cloud.stream.annotation.Input;
import org.springframework.cloud.stream.binder.PollableMessageSource;
public interface PollableSink {
#Input("input")
PollableMessageSource input();
}
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationRunner;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.cloud.stream.binder.PollableMessageSource;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.ParameterizedTypeReference;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHandler;
import org.springframework.scheduling.annotation.Scheduled;
import com.google.cloud.spring.pubsub.support.AcknowledgeablePubsubMessage;
import com.google.cloud.spring.pubsub.support.GcpPubSubHeaders;
import lombok.extern.slf4j.Slf4j;
#Configuration
#EnableBinding(PollableSink.class)
#Slf4j
public class PollConfiguration {
#Autowired
PollableMessageSource destIn;
PolledMessageHandler messageHandler = new PolledMessageHandler();
#Scheduled(fixedRate = 5000)
public void poller() {
log.info("start polling ");
destIn.poll(this.messageHandler, ParameterizedTypeReference.forType(String.class));
log.info("end polling ");
}
static class PolledMessageHandler implements MessageHandler {
#Override
public void handleMessage(Message<?> message) {
AcknowledgeablePubsubMessage ackableMessage = (AcknowledgeablePubsubMessage) message.getHeaders()
.get(GcpPubSubHeaders.ORIGINAL_MESSAGE);
ackableMessage.ack();
System.out.println("get payload : " + message.getPayload());
}
}
}
Try setting spring.cloud.stream.poller.maxMessagesPerPoll value -- it's 1 by default in Spring Cloud Stream.
Reference: https://docs.spring.io/spring-cloud-stream/docs/current/reference/html/spring-cloud-stream.html#_polling_configuration_properties
Related
Here is my code. I have granted the bot Adminastor permissions, and I still manually granted all permissions for the bot. I tried making a test server, but the bot still does not respond in servers; it only responds to direct messages.
import org.javacord.api.DiscordApi;
import org.javacord.api.DiscordApiBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.core.env.Environment;
#SpringBootApplication
public class GramophoneApplication {
#Autowired
private Environment env;
public static void main(String[] args) {
SpringApplication.run(GramophoneApplication.class, args);
}
#Bean
#ConfigurationProperties(value = "token")
public DiscordApi discordApi(){
String token = env.getProperty("token");
DiscordApi api = new DiscordApiBuilder().setToken(token)
.setAllNonPrivilegedIntents()
.login()
.join();
api.addMessageCreateListener(event -> {
if(event.getMessageContent().equals("!ping")){
event.getChannel().sendMessage("Pong!");
}
});
return api;
}
}
I have a spring boot application with JMS messaging included. The receiver and the connection Factory is as below,The problem I am having is the listener always looks the queue and prints in logs, Logs attached for reference. It should trigger the process() when there is a message in the queue.
Any help appreciated
package uk.gov.hmrc.cds.dmscaseemulator.receiver;
import javax.jms.JMSException;
import javax.jms.Message;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.jms.annotation.EnableJms;
import org.springframework.jms.annotation.JmsListener;
import org.springframework.stereotype.Service;
import lombok.extern.slf4j.Slf4j;
import uk.gov.hmrc.cds.dmscaseemulator.bo.DMSMessageProcessBO;
#Service
#Slf4j
public class DMSQReceiver {
#Value("${queue.DMS_Q}")
private String listeningQ;
#Autowired DMSMessageProcessBO processBo;
#JmsListener(destination="${queue.DMS_Q}", containerFactory = "myFactory")
public void recv(Message message) throws JMSException{
try{
log.info("Listening to : {} ", listeningQ);
processBo.process(message);
}catch(JMSException ex){
log.error("ERROR IN DMS Case Emulator while receiving " + listeningQ +". ERROR IS : ", ex);
throw ex;
}
}
}
package uk.gov.hmrc.cds.dmscaseemulator.config;
import javax.jms.ConnectionFactory;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.springframework.boot.autoconfigure.jms.DefaultJmsListenerContainerFactoryConfigurer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jms.annotation.EnableJms;
import org.springframework.jms.config.DefaultJmsListenerContainerFactory;
import org.springframework.jms.config.JmsListenerContainerFactory;
import org.springframework.jms.connection.CachingConnectionFactory;
import org.springframework.jms.support.converter.MappingJackson2MessageConverter;
import org.springframework.jms.support.converter.MessageConverter;
import org.springframework.jms.support.converter.MessageType;
#Configuration
#EnableJms
public class JMSConfig {
#Bean
public JmsListenerContainerFactory<?> myFactory(
ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
configurer.configure(factory, connectionFactory);
return factory;
}
}
20:23:37.281 [DefaultMessageListenerContainer-1] INFO u.g.h.c.d.receiver.DMSQReceiver - Listening to : DMS.QA.CTRL_MGMT_EXPORT
20:23:37.282 [DefaultMessageListenerContainer-1] DEBUG u.g.h.c.d.b.c.ControlTypeResolver - Found control type
20:23:37.282 [DefaultMessageListenerContainer-1] DEBUG u.g.h.c.d.b.c.ControlTypeResolver - Returning control type value : 3
20:23:37.283 [DefaultMessageListenerContainer-1] INFO u.g.h.c.d.bo.DMSMessageProcessBO - Control Type is: 3
20:23:37.290 [DefaultMessageListenerContainer-1] INFO u.g.h.c.d.receiver.DMSQReceiver - Listening to : DMS.QA.CTRL_MGMT_EXPORT
20:23:37.291 [DefaultMessageListenerContainer-1] DEBUG u.g.h.c.d.b.c.ControlTypeResolver - Found control type
20:23:37.291 [DefaultMessageListenerContainer-1] DEBUG u.g.h.c.d.b.c.ControlTypeResolver - Returning control type value : 3
20:23:37.292 [DefaultMessageListenerContainer-1] INFO u.g.h.c.d.bo.DMSMessageProcessBO - Control Type is: 3
I am trying to configure the Confluent - ConsumerTimestampsInterceptor to support the Confluent KAFKA Replication and have configured Java spring boot application as mentioned below
application.properties
#consumer timestamp interceptor
interceptor.classes=io.confluent.connect.replicator.offsets.ConsumerTimestampsInterceptor
timestamp.producer.security.protocol=PLAINTEXT
timestamp.producer.sasl.mechanism=NONE
#timestamp.producer.ssl.endpoint.identification.algorithm=
#timestamp.producer.sasl.jaas.config=
src.consumer.group.id=lkc-302y2
Consumer.java
package com.example.demo.service;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;
import com.example.demo.model.ProductKafka;
import java.io.IOException;
import java.util.ArrayDeque;
import java.util.LinkedList;
import java.util.Queue;
import java.util.logging.Logger;
#Service
public class Consumer {
private static final Logger LOGGER = Logger.getLogger(Consumer.class.getName());
public static ArrayDeque<ProductKafka> consumeQueue = new ArrayDeque<>();
#KafkaListener(autoStartup = "false", topics = "#{'${spring.kafka.topics}'.split('\\\\ ')}", groupId = "#{'${spring.kafka.groupId}'}")
public void consume(ProductKafka productKafka) throws IOException {
consumeQueue.offer(productKafka);
LOGGER.info(String.format("#### -> Logger Consumed message -> %s", productKafka.toString()));
System.out.printf("#### -> Consumed message -> %s", productKafka.toString());
}
}
and below is the project structure
enter image description here
however it is not working.
Our KAFKA administrator mentioned that though I have the Interceptor added KAFKA consumer is not configured to use the interceptor.
I don't know how to make the consumer use the Interceptor
I am developing an asynchronous mail server in spring boot using kafka.
I have written tests with embedded kafka which starts its own kafka topic in a random port and use it for testing.
When I started this application context is loading and its expecting for kafka cluster in my local. I need to stop application conext from loading.
I replicated the code from https://github.com/code-not-found/spring-kafka/blob/master/spring-kafka-unit-test-classrule/src/test/java/com/codenotfound/kafka/producer/SpringKafkaSenderTest.java which works absolutely fine. When I followed same style in my project, I can see actual apllication starting.
SpringKafkaSenderTest .java
package com.mailer.embeddedkafkatests;
import static org.junit.Assert.assertTrue;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.junit.After;
import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.kafka.listener.ContainerProperties;
import org.springframework.kafka.listener.KafkaMessageListenerContainer;
import org.springframework.kafka.listener.MessageListener;
import org.springframework.kafka.test.rule.EmbeddedKafkaRule;
import org.springframework.kafka.test.utils.ContainerTestUtils;
import org.springframework.kafka.test.utils.KafkaTestUtils;
import org.springframework.test.annotation.DirtiesContext;
import org.springframework.test.context.junit4.SpringRunner;
import com.mailer.model.Mail;
import com.mailer.producer.KafkaMessageProducer;
import com.mailer.serializer.MailSerializer;
#RunWith(SpringRunner.class)
#SpringBootTest
#DirtiesContext
public class SpringKafkaSenderTest {
private static final Logger LOGGER =
LoggerFactory.getLogger(SpringKafkaSenderTest.class);
private static String SENDER_TOPIC = "sender.t";
#Autowired
private KafkaMessageProducer sender;
private KafkaMessageListenerContainer<String, Mail> container;
private BlockingQueue<ConsumerRecord<String, Mail>> records;
#ClassRule
public static EmbeddedKafkaRule embeddedKafka =
new EmbeddedKafkaRule(1, true, SENDER_TOPIC);
#Before
public void setUp() throws Exception {
// set up the Kafka consumer properties
Map<String, Object> consumerProperties =
KafkaTestUtils.consumerProps("sender", "false",
embeddedKafka.getEmbeddedKafka());
consumerProperties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
consumerProperties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, MailSerializer.class);
// create a Kafka consumer factory
DefaultKafkaConsumerFactory<String, Mail> consumerFactory =
new DefaultKafkaConsumerFactory<String, Mail>(
consumerProperties);//, new StringDeserializer(), new JsonDeserializer<>(Mail.class));
// set the topic that needs to be consumed
ContainerProperties containerProperties =
new ContainerProperties(SENDER_TOPIC);
// create a Kafka MessageListenerContainer
container = new KafkaMessageListenerContainer<>(consumerFactory,
containerProperties);
// create a thread safe queue to store the received message
records = new LinkedBlockingQueue<>();
// setup a Kafka message listener
container
.setupMessageListener(new MessageListener<String, Mail>() {
#Override
public void onMessage(
ConsumerRecord<String, Mail> record) {
LOGGER.debug("test-listener received message='{}'",
record.toString());
records.add(record);
}
});
// start the container and underlying message listener
container.start();
// wait until the container has the required number of assigned partitions
ContainerTestUtils.waitForAssignment(container,
embeddedKafka.getEmbeddedKafka().getPartitionsPerTopic());
}
#After
public void tearDown() {
// stop the container
container.stop();
}
#Test
public void testSend() throws InterruptedException {
// send the message
Mail mail = new Mail();
mail.setFrom("vinoth#local.com");
sender.sendMessage(mail);
Thread.sleep(4000);
// check that the message was received
ConsumerRecord<String, Mail> received =
records.poll(10, TimeUnit.SECONDS);
// Hamcrest Matchers to check the value
assertTrue(received.value().getFrom().equals(mail.getFrom()));
System.out.println(received.value().getFrom());
// assertThat(received, hasValue(mail));
// AssertJ Condition to check the key
// assertThat(received).has(key(null));
}
}
Why would you like to stop the spring context from loading? Isn't the purpose of this junit to test your spring application?
In any case just remove the #SpringBootTest annotation and the spring context will not load.
This is the Java EE7 #Startup annotation for reference. I am using the latest GlassFish server and IntelliJ to run the server.
I need to instantiate this service so that it can send packets of data periodically to a websocket for processing, but the fact that I use
session.getBasicRemote().sendObject(message);
in the end forces me to throw IOException and EncodeException.
This is bad as #Startup or #Singleton disallows usage of Exceptions when instantiating:
war exploded: java.io.IOException: com.sun.enterprise.admin.remote.RemoteFailureException: Error occurred during deployment: Exception while deploying the app [keyBoard_war_exploded] : The lifecycle method [printSchedule] must not throw a checked exception.
Here is my code:
package site.xxxx.models;
import site.xxxx.message.Message;
import site.xxxx.message.TextMessage;
import site.xxxx.modules.Packet;
import site.xxxx.websocket.MainServerEndpoint;
import javax.annotation.PostConstruct;
import javax.ejb.Schedule;
import javax.ejb.Singleton;
import javax.ejb.Startup;
import javax.json.Json;
import javax.json.JsonObject;
import javax.websocket.EncodeException;
import javax.websocket.Session;
import java.io.IOException;
import java.util.Set;
#Startup
#Singleton
public class Service {
private static Service service;
Set<Session> peers = MainServerEndpoint.peers;
#PostConstruct
public void mainLoop() throws IOException, EncodeException, InterruptedException {
sendSpamPacket();
}
private void sendSpamPacket() throws IOException, EncodeException {
JsonObject ret = Json.createObjectBuilder()
.add("type", "textMessage")
.add("text", "ayyyy")
.build();
Packet packet = new Packet(new TextMessage(ret));
MainServerEndpoint.sendPacket(packet);
//results in calling session.getBasicRemote().sendObject(message);
}
}