Develop Reference

Wait for RabbitTemplate Correlated confirms

I have this function in RabbitMQ class that is used to publish messages:
#Retryable(value = { AmqpIOException.class, AmqpTimeoutException.class, AmqpConnectException.class, AmqpReplyTimeoutException.class },
maxAttempts = 3,
backoff = #Backoff(delay = 1000))
public void publish(Message msg, SuccessCallback<? super CorrelationData.Confirm> successCallback, FailureCallback failureCallback, String customRoutingKey) {
CorrelationData data = new CorrelationData();
data.getFuture().addCallback(successCallback, failureCallback);
rabbitTemplate.convertAndSend(exchange.getName(), customRoutingKey, msg, data);
}
And here is how I use this function above:
public void sendMessageToRMQ(MyEntity msgObj, String routingKey) throws ServiceUnavailableException {
try {
rmqPublisher.publish(
MessageBuilder
.withBody(RMQPayloadBuilder.buildPayload(msgObj))
.build(),
confirmation -> {
if (confirmation != null && confirmation.isAck()) {
msgObj.setSentDate(OffsetDateTime.now(ZoneOffset.UTC));
repository.save(msgObj);
} else {
StructuredLogger.logErrorMessage("Message un-ack", logValues);
}
},
failure -> StructuredLogger.logErrorMessage("un able to send", logValues),
routingKey
);
} catch (AmqpException amqpException) {
StructuredLogger.logErrorMessage("Unable to publish message", logValues);
}
}
The the above function is used in a for loop to publish messages for huge number of msgObjects.
What I am trying to achieve is something like this:
// msgObjArray is a chunk of the whole msgObjArray
for (MyEntity msgObj : msgObjArrayChunk) {
try {
this.service.sendMessageToRMQ(msgObj, this.routingKey);
} catch (ServiceUnavailableException e) {
StructuredLogger.logErrorMessage("Failed to send ");
}
}
// What Should I write here to wait for the above chunk of msgObj to be acked and finished processing

You need to return the CorrelationData from publish and collect them into a list; then, after the sends are complete, iterate over them and use data.getFuture().get( <some timeout> ) to wait for each confirm.

How to distribute kafka load into openshift pods using kafka partitions

I have a spring boot application and I would like to distribute the load of a Kafka topic into 3 open-shift pods. I have the following example where I can listen from 3 Kafka partitions on three different threads, this spring boot application will load into one openshift pod. But I want to be a able to listen from one Kafka partition on one pod so when I load 3 pods on open-shift each pod will listen from one Kafka partition. This will allow me to scale the application to N partitions on N pods. I am not sure if this is possible or if need to use a different approach. Thanks
public class DepAcctInqConsumerController {
private static final Logger LOGGER = LoggerFactory.getLogger(DepAcctInqConsumerController.class);
#Value("${kafka.topic.acct-info.request}")
private String requestTopic;
#KafkaListener(id = "id-0",containerFactory = "requestReplyListenerContainerFactory",
topicPartitions = { #TopicPartition(topic = "${kafka.topic.acct-info.request}", partitions = "0" )})
public Message<?> listenPartition0(InGetAccountInfo accountInfo, #Header(KafkaHeaders.REPLY_TOPIC) byte[] replyTo,
#Header(KafkaHeaders.CORRELATION_ID) byte[] correlation,#Header(KafkaHeaders.RECEIVED_PARTITION_ID) int id) {
try {
LOGGER.info("Received request for partition id = " + id);
AccountInquiryDto accountInfoDto = getAccountInquiryDto(accountInfo);
return MessageBuilder.withPayload(accountInfoDto)
.setHeader(KafkaHeaders.TOPIC, replyTo)
.setHeader(KafkaHeaders.RECEIVED_PARTITION_ID, id)
.setHeader(KafkaHeaders.CORRELATION_ID, correlation)
.build();
} catch (Exception e) {
LOGGER.error(e.toString(),e);
}
return null;
}
#KafkaListener(id = "id-1",containerFactory = "requestReplyListenerContainerFactory",
topicPartitions = { #TopicPartition(topic = "${kafka.topic.acct-info.request}", partitions = "#{#finder.partitions(${kafka.topic.acct-info.request)}" )})
public Message<?> listenPartition1(InGetAccountInfo accountInfo, #Header(KafkaHeaders.REPLY_TOPIC) byte[] replyTo,
#Header(KafkaHeaders.CORRELATION_ID) byte[] correlation,#Header(KafkaHeaders.RECEIVED_PARTITION_ID) int id) {
try {
LOGGER.info("Received request for partition id = " + id);
AccountInquiryDto accountInfoDto = getAccountInquiryDto(accountInfo);
return MessageBuilder.withPayload(accountInfoDto)
.setHeader(KafkaHeaders.TOPIC, replyTo)
.setHeader(KafkaHeaders.RECEIVED_PARTITION_ID, id)
.setHeader(KafkaHeaders.CORRELATION_ID, correlation)
.build();
} catch (Exception e) {
LOGGER.error(e.toString(),e);
}
return null;
}
#KafkaListener(id = "id-2",containerFactory = "requestReplyListenerContainerFactory",
topicPartitions = { #TopicPartition(topic = "${kafka.topic.acct-info.request}", partitions = "2" )})
public Message<?> listenPartition2(InGetAccountInfo accountInfo, #Header(KafkaHeaders.REPLY_TOPIC) byte[] replyTo,
#Header(KafkaHeaders.CORRELATION_ID) byte[] correlation, #Header(KafkaHeaders.RECEIVED_PARTITION_ID) int id) {
try {
LOGGER.info("Received request for partition id = " + id);
AccountInquiryDto accountInfoDto = getAccountInquiryDto(accountInfo);
return MessageBuilder.withPayload(accountInfoDto)
.setHeader(KafkaHeaders.TOPIC, replyTo)
.setHeader(KafkaHeaders.RECEIVED_PARTITION_ID, id)
.setHeader(KafkaHeaders.CORRELATION_ID, correlation)
.build();
} catch (Exception e) {
LOGGER.error(e.toString(),e);
}
return null;
}

We don't need to have multiple kafka listeners for each partition. We just need one listener.
If you are running single pod, messages from all three partitions will be consumed by that single pod,
If you run more than 1 pod, partitions will be distributed across pods.
We can run as many pods as no of partitions.
All Pods must use same consumer group name.
This is all we need.
#KafkaListener(topics = "${kafka.topic.acct-info.request}")
public void receive(ConsumerRecord<String, String> record)

I can't send message using google pubsub emulator in spring boot

I'm trying to send push message using the emulator of pubsub, I'm using spring boot too, this is my configuration:
Dependency:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-pubsub</artifactId>
</dependency>
My bean:
#Configuration
#AutoConfigureBefore(value= GcpPubSubAutoConfiguration.class)
#EnableConfigurationProperties(value= GcpPubSubProperties.class)
public class EmulatorPubSubConfiguration {
#Value("${spring.gcp.pubsub.projectid}")
private String projectId;
#Value("${spring.gcp.pubsub.subscriptorid}")
private String subscriptorId;
#Value("${spring.gcp.pubsub.topicid}")
private String topicId;
#Bean
public Publisher pubsubEmulator() throws IOException {
String hostport = System.getenv("PUBSUB_EMULATOR_HOST");
ManagedChannel channel = ManagedChannelBuilder.forTarget(hostport).usePlaintext().build();
try {
TransportChannelProvider channelProvider =
FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel));
CredentialsProvider credentialsProvider = NoCredentialsProvider.create();
// Set the channel and credentials provider when creating a `TopicAdminClient`.
// Similarly for SubscriptionAdminClient
TopicAdminClient topicClient =
TopicAdminClient.create(
TopicAdminSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build());
ProjectTopicName topicName = ProjectTopicName.of(projectId, topicId);
// Set the channel and credentials provider when creating a `Publisher`.
// Similarly for Subscriber
return Publisher.newBuilder(topicName)
.setChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build();
} finally {
channel.shutdown();
}
}
}
Of course, I have set PUBSUB_EMULATOR_HOST system variable to localhost:8085, where is the emulator running
I created a rest controller for testing:
for send push message
#Autowired
private Publisher pubsubPublisher;
#PostMapping("/send1")
public String publishMessage(#RequestParam("message") String message) throws InterruptedException, IOException {
Publisher pubsubPublisher = this.getPublisher();
ByteString data = ByteString.copyFromUtf8(message);
PubsubMessage pubsubMessage = PubsubMessage.newBuilder().setData(data).build();
ApiFuture<String> future = pubsubPublisher.publish(pubsubMessage);
//pubsubPublisher.publishAllOutstanding();
try {
// Add an asynchronous callback to handle success / failure
ApiFutures.addCallback(future,
new ApiFutureCallback<String>() {
#Override
public void onFailure(Throwable throwable) {
if (throwable instanceof ApiException) {
ApiException apiException = ((ApiException) throwable);
// details on the API exception
System.out.println(apiException.getStatusCode().getCode());
System.out.println(apiException.isRetryable());
}
System.out.println("Error publishing message : " + message);
System.out.println("Error publishing error : " + throwable.getMessage());
System.out.println("Error publishing cause : " + throwable.getCause());
}
#Override
public void onSuccess(String messageId) {
// Once published, returns server-assigned message ids (unique within the topic)
System.out.println(messageId);
}
},
MoreExecutors.directExecutor());
}
finally {
if (pubsubPublisher != null) {
// When finished with the publisher, shutdown to free up resources.
pubsubPublisher.shutdown();
pubsubPublisher.awaitTermination(1, TimeUnit.MINUTES);
}
}
return "ok";
for get message:
#PostMapping("/pushtest")
public String pushTest(#RequestBody CloudPubSubPushMessage request) {
System.out.println( "------> message received: " + decode(request.getMessage().getData()) );
return request.toString();
}
I have created my topic and subscription in the emulator, I followed this tutorial:
https://cloud.google.com/pubsub/docs/emulator
I'm set the endpoint "pushtest" for get push message in the emulator, with this command:
python subscriber.py PUBSUB_PROJECT_ID create-push TOPIC_ID SUBSCRIPTION_ID PUSH_ENDPOINT
But when I run the test, doesn't reach "/pushtest" endpoint and I'm getting this error:
Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#265d5d05
[Not completed, task = java.util.concurrent.Executors$RunnableAdapter#a8c8be3
[Wrapped task = com.google.common.util.concurrent.TrustedListenableFutureTask#1a53c57c
[status=PENDING, info=[task=[running=[NOT STARTED YET], com.google.api.gax.rpc.AttemptCallable#3866e1d0]]]]]
rejected from java.util.concurrent.ScheduledThreadPoolExecutor#3f34809a
[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
for assurance that the emulator is running ok, I'm run the test in python with the following command:
python publisher.py PUBSUB_PROJECT_ID publish TOPIC_ID
And I'm getting messages correctly in "pushtest" endpoint.
I don't know why sorry for my hazing.
Thanks for your help.

I found the problem.
Only comment this line in the bean
channel.shutdown();
HAHA very simple.

Error in shutting down the Google Pub sub publisher

I have started the publisher of google pub sub using the coding[1].After finishing the publisher I shut down the publisher as[2].But when I run I'm getting an error[3]saying that the publisher is not properly shutdown.
I'm using pubsub 1.61.0 version.
Is there any way to handle this error?
[1]
public class PublisherExample {
// use the default project id
private static final String PROJECT_ID = ServiceOptions.getDefaultProjectId();
/** Publish messages to a topic.
* #param args topic name, number of messages
*/
public static void main(String... args) throws Exception {
// topic id, eg. "my-topic"
String topicId = args[0];
int messageCount = Integer.parseInt(args[1]);
ProjectTopicName topicName = ProjectTopicName.of(PROJECT_ID, topicId);
Publisher publisher = null;
List<ApiFuture<String>> futures = new ArrayList<>();
try {
// Create a publisher instance with default settings bound to the topic
publisher = Publisher.newBuilder(topicName).build();
for (int i = 0; i < messageCount; i++) {
String message = "message-" + i;
// convert message to bytes
ByteString data = ByteString.copyFromUtf8(message);
PubsubMessage pubsubMessage = PubsubMessage.newBuilder()
.setData(data)
.build();
// Schedule a message to be published. Messages are automatically batched.
ApiFuture<String> future = publisher.publish(pubsubMessage);
futures.add(future);
}
} finally {
// Wait on any pending requests
List<String> messageIds = ApiFutures.allAsList(futures).get();
for (String messageId : messageIds) {
System.out.println(messageId);
}
if (publisher != null) {
// When finished with the publisher, shutdown to free up resources.
publisher.shutdown();
publisher.awaitTermination(1, TimeUnit.MINUTES);
}
}
}
}
[2]
if (publisher != null) {
resources.
publisher.shutdown();
publisher.awaitTermination(1, TimeUnit.MINUTES);
}
[3]
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=9, target=pubsub.googleapis.com:443} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:419)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:254)

akka websocket with java, counting clients number, sending message to client

I'm following the akka java websocket tutorial in attempt to create a websocket server. I want to implement 2 extra features:
Being able to display the number of connected clients, but the result
is always 0 or 1 , even when I know I have 100's concurrently
connected clients.
Websocket communication is biDirectional. Currently the server only respond with a message when client sends a message. How do I initiate sending a message from server to client?
Here's original akka java server example code with minimum modification of my client counting implementation:
public class websocketServer {
private static AtomicInteger connections = new AtomicInteger(0);//connected clients count.
public static class MyTimerTask extends TimerTask {
//called every second to display number of connected clients.
#Override
public void run() {
System.out.println("Conncurrent connections: " + connections);
}
}
//#websocket-handling
public static HttpResponse handleRequest(HttpRequest request) {
HttpResponse result;
connections.incrementAndGet();
if (request.getUri().path().equals("/greeter")) {
final Flow<Message, Message, NotUsed> greeterFlow = greeter();
result = WebSocket.handleWebSocketRequestWith(request, greeterFlow);
} else {
result = HttpResponse.create().withStatus(413);
}
connections.decrementAndGet();
return result;
}
public static void main(String[] args) throws Exception {
ActorSystem system = ActorSystem.create();
TimerTask timerTask = new MyTimerTask();
Timer timer = new Timer(true);
timer.scheduleAtFixedRate(timerTask, 0, 1000);
try {
final Materializer materializer = ActorMaterializer.create(system);
final Function<HttpRequest, HttpResponse> handler = request -> handleRequest(request);
CompletionStage<ServerBinding> serverBindingFuture =
Http.get(system).bindAndHandleSync(
handler, ConnectHttp.toHost("****", 1183), materializer);
// will throw if binding fails
serverBindingFuture.toCompletableFuture().get(1, TimeUnit.SECONDS);
System.out.println("Press ENTER to stop.");
new BufferedReader(new InputStreamReader(System.in)).readLine();
timer.cancel();
} catch (Exception e){
e.printStackTrace();
}
finally {
system.terminate();
}
}
//#websocket-handler
/**
* A handler that treats incoming messages as a name,
* and responds with a greeting to that name
*/
public static Flow<Message, Message, NotUsed> greeter() {
return
Flow.<Message>create()
.collect(new JavaPartialFunction<Message, Message>() {
#Override
public Message apply(Message msg, boolean isCheck) throws Exception {
if (isCheck) {
if (msg.isText()) {
return null;
} else {
throw noMatch();
}
} else {
return handleTextMessage(msg.asTextMessage());
}
}
});
}
public static TextMessage handleTextMessage(TextMessage msg) {
if (msg.isStrict()) // optimization that directly creates a simple response...
{
return TextMessage.create("Hello " + msg.getStrictText());
} else // ... this would suffice to handle all text messages in a streaming fashion
{
return TextMessage.create(Source.single("Hello ").concat(msg.getStreamedText()));
}
}
//#websocket-handler
}

Addressing your 2 bullet points below:
1 - you need to attach your metrics to the Message flow - and not to the HttpRequest flow - to effectively count the active connections. You can do this by using watchTermination. Code example for the handleRequest method below
public static HttpResponse handleRequest(HttpRequest request) {
HttpResponse result;
if (request.getUri().path().equals("/greeter")) {
final Flow<Message, Message, NotUsed> greeterFlow = greeter().watchTermination((nu, cd) -> {
connections.incrementAndGet();
cd.whenComplete((done, throwable) -> connections.decrementAndGet());
return nu;
});
result = WebSocket.handleWebSocketRequestWith(request, greeterFlow);
} else {
result = HttpResponse.create().withStatus(413);
}
return result;
}
2 - for the server to independently send messages you could create its Message Flow using Flow.fromSinkAndSource. Example below (this will only send one message):
public static Flow<Message, Message, NotUsed> greeter() {
return Flow.fromSinkAndSource(Sink.ignore(),
Source.single(new akka.http.scaladsl.model.ws.TextMessage.Strict("Hello!"))
);
}

In the handleRequest method you increment and then decrement the counter connections, so at the end the value is always 0.
public static HttpResponse handleRequest(HttpRequest request) {
...
connections.incrementAndGet();
...
connections.decrementAndGet();
return result;
}