Develop Reference

Preserve messages during unprecedented JVM crash in a high throughput system

I am building a high volume system that will be processing up to a hundred million messages everyday. I have a microservice that is reading from a Kafka topic and doing some basic processing on them before forwarding them to the next microservice.
Kafka Topic -> Normalizer Microservice -> Ordering Microservice
Below is what the processing would look like:
Normalizer would be concurrently picking up messages from the Kafka topic.
Normalizer would read the messages from the topic and post them to an in-memory seda queue from where the message would be subsequently picked up, normalized and validated.
This normalization, validation and processing is expected to take around 1 second per message. Within this one second, the message will be stored to the database and will become persistent in the system.
My concern is that during this processing, if a message has been already read from the topic and posted to the seda queue and has either
not yet been picked up from the seda queue or,
has been picked up from the seda queue and is currently processing and has not yet been persisted to the database
and the Normalizer JVM crashes or is force-killed (kill -9), how do I ensure that I do NOT lose the message?
It is critical that I do NOT drop/lose any messages and even in case of a crash/failure, I should be able to retain the message such that I can trigger re-processing of that message if required.
One naïve approach that comes to mind is to push the message to a cache (which will be a very fast operation).
Read from topic -> Push to cache -> Push to seda queue
Needless to say, the problem still exists, it just makes it less probable that I will lose the message. Also, this is certainly not the smartest solution out there.
Please share your thoughts on how I can design this system such that I can preserve messages on my side once the messages have been read off of the Kafka topic even in the event of the Normalizer JVM crashing.

Is RabbitMQ or Kafka message queue a 1:1 messaging system?

As mentioned in the answer,
A message queue is a one-way pipe: one process writes to the queue, and another reads the data in the order
SysV message queue is one example
So, my understanding is,
one message queue is used by two processes, where one process(producer) insert an item in the queue and another process(consumer) consumes the item from the queue
1) Is RabbitMQ or Kafka message queue a 1:1 messaging system? used by only two processes, where one process writes and other process reads......
2) after the consumer consume the item, does the item get deleted? If no, why do we need queue data structure? Why not just shared memory?

Kafka is not strictly 1:1 messaging system. Multiple producers can write into a topic and multiple consumers can read from it. Moreover, in Kafka, multiple consumers can be assigned same or different consumer groups. Every message is consumed by only one consumer from every consumer group (load balancing) and all consumer groups receive a copy of every message (of course, if they are subscribed to corresponding topics and no messages are lost). A good description of this process can be found in this article: Scalability of Kafka Messaging using Consumer Groups.
In Kafka all messages are persisted on the disk and stored until the compaction reaps it, or the retention.ms passes, or the log size is exceeded. That's a very high-level point of view and there are a lot of nuances here. Like: the messages are stored in segments, every segment contains multiple messages. When the retention period passes for a message, it is not removed from the segment at that moment, instead Kafka waits until all messages in that segment are expired and delete the whole segment at once. Also, retention could come before the log exceeds the maximum size or vice versa: the log can exceed the size even before the retention period passes. And so on. Just read the docs and pay attention to topics about "log cleaner" and "retention".
After the Kafka consumer reads the message it is neither compacted, nor expired. So, it's not removed from the log and stays there. It also means that every message could be re-read by a consumer if needed (until it is deleted completely). It can be useful if some of your consumers went offline for some reason and were not able to process the messages as they come in. It also allows interesting features like transaction replays and so on. Persistence is one of the Kafka's features.
Shared memory? Well, strictly speaking shared memory is only allowed inside a single process. So you can't generally use "shared memory" when you need to access it from different processes. And there is absolutely no way to have "shared memory" when you app runs on multiple hosts. However, there are in-memory brokers. Like Redis can be used as a message broker, and it's all in-memory. However, if such a broker restarts for some reason you lose everything. Speaking about Redis: it has two persistence configurations specifically to handle the restarts.
I am not sure about RabbitMQ, but it probably deletes messages after the consumer acknowledged them by default. So it's closer to 1:1 mental model. However, RabbitMQ employs disk persistence as well.

Strategies for keeping and locking messages in a queue with one consumer hiding multiple users

I am trying to implement the following scenario and I could really use and appreciate some help. I am using ActiveMQ 5.14 with camel 2.21.
In the queue, each message corresponds to a single machine. The machines connect to the queue through a single polling consumer and are indistinguishable to the consumer. The messages should be kept in the queue until one machine acknowledges that it has reached the correct machine via a separate request. After each fetch of a message said message should be locked for a certain time.
I could not find any ActiveMQ functionality that translates to my problem. My approach would be to send the message after each fetch to a second queue, which serves as a lock mechanism and send it back to the fetchable queue after the specified timeout.
Maybe a better approach would be to rollback the session after each fetch if the message has not been acknowledged by the machine.
Do you have any suggestions what a viable solution to this problem would look like?
edit: more details to clarify the situation
The application communicates to the clients via exposing a REST API to the web with two calls: GET and DELETE.
GET fetches the next message from the queue and DELETE deletes the message from the queue. I need to make sure that a message is only fetched once in a given time period and that it makes its way back to the queue if the client doesn't send a DELETE request. Currently I have a route from the rest service to a bean which fetches a message from the queue returns it to the GET request and sends it back to the queue after. On a DELETE request I dequeue the message from the queue with the given id.
I still need to find a way to ensure that the last fetched message cant be accessed for a specified time period.

I am a bit confused about the part with the indistinguishable machines, but I understood the following:
You have 1 queue with messages
You have 1 consumer
The consumer takes a message and calls a service or similar
If the call is successful the message can be deleted
If the call fails the message must be reprocessed
If these assumptions are correct, you can build a Camel route that consumes messages from the queue (transacted) and calls the service.
If the Camel route fails to complete (service returns error) and the error is not handled, the broker does a rollback and the message is redelivered (immediately)
If the route fails multiple times (when max redelivery value is reached), the message is sent to the dead letter queue (by the broker) to move it out of the way but save it
If the route succeeds the message consumption is committed to the broker and the message deleted
In such a setup you could also configure more consumers to process the messages in parallel (if the called service allows this)
This behaviour is more or less the default behaviour if
Your broker is configured as persistent (avoid message loss)
You consume messages transacted (a local transaction with the broker is enough)
Camel does not handle the errors (when they are handled, the message is committed because the broker does not "see" any error)
You get an error from the service or you can at least decide if there was a problem and throw the error yourself. The broker must get an Exception so that a rollback is done
EDIT
Based on your clarification I understand that it is the other way round than I assumed.
Well then I would probably see the two request types as "workflow steps" since they are triggered from the clients.
GET
Consume a message, send it to requestor
Add a timestamp to the message header
Send the message to another queue (let's call it delievered)
DELETE
Dequeue the message from the delievered queue
Not deleted messages
Use the timestamp header and message selectors to consume not deleted messages after a certain amount of time
Move them back to the source queue
With a second queue you have various advantages
Messages in processing cannot be consumed again and therefore need no "lock"
The source queue contains only waiting messages, the delievered queue only messages in processing
You could increase message priority when sending not deleted messages back to the source queue so they are re-consumed fast
You could also add a counter header when sending not deleted messages back to the source queue to identify messages that are failed multiple times and process them in another way.

Multithreaded JMS application

I am working on a multithreaded JMS receiver and publisher code.
XML message is received from a Queue, stored procedures(takes 70 sec to execute) are called and response is send to Topic within 90 sec.
I need to handle a condition when broker is down.
i.e. a condition in which messages are received from Queue and are being processed in java, in the mean time both Queue and Topic will be down. Then how to handle those messages which are not on queue and not send to topic but are in java memory?
Different options available:
1.To use CLIENT_ACKNOWLEDGE
2.To separate publisher code from receiver code.
3.To have error utility which will take messages from log and process them and send to Topic(least preferred)
Please suggest me the right option

Use a transacted session. Consume the message and send the response under a single unit of work and explicitly call COMMIT after sending the response. Then if the broker dies while the transaction is outstanding the input message will be rolled back. If you include the DB update in a two-phase coordinated transaction then it too can be rolled back of the broker goes down. This requires the consumer and responder to be within the same thread because JMS scopes sessions per thread, but you can have several threads running sessions in parallel.
Be aware that keeping many transactions open for 90 seconds might require some tuning at the broker side.

One solution is to use db for intermediate stored procedure. So the steps would be:
1. Consume Message from Queue and store into db and use Client_Acknowledge.
2. Run stored procedures on the consumed message.
3. Once the procedure is over, send message from db to the topic.
4. Delete message once acknowledgement is recieved.
If queue and topic goes down in between, you just need to send message again when acknowledgement is not recieved from topic.
Well I am not sure whether this is the best alternative and want to see how community responds on this question.

Way to place a bad request in a dead-letter queue when using JAX-RPC?

I'm using JAX-RPC 1.1 to generate services for a set of applications running on WAS 6.0, communicating via Websphere MQ. A mainframe will be sending messages to the services, so if for some reason the message can't be converted to objects (likely due to EBCDIC-to-ASCII weirdness), the message needs to be placed on the dead-letter queue for examination.
Does anyone know if there's a standard way to do this (i.e. by providing the jms:/ address with a DLQ name), or if DLQ-forwarding would need to be performed manually somehow?

First, let me recommend that you do not rely on the DLQ for this. The DLQ is a QMgr-level resource that is used to store messages that the channel cannot resolve. Because it is a potential attack vector, most security-conscious shops will not grant access to applications, or if they do it is put-only access.
The best way to do this is to create an app-specific exception queue. If the app has several input queues, they can all use the same exception queue and then the application support team can manage any messages that land there without the security issues.
Whichever way you go, getting the app to honor this is extremely easy. For example, suppose your app is reading from JAX.SVC.REQUEST. You define the exception queue and point the BOQ* fields at it:
DEF QL(JAX.SVC.EXCEPTION)
ALTER QL(JAX.SVC.REQUEST) BOQNAME(JAX.SVC.EXCEPTION) BOQTHRESH(5)
The JMS classes will inquire on the BOQ* attributes of the queue when it is opened and check the backout count of each message read. In your program use a transacted session and if you can't process the message, call the session.backout() method. The message will be read and backed out no more than BOTHRESH times, then requeued to the queue named in BOQNAME. If that queue is full or unavailable the DLQ will be tried. If that fails, the classes will throw an exception.
We pick a BOTHRESH > 1 to allow for the possibility that a processable message will be backed out if, for example, the QMgr is shut down.
I usually trigger the exception queue so that it can raise an alarm or send an email if something lands on it. If you have a monitoring tool you can check for depth > 0 instead.
If for some reason yo don't want to use the JMS functionality to automatically requeue the message, the application would need logic to requeue it. If you were to place the message on the DLQ you will want to prepend a DLQ header. Failure to do so will probably break the DLQ handler or any other instrumentation watching that queue.