We have a requirement wherein we have to develop service(s) that will consume an inbound XML message from MQ, parse that XML message as it has the data for the outbound messages and then publishes multiple outbound messages (count be 1k to 5k) to destination/outbound queue.
Since we would be receiving multiple inbound messages from source queue and for each inbound message we will generate and publish multiple outbound messages all in 1 transaction, we have to ensure that the transaction completes in minimal time - Either we can do this all in one microservice but then if there's some slowness in cloud envs (either MQ or service) we could face timeout issues or we can create multiple microservices wherein transactions will be spanned across, we can persist the messages in DB and all.
Let me know your views
In my opinion, transactions are a good indication to determine the cut of microservices. Distributed transactions have a very high complexity and some disadvantages. Better leave a transaction inside one deployment artifact.
An alternative approach could be horizontal scaling. If you have inbound and outbound queues, it is quite possible to deploy a single service in multiple instances to handle the load.
My advice would be to start small, with one service. This can be designed as a modulith from the beginning, so it can still be split into multiple services later without much effort if there are reasons to do so.
I code all my micro-service in java. I want use Multiple Consumers with Amazon SQS but each consumer has multiple instances on AWS behind a load balancer.
I use SNS for input stream
I use SQS Standard Queue after SNS.
I find the same question on stackoverflow (Using Amazon SQS with multiple consumers)
This sample is
https://aws.amazon.com/fr/blogs/aws/queues-and-notifications-now-best-friends/
When I read SQS Standard Queue documentation, I see that occasionally more than one copy of a message is delivered.:
Each message has a message_id. How to detect that there are not multiple instances of a same micro-service processes the same message that would have been sent multiple times. I got an idea by registering the message_id in a dynamodb database but if this is done by several instances of the same micro-service, how to make a lock on the get (a bit like a SELECT FOR UPDATE)?
for example multiple instances of a same micro-service "Scan Metadata".
As you have mentioned, standard SQS queues can sometimes deliver the same message more than once. This is due to the distribute nature of SQS service. Each message is stored on multiple servers for redundancy and there is a change that one of those servers is down when you are calling sqs:DeleteMessage, therefore the message will not be deleted from all of the servers and once the failed server comes back online, it doesn't know that the you have deleted the message and it will be processed again.
Easiest way to solve the issue with duplicate messages is to switch to using FIFO queue which provides you with exactly once processing. You can choose to use deduplication based on either content or unique ID generated by sender. If you choose to use content deduplication, when queue receives two messages with the same content in 5 min. deduplication interval, the message will be discarded.
If two messages can have the same content yet you need to treat them as different messages, you can use deduplication based on ID that you can pass to sqs:SendMessage or sqs:SendMessageBatch calls via MessageDeduplicationId argument.
I would definitely check FIFO queues before thinking about using DynamoDB to store the state of message processing. It will be cheaper and this deduplication functionality is provided for you by default without you having to implement any complex logic.
I am trying to do event scouring with Apache Camel.
For messaging bus I am using ActiveMQ.
Use cases
I want to audit each messages that are pushed to ActiveMQ using MongoDB as persistent storage. I have tried with mirrored queues in ActiveMQ. This pushes the message to a topic with the same name as queue.
But I have to implement worker based (load balancing) approach. This is not possible with topic (message duplication not allowed).
So I planned to use ActiveMQ with Camel by using the wiretap pattern.
Desired output:
Can I pull the message from wiretap destination and insert it into MongoDB or is there a way that can Camel directly insert it into MongoDB?
One possible way to tackle this on the broker side is with Composite Destinations. You can instruct the broker to forward messages sent to a Queue on to another Queue. Some care needs to be taken when doing this as by default this only happens when the Queue exists (static configuration of destinations can get round this). There is an option to always forward and you also have the option of applying selectors to reduce what gets sent. The thing to keep in mind is that unless you have something periodically purging the audit queue you will eventually run out of space.
You can configure the forwarding as follows:
<compositeQueue name="myQueue" forwardOnly="false">
<forwardTo>
<queue physicalName="myAuditQueue" />
</forwardTo>
</compositeQueue>
I need to implement a demo system for prove of concept.
Basically, the system description can be reduced to 2 modules:
Module 1 sends requests
Module 2 picks them up, processes and sends the response back
(Note: the modules reside in the same intranet, so I probably want the protocol to be faster than http.
I thought of the following options:
Message Queue
ESB
Protobuf
Ideally, the system would be (but not limited to) java-based, run on Linux RH and be able to scale linearly.However, the performance is out of the scope for the POC.
I was looking at ServiceMix and ActiveMQ.
My idea was to implement in java theses modules. The architecture will be message-driven. The modules will communicate over message queue or the service bus.
The 'consumer' sends the requests as messages to the message queue, the 'producer' picks them up by certain subscription topic, processes the requests and posts the response back to the same queue. The 'consumer' that is subscribed on 'response' topic picks the results from the queue. END.
My questions are:
What are other good options (protocols, architecture, existing libraries) to be considered in order to implement the above functionality?
In order to achieve the above I tried to look at ServiceMixESB User Guide but it seems like in order to get something like above running I got to learn bunch of stuff I am not familiar with: JBI, NMR, Karaf, Camel etc and I do not have time to do it. So, I wonder: is there any quick start guide or java sample code for ESB/Message Queue 'Hello World' application that could help just set everything in motion?
ActiveMQ with XML messages should be enough, unless your messages are big and lots of them, in which case I would go for protobuf(disclaimer: I used them on the last project).
As a matter of fact I would probably go for some amqp implementation, like Apache Qpid(disclaimer: used that also some time ago) over ActiveMQ. But this is more a personal reason.
The downside of protobuf is that you need some knowledge about them, there are hello worlds all around the web, but once you try to face 'real problems' it does not get too easy.
You will also need a maven plugin to build and compile the files, unless you want to do it manually.
ActiveMQ is simply a JMS Provider, and I am sure you already looked at this examples:
Hello World ActiveMQ
On an implementation side, when module1 sends the request you want to be sure that the response will be read by the same module. Temporary queues is what I would suggest. Send the request to some queue (and also the temporary queue name for example that the response is expected to come to); module2 processes the message and sends the response to the temporary queue, where it is read by module1 with a message listener.
Now, you have to delete this temporary queues really fast so that don't pile up, and also check that ActiveMQ provides them with unique names.
In QPID with a simple parameter auto-delete=true, when there are no active listeners the queue is deleted, I have no idea how that is handled in ActiveMQ, but there should be a way.
Just my 0.02$
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I was just reading abit about JMS and Apache ActiveMQ.
And was wondering what real world use have people here used JMS or similar message queue technologies for ?
JMS (ActiveMQ is a JMS broker implementation) can be used as a mechanism to allow asynchronous request processing. You may wish to do this because the request take a long time to complete or because several parties may be interested in the actual request. Another reason for using it is to allow multiple clients (potentially written in different languages) to access information via JMS. ActiveMQ is a good example here because you can use the STOMP protocol to allow access from a C#/Java/Ruby client.
A real world example is that of a web application that is used to place an order for a particular customer. As part of placing that order (and storing it in a database) you may wish to carry a number of additional tasks:
Store the order in some sort of third party back-end system (such as SAP)
Send an email to the customer to inform them their order has been placed
To do this your application code would publish a message onto a JMS queue which includes an order id. One part of your application listening to the queue may respond to the event by taking the orderId, looking the order up in the database and then place that order with another third party system. Another part of your application may be responsible for taking the orderId and sending a confirmation email to the customer.
Use them all the time to process long-running operations asynchronously. A web user won't want to wait for more than 5 seconds for a request to process. If you have one that runs longer than that, one design is to submit the request to a queue and immediately send back a URL that the user can check to see when the job is finished.
Publish/subscribe is another good technique for decoupling senders from many receivers. It's a flexible architecture, because subscribers can come and go as needed.
I've had so many amazing uses for JMS:
Web chat communication for customer service.
Debug logging on the backend. All app servers broadcasted debug messages at various levels. A JMS client could then be launched to watch for debug messages. Sure I could've used something like syslog, but this gave me all sorts of ways to filter the output based on contextual information (e.q. by app server name, api call, log level, userid, message type, etc...). I also colorized the output.
Debug logging to file. Same as above, only specific pieces were pulled out using filters, and logged to file for general logging.
Alerting. Again, a similar setup to the above logging, watching for specific errors, and alerting people via various means (email, text message, IM, Growl pop-up...)
Dynamically configuring and controlling software clusters. Each app server would broadcast a "configure me" message, then a configuration daemon that would respond with a message containing all kinds of config info. Later, if all the app servers needed their configurations changed at once, it could be done from the config daemon.
And the usual - queued transactions for delayed activity such as billing, order processing, provisioning, email generation...
It's great anywhere you want to guarantee delivery of messages asynchronously.
Distributed (a)synchronous computing.
A real world example could be an application-wide notification framework, which sends mails to the stakeholders at various points during the course of application usage. So the application would act as a Producer by create a Message object, putting it on a particular Queue, and moving forward.
There would be a set of Consumers who would subscribe to the Queue in question, and would take care handling the Message sent across. Note that during the course of this transaction, the Producers are decoupled from the logic of how a given Message would be handled.
Messaging frameworks (ActiveMQ and the likes) act as a backbone to facilitate such Message transactions by providing MessageBrokers.
I've used it to send intraday trades between different fund management systems. If you want to learn more about what a great technology messaging is, I can thoroughly recommend the book "Enterprise Integration Patterns". There are some JMS examples for things like request/reply and publish/subscribe.
Messaging is an excellent tool for integration.
We use it to initiate asynchronous processing that we don't want to interrupt or conflict with an existing transaction.
For example, say you've got an expensive and very important piece of logic like "buy stuff", an important part of buy stuff would be 'notify stuff store'. We make the notify call asynchronous so that whatever logic/processing that is involved in the notify call doesn't block or contend with resources with the buy business logic. End result, buy completes, user is happy, we get our money and because the queue is guaranteed delivery the store gets notified as soon as it opens or as soon as there's a new item in the queue.
I have used it for my academic project which was online retail website similar to Amazon.
JMS was used to handle following features :
Update the position of the orders placed by the customers, as the shipment travels from one location to another. This was done by continuously sending messages to JMS Queue.
Alerting about any unusual events like shipment getting delayed and then sending email to customer.
If the delivery is reached its destination, sending a delivery event.
We had multiple also implemented remote clients connected to main Server. If connection is available, they use to access the main database or if not use their own database. In order to handle data consistency, we had implemented 2PC mechanism.
For this, we used JMS for exchange the messages between these systems i.e one acting as coordinator who will initiate the process by sending message on the queue and others will respond accordingly by sending back again a message on the queue.
As others have already mentioned, this was similar to pub/sub model.
I have seen JMS used in different commercial and academic projects. JMS can easily come into your picture, whenever you want to have a totally decoupled distributed systems. Generally speaking, when you need to send your request from one node, and someone in your network takes care of it without/with giving the sender any information about the receiver.
In my case, I have used JMS in developing a message-oriented middleware (MOM) in my thesis, where specific types of object-oriented objects are generated in one side as your request, and compiled and executed on the other side as your response.
Apache Camel used in conjunction with ActiveMQ is great way to do Enterprise Integration Patterns
We have used messaging to generate online Quotes
We are using JMS for communication with systems in a huge number of remote sites over unreliable networks. The loose coupling in combination with reliable messaging produces a stable system landscape: Each message will be sent as soon it is technically possible, bigger problems in network will not have influence on the whole system landscape...