Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I was just reading abit about JMS and Apache ActiveMQ.
And was wondering what real world use have people here used JMS or similar message queue technologies for ?
JMS (ActiveMQ is a JMS broker implementation) can be used as a mechanism to allow asynchronous request processing. You may wish to do this because the request take a long time to complete or because several parties may be interested in the actual request. Another reason for using it is to allow multiple clients (potentially written in different languages) to access information via JMS. ActiveMQ is a good example here because you can use the STOMP protocol to allow access from a C#/Java/Ruby client.
A real world example is that of a web application that is used to place an order for a particular customer. As part of placing that order (and storing it in a database) you may wish to carry a number of additional tasks:
Store the order in some sort of third party back-end system (such as SAP)
Send an email to the customer to inform them their order has been placed
To do this your application code would publish a message onto a JMS queue which includes an order id. One part of your application listening to the queue may respond to the event by taking the orderId, looking the order up in the database and then place that order with another third party system. Another part of your application may be responsible for taking the orderId and sending a confirmation email to the customer.
Use them all the time to process long-running operations asynchronously. A web user won't want to wait for more than 5 seconds for a request to process. If you have one that runs longer than that, one design is to submit the request to a queue and immediately send back a URL that the user can check to see when the job is finished.
Publish/subscribe is another good technique for decoupling senders from many receivers. It's a flexible architecture, because subscribers can come and go as needed.
I've had so many amazing uses for JMS:
Web chat communication for customer service.
Debug logging on the backend. All app servers broadcasted debug messages at various levels. A JMS client could then be launched to watch for debug messages. Sure I could've used something like syslog, but this gave me all sorts of ways to filter the output based on contextual information (e.q. by app server name, api call, log level, userid, message type, etc...). I also colorized the output.
Debug logging to file. Same as above, only specific pieces were pulled out using filters, and logged to file for general logging.
Alerting. Again, a similar setup to the above logging, watching for specific errors, and alerting people via various means (email, text message, IM, Growl pop-up...)
Dynamically configuring and controlling software clusters. Each app server would broadcast a "configure me" message, then a configuration daemon that would respond with a message containing all kinds of config info. Later, if all the app servers needed their configurations changed at once, it could be done from the config daemon.
And the usual - queued transactions for delayed activity such as billing, order processing, provisioning, email generation...
It's great anywhere you want to guarantee delivery of messages asynchronously.
Distributed (a)synchronous computing.
A real world example could be an application-wide notification framework, which sends mails to the stakeholders at various points during the course of application usage. So the application would act as a Producer by create a Message object, putting it on a particular Queue, and moving forward.
There would be a set of Consumers who would subscribe to the Queue in question, and would take care handling the Message sent across. Note that during the course of this transaction, the Producers are decoupled from the logic of how a given Message would be handled.
Messaging frameworks (ActiveMQ and the likes) act as a backbone to facilitate such Message transactions by providing MessageBrokers.
I've used it to send intraday trades between different fund management systems. If you want to learn more about what a great technology messaging is, I can thoroughly recommend the book "Enterprise Integration Patterns". There are some JMS examples for things like request/reply and publish/subscribe.
Messaging is an excellent tool for integration.
We use it to initiate asynchronous processing that we don't want to interrupt or conflict with an existing transaction.
For example, say you've got an expensive and very important piece of logic like "buy stuff", an important part of buy stuff would be 'notify stuff store'. We make the notify call asynchronous so that whatever logic/processing that is involved in the notify call doesn't block or contend with resources with the buy business logic. End result, buy completes, user is happy, we get our money and because the queue is guaranteed delivery the store gets notified as soon as it opens or as soon as there's a new item in the queue.
I have used it for my academic project which was online retail website similar to Amazon.
JMS was used to handle following features :
Update the position of the orders placed by the customers, as the shipment travels from one location to another. This was done by continuously sending messages to JMS Queue.
Alerting about any unusual events like shipment getting delayed and then sending email to customer.
If the delivery is reached its destination, sending a delivery event.
We had multiple also implemented remote clients connected to main Server. If connection is available, they use to access the main database or if not use their own database. In order to handle data consistency, we had implemented 2PC mechanism.
For this, we used JMS for exchange the messages between these systems i.e one acting as coordinator who will initiate the process by sending message on the queue and others will respond accordingly by sending back again a message on the queue.
As others have already mentioned, this was similar to pub/sub model.
I have seen JMS used in different commercial and academic projects. JMS can easily come into your picture, whenever you want to have a totally decoupled distributed systems. Generally speaking, when you need to send your request from one node, and someone in your network takes care of it without/with giving the sender any information about the receiver.
In my case, I have used JMS in developing a message-oriented middleware (MOM) in my thesis, where specific types of object-oriented objects are generated in one side as your request, and compiled and executed on the other side as your response.
Apache Camel used in conjunction with ActiveMQ is great way to do Enterprise Integration Patterns
We have used messaging to generate online Quotes
We are using JMS for communication with systems in a huge number of remote sites over unreliable networks. The loose coupling in combination with reliable messaging produces a stable system landscape: Each message will be sent as soon it is technically possible, bigger problems in network will not have influence on the whole system landscape...
Related
I came across the concept called webhook and publisher/subscriber. In webhook the third party application send the information whenever the updation happened in the depend application, thirdparty will send a HTTP post request to the mention URL for your application, and In publisher and subscriber, subscriber will register the topic and publisher write on that topic then registrar(like the third party) will send the information to the subscriber based on the topic subscribed.
both are similar or difference?
I am confused can anyone give solve to this?
Well, conceptually both of these methods were used to notify a client when an event occurs.
But practically I would use them in two different scenarios.
The webhook (from Wikipedia):
A webhook in web development is a method of augmenting or altering the behavior of a web page, or web application, with custom callbacks. These callbacks may be maintained, modified, and managed by third-party users and developers who may not necessarily be affiliated with the originating website or application. The term "webhook" was coined by Jeff Lindsay in 2007 from the computer programming term hook.
The webhook approach is relevant when you want to communicate asynchronous changes or updates between a third party to your backend server.
That means that the 3rd party needs to register the webhook address for each client and trigger an http request with the information needed to be communicated.
Some of the considerations using webhook is that the failure handling in case the webhook address isn't responding or for any given temporary failure, the retry responsibility and handling is done by the publisher.
Here are a couple of examples of when webhook approach is used:
- SendGrid.com - an email service that let's you send emails and campaign emails through an api. One example when you would like to expose a webhook on your backend would be if you want to get notified every time a user unsubscribes from a list.
- Braintree.com - a billing gateway the lets you charge your customers for products they buy on your website - an example would be a webhook you expose on your backend in order to get a notification every time a recurring payment was successfully executed
When it comes to publisher/subscriber this is more of a messaging pattern (from Wikipedia) :
In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead categorize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.
Advantages
Loose coupling
Publishers are loosely coupled to subscribers, and need not even know of their existence. With the topic being the focus, publishers and subscribers are allowed to remain ignorant of system topology. Each can continue to operate normally regardless of the other. In the traditional tightly coupled client–server paradigm, the client cannot post messages to the server while the server process is not running, nor can the server receive messages unless the client is running. Many pub/sub systems decouple not only the locations of the publishers and subscribers, but also decouple them temporally. A common strategy used by middleware analysts with such pub/sub systems is to take down a publisher to allow the subscriber to work through the backlog (a form of bandwidth throttling).
Scalability
Provides the opportunity for better scalability than traditional client–server, through parallel operation, message caching, tree-based or network-based routing, etc. However, in certain types of tightly coupled, high-volume enterprise environments, as systems scale up to become data centers with thousands of servers sharing the pub/sub infrastructure, current vendor systems often lose this benefit; scalability for pub/sub products under high load in these contexts is a research challenge.
Outside of the enterprise environment, on the other hand, the pub/sub paradigm has proven its scalability to volumes far beyond those of a single data centre, providing Internet-wide distributed messaging through web syndication protocols such as RSS and Atom. These syndication protocols accept higher latency and lack of delivery guarantees in exchange for the ability for even a low-end web server to syndicate messages to (potentially) millions of separate subscriber nodes.
Disadvantages
The most serious problems with pub/sub systems are a side-effect of their main advantage: the decoupling of publisher from subscriber.
I would recommend the following post for further info about pub/sub :
pub-sub-messsaging using aws
Pub sub on Wikipedia
Webhook is a technology implementation for PubSub over HTTP which makes Webhook technology a subset of PubSub. Apart from Webhooks, PubSub could user other means of subscription and publishing (e.g Email).
i am going to integrate some applications using RabbitMQ. Now i am facing the design issue. Right now i am having one application producing message and one application consuming it (in future more are possible). Both applications have access to some database. Application A is some kind of registration application when it receives registration request it sends message to on rabitmq. Now application b receives this message and its task is to load the registration data to elasticsearch server. Now i have some options
consumer will read the message and id from q and load the data and send it to the elastic search server
fastest throughput. Because things will move in asynchronous way. other process which may be running on separate
server will loading the data and sending to elastic server
consumer will read the message and id from the q and then call the rest service to load the company data.
will take more time for processing each request as it will be having network overhead.although it will save time to data load
but will add network delay. And it will by pass the ESB(Message Broker) also. (i personally think if i am using esb in my application
it is not necessary that i use it for every single method call)
send all the registration data in the message. consumer will receive it and just upload it to elasticsearch server.
which approach i should follow?
Apparently there are many components to your application set up that is hard to take into account and suggest a straightforward answer. I would suggest that you should look into each design and identify I/O points, calls over the network and data volume exchanged over the network. Then depending on the load you expect and the volume of data you expect to store over time I would suggest you hierarchize these bottlenecks giving a higher score depending on the severity of it. Identify the one solution that has the lowest score and go with that.
I would suggest you should benchmark the difference between sending only the iq or sending the whole object. I would expect that the difference is negligible.
One suggestion. Make your objects immutable. It is not directly relevant with what you are describing but in situations like yours, where components are operating "blindly" you will find that knowing that an object has not changed state is a big assurance.
I'm creating a monitor application that monitors the activities of a user. There are four elements in my system:
EventCatcher: The EventCatcher is responsible for catching all the events that happen in a subsystem and pushes the data to the EventHandler. Based from observation, there is an average of 10 events per second that is being pushed to the EventHandler. Some events are UserLogin, UserLogout.
EventHandler: The EventHandler is a singleton class that handles all the incoming events from the EventCatcher. It also keeps track of all the logged in users in the system. So, whenever the EventHandler receives a UserLogin event, the User object is extracted from the event and is stored in a HashMap. When a UserLogout event is received, that User object will be remove from the HashMap. This class also maintains a Set of all active Websocket sessions because everytime an event has occurred, I would want to inform all the open sessions that a particular event happened.
Websocket Endpoint: This is just a simple Java class annotated with #ServerEndpoint.
Clients: The system I will be building is for internal (company) use only. At production, at most, there will only be around 5 - 10 clients. All the clients will be receiving the same information every time an event has occurred.
So right now I am trying to convince my supervisor that Websockets is the way to go, however, my supervisor finds it really unnecessary because a simple polling solution would do the trick.
His points are:
We don't really need up-to-date information by the millisecond. We can poll every second.
If I was to maintain a list of open WebSocket sessions, how would that work in a clustered environment (we use a load balancer)
If I plan to send information to the client every time an event (UserLogin, UserLogout) has occurred, I should be able to just send small updates to all WebSocket sessions - meaning, I can't be sending a whole JSON dump of everything. So that means, for every WebSocket instance, I would have to maintain another Set of Users and properly maintain it to mirror the Set contained in the EventHandler.
What my supervisor suggests is that I lose the WebSocket and just convert it to a simple Servlet and let the clients poll every second to receive the entire JSON dump.
In this scenario, should I stick with WebSockets? Or should I just poll?
The main advantage, as far as I've read, of Websockets vs. polling is that by using Websockets, you will have a persistent connection from client to server. HTTP is not really meant for real-time data.
Also, polling requires sending an HTTP request every time and every request comes with HTTP headers. If an HTTP request header contains 800 bytes, then that's 48kb sent per minute per client. With a WebSocket, this isn't problem.
But then again, we won't really have a lot of active clients. We're not concerned about third parties sniffing our requests because this system is for company use only - internal use! And I believe my supervisor wants something simple and reliable.
I am fine with either way. I just want to be sure whether I'm using the right tool for the job.
Additional question: If WebSockets is the way to go, is there any reason why I should consider polling?
The entire purpose of WebSocket is to efficiently support continuing connections between client and server.
I’m not clear on how you are implementing your app. If this is a web app running in a Servlet environment leveraging WebSocket support in the web server, be aware that you need to use recent versions of the Servlet container. For example, with Tomcat you must use either version 8 or the latest updates to version 7.
And of course the web browser must have support for WebSocket.
Be aware that WebSocket is still a new technology that has been changing and evolving in both the specs and the implementations.
Atmosphere
You may want to consider using the Atmosphere framework. Atmosphere supports multiple techniques of Push including WebSocket & Comet.
The Vaadin web-app framework leverages Atmosphere to provide automatic support for Push in your app. By default, WebSocket is automatically attempted first. If WebSocket is not available, Vaadin+Atmosphere falls back automatically to the other techniques including polling.
My app takes care of user registration (with the option to receive email announcements), and can easily handle the actual template-based rendering of email for a given user. JavaMail provides the mail transport layer. But how should I design the application layer between the business objects (e.g. User) and the mail transport?
The straightforward approach would be a simple, synchronous loop: iterate through the users, queue the emails, and be done with it. "Queue" might mean sending them straight to the MTA (mail server), or to an in-memory queue to be consumed by another thread.
However, I also plan to implement features like throttling the rate of emails, processing bounced emails (NDRs), and maintaining status across application restarts. My intuition is that a good design would decouple this from both the business layer and the mail transport layer as much as possible. I wondered if others had solved this problem before, but after much searching I haven't found any Java libraries which seem to fit this problem. Standalone mail apps such as James or list servers are too large in scope; packages like Spring's MailSender or Commons Email are too small in scope (being basically drop-in replacements for JavaMail). For other languages I haven't found anything appropriate either.
I'm curious about how other developers have gone about adding bulk mailing to their applications.
The approach I've been happiest with is to provide an interface to my application to "send" mails. In reality, the implementation of this interface simply queues the mail into a database for later processing. From the application's perspective, this interface is fast, as it performs very little actual work. Plus, the persistence survives server downtime, as you mentioned.
Another thread reads from the queue and takes it's sweet time sending mail up to it's configured throttle, and flags messages in the queue after successfully processing them (effectively dequeuing them without deleting them). This provides both a history of sent mail, and a reference when mail is bounced, etc. I delete from the queue 7 days after a successful send.
In terms of decoupling the solution from the mail transport layer... I've applied this approach to an automated Twitter client, and found it to be equally successful.
One option is to use a hardware appliance. My company uses Strongmail, at least for marketing communications: http://www.strongmail.com/index.php. I don't know much about it, but I think it handles bulk e-mail concerns like do-not-contact lists, throttling, avoiding getting spam filtered, etc.
I will like to know:
I have a scenario. If a user adds a product to the system (I'm developing), there's a listener that sends a notification to the user's client base notifying of a new product added by the user.
I've read this thread and (seeing I've never used JMS nor ThreadPool before) I was wondering whether I should use JMS or ThreadPooling.
I am using Tomcat 5.5 and higher and JBoss 5 and higher (depending on company last resort) to deploy my web application.
If I use JMS, do I use Apache ActiveMQ or JBoss Messaging? Are they both compatible to run on both platforms (Tomcat and JBoss)?
Thanks in advance.
For communicating between applications, JMS is a very good solution, especially for events and notifications. JMS allows for such notifications to be sent and received using what is known as asynchronous messaging whereby the sender and receiver have no knowledge of one another and no requirement to be available at the same time.
ActiveMQ is a very widely used message broker that provides client APIs for Java, C/C++, C#, Perl, PHP, Python, Ruby and more. This allows the use of JMS with applications written in Java and other languages.
I have implemented JMS messaging many, many times for a large variety of business situations to handle events and notifications. The vast majority of these times, I have recommended and/or used Spring JMS no matter what message broker is being used. Spring JMS is incredibly easy to use, extremely robust and highly scalable. Spring JMS removes the complexity of creating your own message producers and message consumers, which can save you a tremendous amount of time.
To see how easy it is to send messages using Spring JMS, check out a blog post I wrote recently titled Using the Spring JmsTemplate to Send JMS Messages. I'm also working on a blog post about receiving messages using Spring JMS.
If you have any further questions, let me know.
Bruce
I had a similar requirement once, and we used JMS. Then main problem was how to deal with errors because SMTP is indeed not transactional:
is it ok if some email are lost?
is it ok if some email are sent twice?
We decided it was better to send the message twice, and here is more or less the design we had:
We relied on container managed transaction and if for some reason the email can not be sent, we decided to rollback the JMS transaction; the message would be redelivered later by JMS and an new attempt to send the message was done.
If the JMS message delivery transaction failed after the email was sent (e.g. because of a problem with JMS), the transaction would be rolled back automatically and the message was redelivered later. In this case, the email was sent twice because STMP is not transactional.
Even if the email can be sent (from point of view of code), the SMTP server can still have problem later. In this case, the JMS have been delivered and consumed, so we had no way to know which email had been processed and how to re-send them manually.
But we were already using JMS. I would not introduce JMS just for that given that the main argument is that JMS is transactional, but SMTP isn't anyway.
I would go for something lighter -- possibly with a ThreadPool -- and store the state in a database to know which email need to be sent or has been sent. If there are some problem, you can look at the database and take the ad-hoc decisions.
I know that this reply is very late to this dicsussion, but I hope it will still be valuable for folks seeking info on integrating ActiveMQ and Tomcat.
I've had many people ask me for help with issues they have had integrating ActiveMQ and Tomcat so I decided to write some articles about it. Not only is this topic covered in ActiveMQ in Action (see chapter 8), but I also published a series of articles on it titled ActiveMQ and Tomcat: Perfect Partners. Hopefully people will find this helpful.
I would go for a persistent JMS (I have used only WLS JMS and Websphere MQ so can't compare AQ vs JBoss, whichever offers a better guarantee for delivery). Also, I would seriously consider making the email engine a completely separate application, depending on how much you expect the traffic to grow.