I need some design and developments inputs on reading messages from queue. i have following requirements and constraints
i need read message from queue and inert to db.
messages can come at any interval (100's at same time or 1 by one with few mins gap)
don't have any MDB container to host (just plain tomcat server)
Need to write java application to perform the above.
so not very sure how to put this simple application.
if is use quartz scheduler to trigger job to read all messages in the queue then not sure before even that complete next instance of scheduler might start and create problem.
please suggest me any inputs.
this is basically some utility so i don't want to spend too long time nor too much resources on this.
thanks & regards
LR
The usage of an ESB like Mule or Camel would simplify a lot your development. You'd find already developed components (called endpoints) for reading from a queue, and writing into a db. Also for scheduling jobs with quartz.
Related
Apologies for the wording of my question.
I am using Tomee.
I have an ActiveMQ queue set up and receiving messages from a Producer (the Tomee provided example)
It is persisted in MySQL (in case that matters)
My scenario is this...
A message comes into the queue
A Consumer/Monitor reads the message and starts a thread to run a process (backup, copying, processing etc...) that could take some time to complete.
At anyone time I could have 5 messages to process or 500+ (and anything in between)
Ideally, I would like some Java/Apache library that is designed to monitor the queue and read 10 messages (for example) and then start the threads and then wait for one to finish before starting any more. For all intents and purposes I am trying to create a 'thread pool' or 'work queue' that prevents too many processes from starting up at any one time.
OR
Does this need to be thread pooled outside of ActiveMQ ?
I'm new to JMS and am beginning to understand it but still a long way to go.
Any help is appreciated.
Trevor
What you are looking to do sounds like something that could easily be solved using Apache Camel. Take a look at the Camel documentation for the competing consumers EIP which sounds like an ideal fit for your case.
I'm trying to write a Spring web application on a Weblogic server that makes several independent database SELECTs(i.e. they can safely be called concurrently), one of which takes 15 minutes to execute.
Once all the results are fetched, an email containing the results will be sent to a user list.
What's a good way to get around this problem? Is there a Spring library that can help or do I go ahead and create daemon threads to do the job?
EDIT: This will have to be done at the application layer (business requirement) and the email will be sent out by the web application.
Are you sure you are doing everything optimally? 15 minutes is a really long time unless you have a gabillion rows across dozens of tables and need a heckofalot of joins....this is your highest priority -- why is it taking so long?
Do you do the email job at set intervals, or is it invoked from your web app? If set intervals, you should do it in an outside job, possibly on another machine. You can use daemons or the quartz scheduler.
If you need to fire this process off from the web app, you need to do it asynchronously. You could use JMS, or you could just have a table into which you enter a new job request, with daemon process that looks for new jobs every X time period. Firing off background threads is possible, but its error prone and not worth the complication, especially since you have other valid options that are simpler.
If you are asking about Spring support for long-running, possibly asynchronous tasks, you have a choice between Spring JMS support and Spring Batch.
You can use spring quartz to schedule the job. That way the jobs will run in the same container but will not require an http request to trigger them.
I'm looking for an existing system to replace an existing slow and complicated self written mechanism of job management.
The existing system:
1 MySQL DB with a long massive table of jobs - the queue
Multiple servers (written in java) all extracting jobs from the queue and processing them
a job might NOT be deleted from the queue after processing it, to rerun it later
a job might create other jobs and insert them to the queue
The limitations:
As more and more jobs are created and inserted in to the queue, it takes longer to extract jobs from it. (The jobs are chosen by priority and type) - create a bottle neck
I'm looking for an existing system that can replace this one, and improve it's performance.
Any suggestions?
Thanks
I don't generally recommend JMS, but it sounds like it really is what you need here. Distributed, transactional, persistent job queue management is what JMS is all about.
Popular open-source implementations include HornetQ and ActiveMQ.
You could:
submit your jobs to Amazon's Simple Queue Service (maybe JAXB marshalled)
dynamically start some EC2 instances according to your queue's length and probably
submit the results (or availability notice for some files on S3) to Simple Notification Service (again JAXB marshalled).
That exactly what we do, using EC2 Spot instances to minimize costs. And that's what I call serious cloud computing ;)
I have a java class Processor that is listening to a jms topic and it is struggling to keep up with the speed in which messages are arriving so we've decided to go concurrent:
A single class listening to the topic who's job is to hand the messages out to a pool of worker threads, effectively being a load balancer. It also has to prevent 2 workers processing messages for the same customer.
I expected there to be quite a lot of information on the internet about this but everything seems to suggest the use of EJBs where the app server manages the pool and does the balancing. I'm sure this must be a really common problem, but can't seem to find any libraries or design patterns to assist. Am I making more out of it than it is and should just delve in and write my own code?
Why don't you just use a queue instead of a topic and have several instances of the same application handle messages from this queue ?
This is an easy problem to solve with a pool of listeners. That's what the app server would be doing for you.
I'd get a good app server and use its MDBs to solve this quickly. Size the pool to keep up and you'll be fine.
If you insist on writing your own code, get a good open source pool implementation and use it.
If it must be non-EJB, consider Spring. It has message driven POJOs that could be just what you need.
I need to design a Swing application, which will need to send out multiple jobs as customer requested. each job is running the same shell scripts which will take 10-30 mins to return a value. (the jobs are not running on application server or as web services. )then the Swing application will need to decide what to do next according to the return value.
my question is if I can use JMS to send out jobs. if not, what do you suggest I look into?
multithreading....
Thank you very much!
Multi-threading is the obvious first approximation here. Take a look at SwingWorker, launch the process in a background thread, monitor the progress (as in show the user if it is still running, perhaps even a view into what is being emitted to the console), etc. These are the obvious choices.
What JMS would solve for you (and you would have to find a light weight JMS implementation that would run on the desktop) is to allow for retries and guarantees that the process runs to completion. Something that takes 20 minutes to run in a shell script doesn't sound like it is a candidate for a retry, but if it is, and it is important that the message really get through instead of just having the thread die and the process forgotten if the user closes the java application, then JMS is the type of thing to look at.
JMS can most certainly be used in a Swing based application. If the shell scripts are to be executed on a server by a service that is listening to the JMS queue and respond on another queue or topic.
There is nothing limiting you from using JMS queues or topics in a desktop application.
JMS is generally used to communicate between processes and between client/server, not really what you're looking for here, unless you're sending them out to a server to be processed, but it doesn't sound like that here. It sounds like you're looking for a work queue, such that a swing app has a button that adds a new task to the queue (where the task is running the shell script). You can then have multiple threads taking tasks of the queue and running the scripts.
You may - or may not - profit from using a job scheduler, like Quartz. Maybe it's overkill, maybe it's just what you need.