I am investigating whether or not Quartz can be used for a project I am working on. I need to:
Limit the execution of jobs to specific time ranges (which I know Quartz is great at).
Limit jobs based on "resources".
When I say resources, I referring to both exclusive and quantitative resources. For example, I would like to define a resource something like "LINUX_MACHINE" with a count of 5. Only a maximum of 5 jobs requiring the LINUX_MACHINE machine resource can be run at any one time. Is this possible to do using Quartz?
So it looks like you can limit jobs based on resources by creating multiple schedulers and then limiting the thread pool for each scheduler.
Info on multiple schedulers: http://www.quartz-scheduler.org/documentation/quartz-2.2.x/cookbook/MultipleSchedulers
Info on Quartz config to set the thread pool for the scheduler: http://www.quartz-scheduler.org/documentation/quartz-2.2.x/configuration/ConfigThreadPool
Related
Quartz scheduler is used to schedule timed java jobs at my workplace. The scheduler itself is deployed as an application to a Weblogic servers (a cluster of machines). This scheduler can schedule jobs which implement the Job interface and override the execute() method. These jobs are deployed to the Weblogic servers as libraries which are then used by the scheduler. (One library includes multiple jobs.)
I have not managed to find informative sources on how these jobs are run or how they share resources.
I looked at the Quartz documentation but could not find what I was looking for.
I have multiple questions, though I believe a single answer may cover all of them.
Do all the jobs created through the scheduler share a single JVM? If they don't, then based on what is a job allocated to a given JVM?
I assume that a separate thread is allocated to each job - is that correct?
If all scheduled jobs run in the same JVM and each has its own thread, then concurrent execution of the same job with different parameters will create a need for making the job thread safe or a need to disable concurrent execution of the job, will not it?
Thank you.
You may want to read Quartz scheduler tutorial to find out how Quartz works. To answer your questions:
This depends on whether you run a Quartz scheduler cluster (i.e. multiple Quartz scheduler instances sharing the same job store) or a standalone Quartz scheduler instance. In clustered deployments, individual Quartz scheduler instances compete to execute jobs by creating DB row locks. The scheduler instance that first manages to create the row lock, is the scheduler instance that executes a particular job. In a standalone Quartz scheduler deployment, the completion does not exist and it is always the single Quartz scheduler instance that ends up executing all jobs.
Quartz uses a thread pool and when it needs to execute a job, it simply allocates a free thread from the pool and uses it to execute the job. After the job finishes executing, Quartz returns the thread back to the pool.
Instances of Quartz job implementation classes are not shared. That means, when Quartz is about to execute a job, it instantiates the configured org.quartz.Job class and invokes its execute method passing it the job execution context as a parameter. Once the job execution completes, the org.quartz.Job instance is discarded and eventually garbage-collected, i.e. it is not reused by Quartz. If your org.quartz.Job class declares / accesses some static fields, singletons etc., then you may need to synchronize access to these shared resources where necessary.
I have multiple jobs and they all share the same resource. This resource is some ad-hoc build script, and so it cannot be ran concurrently.
Is it possible to define in Quartz that some jobs cannot run concurrently?
So, if one of the jobs is already running, the spawned job is queued.
I had encounter a similar scenario in my application, try out the below approach and see if it works for you.
Put the code that runs your ad-hoc build script in a synchronized block.
With this only one thread will run your ad-hoc script at a time, even when multiple threads are trying to run the same resource.
With this, you can increase the thread count to a suitable value as well instead of setting it to 1, like below.
spring.quartz.properties.org.quartz.threadPool.threadCount=5
If you want to run multiple quartz scheduler instances on different machines sharing a single database, then you should consider Configure Clustering with JDBC-JobStore. Please refer this link for more info http://www.quartz-scheduler.org/documentation/quartz-2.2.2/configuration/ConfigJDBCJobStoreClustering.html
I'm working in Spring-quartz batch. I'm trying to implement Multi-threading for the Batch application.
I come across 2 possible way of multi threading,
Use Quartz Thread pool
Use Task Executors.
I used Quartz thread pool and it is working fine but was wondering what the advantage i will get if i also implement task Executor.
I'm doing all this as xml configuration.
Please suggest me which should be used and what is the benefit of one over the other.
Thanks
I would choose task executors if all you need is to keep N workers picking pieces of work from the common queue. The advantage is that you do not need any external libraries for this. Quartz thread pool was created before Java 5 - that is why it exists.
Executor is good enough for running concurrent tasks within a JVM. But if you want to distribute tasks across multiple JVMs in a clustered environment, then you should explore Quartz using the JDBC Store.
Quartz is more of a scheduling framework where you can setup jobs to run on a periodic basis. But I have also used it heavily for concurrent programming.
I'm working on an application that uses Quartz for scheduling Jobs. The Jobs to be scheduled are created programmatically by reading a properties file. My question is: if I have a cluster of several nodes which of these should create schedules programmatically? Only one of these? Or maybe all?
i have used quartz in a web app, where users, among other things, could create quartz jobs that performed certain tasks.
We have had no problems on that app provided that at least the job names are different for each job. You can also have different group names, and if i remember correctly the jobgroup+jobname combination forms a job key.
Anyway we had no problem with creating an running the jobs from different nodes, but quartz at the time(some 6 months ago, i do not believe this has changed but i am not sure) did not offer the possibility to stop jobs in the cluster, it only could stop jobs on the node the stop command was executed on.
If instead you just want to create a fixed number of jobs when the application starts you better delegate that job to one of the nodes, as the jobs name/group will be read from the same properties file for each node, and conflicts will arise.
Have you tried creating them on all of them? I think you would get some conflict because of duplicate names.
So I think one of the members should create the schedules during startup.
You should only have one system scheduling jobs for the cluster if they are predefined in properties like you say. If all of the systems did it you would needlessly recreate the jobs and maybe put them in a weird state if every server made or deleted the same jobs and triggers.
You could simply only deploy the properties for the jobs to one server and then only one server would try to create them.
You could make a separate app that has the purpose of scheduling the jobs and only run it once.
If these are web servers you could make a simple secured REST API that triggers the scheduling process. Then you could write an automated script to access the API and kick off the scheduling of jobs as part of a deployment or whenever else you desired. If you have multiple servers behind a load balancer it should go to only one server and schedule the jobs which quartz would save to the database backed jobstore. The other nodes in the cluster would receive them the next time they update from the database.
I'm using Quartz together with Spring. The JobStore that I'm using is the RAMJobStore.
I create a couple of jobs with the same identification (they have the same instance definition (JobDetail)). Because I want to make sure that these jobs aren't executed in parallel, I annotated their job class with #DisallowConcurrentExecution.
My problem is that the RAMJobStore doesn't allow more than one job with the same identification in the same time in the store, so when I try to add the job I get the exception:
org.quartz.ObjectAlreadyExistsException: Unable to store Job :
'jobX', because one already exists with this identification.
Do you have any idea about how I can overcome this problem?
Thanks a lot!
If you have two different jobs that are running on two different triggers, then I'm not aware of any Quartz annotations that would prevent the two jobs from running in parallel. You could reference the Scheduler instance in each of the jobs to determine if the other job is executing. Then you could pause or reschedule jobs to prevent them from running in parallel.
It is clear from the RAMJobStore source code that there can't be two jobs with the same key in the same time in the RAMJobStore.
Have a look here at the source code.