Develop Reference

Are Quartz jobs running in the same JVM in their own threads?

Quartz scheduler is used to schedule timed java jobs at my workplace. The scheduler itself is deployed as an application to a Weblogic servers (a cluster of machines). This scheduler can schedule jobs which implement the Job interface and override the execute() method. These jobs are deployed to the Weblogic servers as libraries which are then used by the scheduler. (One library includes multiple jobs.)
I have not managed to find informative sources on how these jobs are run or how they share resources.
I looked at the Quartz documentation but could not find what I was looking for.
I have multiple questions, though I believe a single answer may cover all of them.
Do all the jobs created through the scheduler share a single JVM? If they don't, then based on what is a job allocated to a given JVM?
I assume that a separate thread is allocated to each job - is that correct?
If all scheduled jobs run in the same JVM and each has its own thread, then concurrent execution of the same job with different parameters will create a need for making the job thread safe or a need to disable concurrent execution of the job, will not it?
Thank you.

You may want to read Quartz scheduler tutorial to find out how Quartz works. To answer your questions:
This depends on whether you run a Quartz scheduler cluster (i.e. multiple Quartz scheduler instances sharing the same job store) or a standalone Quartz scheduler instance. In clustered deployments, individual Quartz scheduler instances compete to execute jobs by creating DB row locks. The scheduler instance that first manages to create the row lock, is the scheduler instance that executes a particular job. In a standalone Quartz scheduler deployment, the completion does not exist and it is always the single Quartz scheduler instance that ends up executing all jobs.
Quartz uses a thread pool and when it needs to execute a job, it simply allocates a free thread from the pool and uses it to execute the job. After the job finishes executing, Quartz returns the thread back to the pool.
Instances of Quartz job implementation classes are not shared. That means, when Quartz is about to execute a job, it instantiates the configured org.quartz.Job class and invokes its execute method passing it the job execution context as a parameter. Once the job execution completes, the org.quartz.Job instance is discarded and eventually garbage-collected, i.e. it is not reused by Quartz. If your org.quartz.Job class declares / accesses some static fields, singletons etc., then you may need to synchronize access to these shared resources where necessary.

what is the difference between spark scheduling mode and application queue in spark?

While testing the behavior of spark jobs when multiple jobs are submitted to run concurrently or smaller jobs submitted later. i came across two settings in spark ui. one is scheduling mode available withing spark as shown in below image
And one is under scheduler as show below
I want to understand the difference between two settings and preemption. My Requirement is that while running the bigger job, small jobs submitted in between must get the resources without waiting longer.

Let me explain it for the Spark On Yarn mode.
When you submit a scala code to spark, spark client will interact with yarn and launch a yarn application. This application will be duty on all the jobs in your scala code. In most cases, each job correspond to an Spark Action like reduce()、collect().Then ,the problem comes, how to schedule different jobs in this application, for example, in your application , there a 3 concurrent jobs comes out and waiting for execution? To deal with it , Spark make the scheduler rule for job, including FIFO and Fair.That is to say , spark scheduler ,including FIFO and Fair, is on the level of job, and it is the spark ApplicationMaster which is do the scheduling work.
But yarn's scheduler, is on the level of Container.Yarn doesn't care what is running in this container, maybe the container it is a Mapper task , a Reducer task , a Spark Driver process or a Spark executor process and so on. For example, your MapReduce job is currently asking for 10 container, each container need (10g memory and 2 vcores), and your spark application is currently asking for 4 container ,each container need (10g memory and 2 vcores). Yarn has to decide how many container are now available in the cluster and how much resouce should be allocated for each request by a rule ,this rule is yarn's scheduler, including FairScheduler and CapacityScheduler.
In general, your spark application ask for several container from yarn, yarn will decide how many container can be allocated for your spark application currently by its scheduler.After these container are allocated , Spark ApplicationMaster will decide how to distribute these container among its jobs.
Below is the official document about spark scheduler:https://spark.apache.org/docs/2.0.0-preview/job-scheduling.html#scheduling-within-an-application

I think Spark.scheduling.mode (Fair/FIFO), shown in the figure, is for scheduling tasksets (single-same stage tasks) submitted to the taskscheduler using a FAIR or FIFO policy etc.. These tasksets belong to the same job.
To be able to run jobs concurrently, execute each job (transformations + action) in a separate thread. When a job is submitted to the DAG the main thread is blocked until job completes and result is returned or saved.

How to set a dedicated quartz worker thread for a specific job?

I was wondering if is possible to configure quartz to execute a job under a dedicated worker thread. In another words, say I have quartz configured with a SimpleThreadPool of size 5. And I have a job that fires every 10 seconds, and i want a dedicated worker thread to run this job. Is there a way to configure quartz trigger|job|scheduler to do that?
Other workers in thread pool may free to execute any others jobs scheduled.
I want that specific job execute with no time waiting, if other workers are busy.

Difference between Quartz Thread Pool and Task Executor

I'm working in Spring-quartz batch. I'm trying to implement Multi-threading for the Batch application.
I come across 2 possible way of multi threading,
Use Quartz Thread pool
Use Task Executors.
I used Quartz thread pool and it is working fine but was wondering what the advantage i will get if i also implement task Executor.
I'm doing all this as xml configuration.
Please suggest me which should be used and what is the benefit of one over the other.
Thanks

I would choose task executors if all you need is to keep N workers picking pieces of work from the common queue. The advantage is that you do not need any external libraries for this. Quartz thread pool was created before Java 5 - that is why it exists.

Executor is good enough for running concurrent tasks within a JVM. But if you want to distribute tasks across multiple JVMs in a clustered environment, then you should explore Quartz using the JDBC Store.
Quartz is more of a scheduling framework where you can setup jobs to run on a periodic basis. But I have also used it heavily for concurrent programming.

Spring Scheduler Disallow Concurrent Alternative

When scheduling a task in Quartz, you have the ability to set misfires and rescheduling. This could be used in the example scenario whereby there is a job that runs every 30 mins, and potentially there could be a backlog and and the job would execute for longer than 30 mins. To prevent the same job running twice you could use the #DisallowConcurrentExecution. Once complete the job would then execute the second instance that is queued by using simpleSchedule().withMisfireHandlingInstructionNowWithExistingCount().
Now in Spring Scheduler there doesn't appear to be this fine grained ability, with just the fixed-rate and fixed-delay options to schedule it every 30 mins or wait 30 mins after the previous job completed. Without using the hammer route of restricting to a single thread, as I want to increase the thread count for other batch jobs to run concurrently, what would be the best method of recreating the Quartz behaviour?

So it looks like with the basic Spring Scheduler there isn't such a mechanism. To do this either use the Spring Quartz Scheduler, or Quartz directly.