Develop Reference

Quartz in cluster mode with lockoninsert=false

I have application having 4 nodes. Sometimes all my jobs will be stuck waiting for lock from 'Select * from QRTZ_LOCKS where lock_name='TRIGGER_ACCESS' for update'
While reading some of the articles someone suggested to turn off global lock using this property
org.quartz.jobStore.lockOnInsert=false
Has anyone tried to run Quartz in cluster mode with lockoninsert=false?
I am planning to use following configuration
#============================================================================
# Configure Main Scheduler Properties
#============================================================================
org.quartz.scheduler.instanceName = StandardScheduler
org.quartz.scheduler.instanceId = AUTO
#============================================================================
# Configure ThreadPool
#============================================================================
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 3
org.quartz.threadPool.threadPriority = 5
#============================================================================
# Configure JobStore
#============================================================================
org.quartz.jobStore.misfireThreshold = 300000
org.quartz.jobStore.class=org.springframework.scheduling.quartz.LocalDataSourceJobStore
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
org.quartz.jobStore.useProperties=false
org.quartz.jobStore.tablePrefix=QRTZ_
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.lockOnInsert=false
org.quartz.jobStore.acquireTriggersWithinLock=true
org.quartz.jobStore.lockHandler.class=org.quartz.impl.jdbcjobstore.UpdateLockRowSemaphore
org.quartz.scheduler.jmx.export=true

A found some comment in the 1.8.6' source
public boolean isLockOnInsert() {
return lockOnInsert;
}
/**
* Whether or not to obtain locks when inserting new jobs/triggers.
* Defaults to <code>true</code>, which is safest - some db's (such as
* MS SQLServer) seem to require this to avoid deadlocks under high load,
* while others seem to do fine without.
*
* <p>Setting this property to <code>false</code> will provide a
* significant performance increase during the addition of new jobs
* and triggers.</p>
*
* #param lockOnInsert
*/

Quartz Scheduler worker threads not running jobs even with huge number of jobs in queue

We have 30 worker threads in quartz scheduler, and there are 104k jobs in the queue (they were piled up since quartz was not processing fast enough and the lots of bulk jobs were submitted). As you can see in the thread dump, most of these threads are just waiting without working. Why are these worker threads not running all those jobs which are in the queue missing the fire time?
Could there be any bottlenecks before the jobs are distributed to the workers? We see only a few jobs running at any time instead of upto 30 jobs at once.
I should note that this happens irrespective of only one scheduler process or many.
Any pointers to find why many (at most 30) jobs are not getting picked up and executed at once would be really appreciated.
Quartz version: 2.1.3 (with Oracle JDBC jobstore)
Config File
#============================================================================
# Configure Main Scheduler Properties
#============================================================================
org.quartz.scheduler.instanceName = ****_SCHEDULER
org.quartz.scheduler.instanceId = AUTO
#============================================================================
# Configure ThreadPool
#============================================================================
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 30
org.quartz.threadPool.threadPriority = 5
#============================================================================
# Configure JobStore
#============================================================================
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.dataSource = myDS
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval = 20000
org.quartz.jobStore.tablePrefix = QRTZ_
#============================================================================
# Configure Datasources
#============================================================================
org.quartz.dataSource.myDS.driver = oracle.jdbc.driver.OracleDriver
org.quartz.dataSource.myDS.URL = jdbc:oracle:thin:#*****
org.quartz.dataSource.myDS.user = *****
#org.quartz.dataSource.myDS.password = *****
org.quartz.dataSource.myDS.maxConnections = 10
org.quartz.dataSource.myDS.validationQuery=select 0 from dual
#============================================================================
# Configured to batch trigger acquisition
#============================================================================
org.quartz.scheduler.skipUpdateCheck = false
org.quartz.scheduler.batchTriggerAcquisitionMaxCount = 30
org.quartz.jobStore.acquireTriggersWithinLock = true
~
Thread Dump
I see the QuartzSchedulerThread (i assume is like manager thread?) and MisfireHandler threads are always running with same stacktrace (except for locked monitor id).
Stacktrace for QuartzSchedulerThread
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:300)
at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1044)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1199)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
- locked [0x00000007f5d0fd30] (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:91)
at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:115)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3803)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggersFired(JobStoreSupport.java:2893)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:326)
Stacktrace for MisfireHandler thread:
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:300)
at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1044)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1199)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
- locked [0x00000007f58c8010] (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.hasMisfiredTriggersInState(StdJDBCDelegate.java:379)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:946)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3187)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3946)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3967)

Tomcat/TomEE/OpenEJB Datasource Connection to SQL Server - CPU Creep

We are using TomEE (Tomcat with OpenEJB) to connect to a SQL Server database. The connection works fine, but we are seeing odd CPU creeping issues. The application runs and performs well for about 24 hours (SQL Server CPU at about 15%-20%), and then performance starts to suffer after that. We check and the CPU is now at 60%. If we shut down the running jobs (but leave Tomcat running), the CPU drop down to almost zero. If we restart the processing, the CPU jumps back up immediately to the 60%. If we stop the Tomcat instance and then restart, then the CPU usage on the SQL Server machine stays at its low usage for about 24 hours and then it begins to creep back up again.
I think that this must be due to the datasource configuration, but I'm not sure what changes we should make. Can someone help out? Here is the datasource configuration we are using:
<Resource id="Datasource" type="javax.sql.DataSource">
accessToUnderlyingConnectionAllowed = false
defaultAutoCommit = true
ignoreDefaultValues = false
initialSize = 0
jdbcDriver = com.microsoft.sqlserver.jdbc.SQLServerDriver
jdbcUrl = jdbc:sqlserver://{url}
jtaManaged = true
maxActive = 200
maxIdle = 20
maxOpenPreparedStatements = 0
maxWaitTime = -1 millisecond
minEvictableIdleTime = 30 minutes
minIdle = 0
numTestsPerEvictionRun = 3
password = {password}
passwordCipher = Static3DES
poolPreparedStatements = false
testOnBorrow = true
testOnReturn = false
testWhileIdle = false
timeBetweenEvictionRuns = -1 millisecond
userName = {user}
validationQuery = SELECT 1
</Resource>
Thanks in advance!

about misfire in java quartz scheduling

I have set a trigger using cronScheduler with misfireInstruction like follows
trigger = newTrigger().withIdentity("autoLockTrigger", "autoLockGroup").startNow() .withSchedule(cronSchedule(croneExpression).withMisfireHandlingInstructionFireAndProceed())
.forJob("autoLockJob","autoLockGroup")
.build();
my quartz.properties is like follows
org.quartz.scheduler.instanceName =MyScheduler
# Configuring ThreadPool
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 1
org.quartz.threadPool.threadPriority = 9
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.dataSource = myDS
org.quartz.jobStore.tablePrefix = QRTZ_
#org.quartz.dataSource.myDS.jndiURL = jdbc/vikas
org.quartz.dataSource.myDS.driver = com.mysql.jdbc.Driver
org.quartz.dataSource.myDS.URL = jdbc:mysql://staging:3307/facao
org.quartz.dataSource.myDS.user = root
org.quartz.dataSource.myDS.password = toor
org.quartz.dataSource.myDS.maxConnections = 30
#org.quartz.jobStore.nonManagedTXDataSource = myDS
#to store data in string format (name-value pair)
org.quartz.jobStore.useProperties=true
org.quartz.jobStore.misfireThreshold = 60000
In my code if I set some trigger at particular time and if server is in running state then scheduler runs properly but if server is down for the time in which scheduler is suppose to be run and then started after some time then scheduler should run the misfired instruction. But in my case the misfired instruction is not running all the time it runs some time not always so my purpose is not fulfilled. Please give some solution. Thank you in advance.

I am not sure about the cron triggers but for simple triggers yeah,
if the end time of the trigger has been passed then some of the provided misfire instruction
will not work. See the javadoc snippet for more info.
I guess the same would be the case with cron trigger too.
So, it totally depends on what cron expression you use.

porting Quartz from IAS to JBoss AS

I am porting our code from IAS to JBoss AS.
There is strange behavior where quartz does not trigger any event at all, and no errors appear at quartz logs. I have also noticed that the Quartz tables are not populated (QRTZ_JOB_DETAILS, QRTZ_TRIGGERS etc.).
I am using JOBStoreCMT with quartz version 1.5.2. datasource is well declared. Jobs and triggers worked well in IAS and declared within the code.
quartz properties:
#============================================================================
# Configure Main Scheduler Properties
#============================================================================
org.quartz.scheduler.instanceName = bitbandScheduler
org.quartz.scheduler.instanceId = AUTO
#============================================================================
# Configure ThreadPool
#============================================================================
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 15
org.quartz.threadPool.threadPriority = 5
#============================================================================
# Configure JobStore
#============================================================================
org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreCMT
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties = false
org.quartz.jobStore.dataSource = bitband_pluginDS
org.quartz.jobStore.nonManagedTXDataSource = bitband_pluginDSTX
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = false
org.quartz.jobStore.clusterCheckinInterval = 20000
#============================================================================
# Configure Datasources
#============================================================================
org.quartz.dataSource.bitband_pluginDS.jndiURL=java:bitband_pluginDS
org.quartz.dataSource.bitband_pluginDSTX.jndiURL=java:bitband_pluginDS
oracle-ds.xml:
<xa-datasource>
<jndi-name>bitband_pluginDS</jndi-name>
<!-- uncomment to enable interleaving <interleaving/> -->
<isSameRM-override-value>false</isSameRM-override-value>
<xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
<xa-datasource-property name="URL">jdbc:oracle:thin:#ord-rtv063.orca.ent:1521:DB11g</xa-datasource-property>
<xa-datasource-property name="User">RIGHTV7_VS</xa-datasource-property>
<xa-datasource-property name="Password">RIGHTV7_VS</xa-datasource-property>
<max-pool-size>100</max-pool-size>
<min-pool-size>20</min-pool-size>
<valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name>
<check-valid-connection-sql>select 1 from dual</check-valid-connection-sql>
<!-- Uses the pingDatabase method to check a connection is still valid before handing it out from the pool -->
<!--valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name-->
<!-- Checks the Oracle error codes and messages for fatal errors -->
<exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
<!-- Oracles XA datasource cannot reuse a connection outside a transaction once enlisted in a global transaction and vice-versa -->
<no-tx-separate-pools/>
<!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->
<metadata>
<type-mapping>Oracle9i</type-mapping>
</metadata>
</xa-datasource>
What am I missing?
PS, When using JobStoreTX, everything works well, so I guess it is something related to the container transaction manager.

After hanging around the problem for the last couple of days I found a solution.
Adding the property below to quartz.properties file. As simple as that.
org.quartz.jobStore.dontSetAutoCommitFalse=false
Setting this parameter to true tells Quartz not to call setAutoCommit(false) on connections obtained from the DataSource(s). This can be helpful in a few situations, such as if you have a driver that complains if it is called when it is already off. This property defaults to false, because most drivers require that setAutoCommit(false) is called.
For some reason JBoss overrides the default value, so I had to add it explicitly.
The credit goes to unknown user at:
http://osdir.com/ml/java.quartz.user/2007-10/msg00123.html