I've been looking all around and cannot find any good explanations or examples of using Spring 4.x (Springboot) with an EmbdeddedTomcat container and manually setting the startStopThreads attribute described here
I've noticed our application's startup latency has been getting worse over time, but also see (in the logs) the thread pool labeled [localhost-startStop-1] is the only thread ever executing beyond [main]. I would like to add more threads into this pool to speed up our asynch startup.
Can anybody help me with Tomcat's startStopThreads issue?
UPDATE -
A good usecase for this is dynamically creating DynamoDb tables upon startup. The creation code is wrapped in an Executor.submit call, however the logs show these being executed serially. Here's an example of whats in the logs:
2017-02-22 15:000:000:01,000 [main ] INFO Creating table 1
2017-02-22 15:000:000:05,000 [localhost-startStop-1] INFO Creating table 2
2017-02-22 15:000:000:10,000 [localhost-startStop-1] INFO Creating table 3
Unfortunately, I am never seeing localhost-startStop-2 execute anything, which it should if it were in the threadpool
Configuring startStopThreads won't have any effect. It's used by each container in Tomcat (Service, Host, Context, etc) when starting their children. With more than one thread available a container with multiple children will start them in parallel. It won't have any effect in a typical Spring Boot application as each Tomcat container only has a single child.
Rather than trying to use Tomcat's threads to perform some initialisation in parallel, I'd recommend using a Java Executor or Spring Framework's TaskExecutor.
Related
In our project, we have several Spring-based modules which are deployed on WAS as web applications. We need to skip deployment, or stop a module if its Spring context initialization fails (i.e. ContextLoaderListener#contextInitialized or DispatcherServlet#init throws an exception). Now, if such happens, app is got deployed and starts, but returns HTTP 500 for any request.
Websphere 8.5.5
Related question: https://stackoverflow.com/a/272747/3459206
This APAR seems to be relevant:
https://www-01.ibm.com/support/docview.wss?uid=swg1PI58875
From the APAR text:
Listener exceptions typically should not stop the application
from starting up for service. However, some applications depend
on their listeners to do the necessary setup before the
application is started for service. Such applications prefer to
stop the application from starting up when there is any
exception in their listeners.
Problem conclusion
The WebContainer Container code was modified to provide an
option to stop the application when there is any listener
exception during the application starting up process.
A new WebContainer custom property needs to be set to enable the
behavior provided by this APAR:
For Full Profiles
com.ibm.ws.webcontainer.stopappstartuponlistenerexception = true
(default is false)
For Liberty Profile
stopappstartuponlistenerexception=true
The fix for this APAR is currently targeted for inclusion in
WebSphere Application Server fix packs 8.5.5.11 and 9.0.0.2,
and Liberty 16.0.0.3
See the APAR link for additional information.
You can use jenkins + maven.
Add the part you need to check under your test like junit.
Then if this module do not pass test, jenkins would not deploy it.
But I prefer fix bugs before deployment
Had a very similar issue.
The thing is - webfear - sorry could not resist ;-) does not initialize everything on startup.
To trigger a controlled request, I added a ScheduledEJB to the startup of the application. This bean itself triggered a http-request to a defined URL, which itself triggered:
any filters to get initialized in the chain
any contexts which are needed are initialized
And this itself ensured that my application (EAR or WAR) got very quickly tested after deployment. This works well with some small amout of requests per minute
If you work with high load, means tons of requests per second, you need to choose a different approach.
In this case I added a polling mechanism into the #Startup of the application, which polled every second or 250ms (depends on the load of the application).
This firing to the server ensured, that my #Startup bean was the very first which triggered the possible init issues in the application. If this happened I initialized a filter which always reported a 500 (or better fitting error) to the requestor.
Of course stop your firing bean, as soon as you get the 500, else your admins may like to kill you. (happend to me, since I produced tons or monitoring issues ;-) )
And of course on the regular operation, after your application started properly, you should also disable the polling
Look for a try-catch in the top level of your application code that is catching the Spring exception and allowing the application to continue running.
If the Spring exceptions being thrown are permitted to propagate to the top of the stack, the JVM will stop and there's no way it can keep running, far as I know.
I have Quartz Scheduler running within WebLogic 12.1.3 and backed by a JobStoreCMT, but its behavior doesn't match the configuration (see below). What am I doing wrong?
Background
Quartz has jobs that are loaded from an XML file on startup and run periodically. Some of those jobs spawn one-time jobs. Also, there are one-time jobs that are manually triggered by users. The user-initiated jobs are done from EJBs that have container-managed transactions.
Questions
The transactions in the job classes are not active in the job's execute() method. I have to call begin/commit/rollback. Shouldn't that be taken care of since wrapJobExecutionInUserTransaction is set to true? That's what the documentation says.
The documentation also says that when using XMLSchedulingDataProcessorPlugin with JobStoreCMT, org.quartz.plugin.jobInitializer.wrapInUserTransaction must be set to true. However, when I do that I get a duplicate transaction exception. What's going on?
WebLogic takes forever to shut down whenever Quartz is enabled even though all of the jobs run quickly. From the logs it looks like WebLogic is waiting for the transactions to time out. Is something in the config contributing to this?
Configuration
org.quartz.scheduler.skipUpdateCheck=true
org.quartz.scheduler.instanceName=MyTaskScheduler
org.quartz.scheduler.threadsInheritContextClassLoaderOfInitializer=true
org.quartz.scheduler.instanceId=AUTO
org.quartz.scheduler.wrapJobExecutionInUserTransaction=true
org.quartz.scheduler.userTransactionURL=javax.transaction.UserTransaction
org.quartz.scheduler.idleWaitTime=30000
org.quartz.scheduler.dbFailureRetryInterval=15000
org.quartz.scheduler.batchTriggerAcquisitionMaxCount=1
org.quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow=0
org.quartz.scheduler.makeSchedulerThreadDaemon=false
org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount=20
org.quartz.threadPool.threadPriority=5
org.quartz.threadPool.makeThreadsDaemons=false
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreCMT
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.weblogic.WebLogicOracleDelegate
org.quartz.jobStore.dataSource=MyDataSource
org.quartz.jobStore.nonManagedTXDataSource=MyDataSourceNonXA
org.quartz.jobStore.tablePrefix=QRTZ_
org.quartz.jobStore.useProperties=false
org.quartz.jobStore.misfireThreshold=60000
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval=15000
org.quartz.jobStore.maxMisfiresToHandleAtATime=20
org.quartz.jobStore.txIsolationLevelSerializable=false
org.quartz.jobStore.txIsolationLevelReadCommitted=false
org.quartz.dataSource.MyDataSource.jndiURL=MyDataSource
org.quartz.dataSource.MyDataSource.java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory
org.quartz.dataSource.MyDataSource.java.naming.provider.url=t3://localhost:7003
org.quartz.dataSource.MyDataSourceNonXA.jndiURL=MyDataSourceNonXA
org.quartz.dataSource.MyDataSourceNonXA.java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory
org.quartz.dataSource.MyDataSourceNonXA.java.naming.provider.url=t3://localhost:7003
org.quartz.plugin.jobInitializer.class=org.quartz.plugins.xml.XMLSchedulingDataProcessorPlugin
org.quartz.plugin.jobInitializer.fileNames=E:/tasks.xml
org.quartz.plugin.jobInitializer.failOnFileNotFound=true
org.quartz.plugin.jobInitializer.scanInterval=0
org.quartz.plugin.jobInitializer.wrapInUserTransaction=false
Any help is appreciated.
I am wondering if anyone has information on the performance overhead of adding more threads to a tomcat thread pool?
We have been seeing issues with an app of ours and everything points to an issue with tomcat adding more threads to the thread pool.Is this true? Does adding more threads to the thread pool cause a slowdown in response time?
We are running a Java REST API using Spring Boot and its embedded tomcat. If our problems are indeed caused by tomcat adding more threads to the pool, is there a way to set a minimum in Spring? According to this thread MinSpareThreads does not exist anymore. https://github.com/spring-projects/spring-boot/issues/3753
If there isn't a way to increase the number of minimum threads with Spring Embedded Tomcat, what are other tools that you use for thread pools?
Edit:
I have attached a thread profile. The profile is not of the time in question but later.
We are now able to add the both edges of configuration:
wilkinsona commented Apr 19, 2016 ->
maxThreads has always been configurable. minSpareThreads can now be configured too (see 34eb369)
Add configuration option for maxSpareThreads - (Old bug, finally fixed)
I'm trying to implement the failover strategy when executing jbpm6 processes. My setup is the following:
I'm using jbpm6.2.0-Final (latest stable release) with persistence enabled
I'm constructing an instance of org.kie.spring.factorybeans.RuntimeManagerFactoryBean with type SINGLETON to get KSession to start/abort processes and complete/abort work items
all beans are wired by Spring 3.2
DB2 is used a database engine
I use Tomcat 7.0.27
In the positive scenario everything is working as I expect. But I would like to know how to resume the process in the case of server crash. To reproduce it I started my process (described as BPMN2 file), got at some middle step and killed the Tomcat process. After that I see uncompleted process instance in the PROCESS_INSTANCE_INFO table and uncompleted work item in the WORK_ITEM_INFO table. Also there is a session in the SESSION_INFO table.
My question is: could you show me the example of code which would take that remaining process and resume it starting from the last node (if it is possible).
Update
I forgot to mention that i'm not using jbpm-console, but I'm embedding jbpm into my javaee application.
If you initialize your RuntimeManager on init of your application Server it should take care of reloading and resuming the processes.
You need not worry about reloading it again by yourself.
It appears that our implementation of using Quartz - JDBCJobStore along with Spring, Hibernate and Websphere is throwing unmanaged threads.
I have done some reading and found a tech article from IBM stating that the usage of Quartz with Spring will cause that. They make the suggestion of using CommnonJ to address this issue.
I have done some further research and the only examples I have seen so far all deal with the plan old JobStore that is not in a database.
So, I was wondering if anyone has an example of the solution for this issue.
Thanks
We have a working solution for this (two actually).
1) Alter the quartz source code to use a WorkManager daemon thread for the main scheduler thread. It works, but requires changing quarts. We didn't use this though since we didn't want maintain a hacked version of quartz. (That reminds me, I was going to submit this to the project but completely forgot)
2) Create a WorkManagerThreadPool to be used as the quartz threadpool. Implement the interface for the quartz ThreadPool, so that each task that is triggered within quartz is wrapped in a commonj Work object that will then be scheduled in the WorkManager. The key is that the WorkManager in the WorkManagerThreadPool has to be initialized before the scheduler is started, from a Java EE thread (such as servlet initialization). The WorkManagerThreadPool must then create a daemon thread which will handle all the scheduled tasks by creating and scheduling the new Work objects. This way, the scheduler (on its own thread) is passing the tasks to a managed thread (the Work daemon).
Not simple, and unfortunately I do not have code readily available to include.
Adding another answer to the thread, since i found a solution for this, finally.
My environment: WAS 8.5.5, Quartz 1.8.5, no Spring.
The problem i had was the (above stated) unmanaged thread causing a NamingException from ctx.lookup(myJndiUrl), that was instead correctly working in other application servers (JBoss, Weblogic); actually, Webpshere was firing an "incident" with the following message:
javax.naming.ConfigurationException: A JNDI operation on a "java:" name cannot be completed because the server runtime is not able to associate the operation's thread with any J2EE application component. This condition can occur when the JNDI client using the "java:" name is not executed on the thread of a server application request. Make sure that a J2EE application does not execute JNDI operations on "java:" names within static code blocks or in threads created by that J2EE application. Such code does not necessarily run on the thread of a server application request and therefore is not supported by JNDI operations on "java:" names.
The following steps solved the problem:
1) upgraded to quartz 1.8.6 (no code changes), just maven pom
2) added the following dep to classpath (in my case, EAR's /lib folder), to make the new WorkManagerThreadExecutor available
<dependency>
<groupId>org.quartz-scheduler</groupId>
<artifactId>quartz-commonj</artifactId>
<version>1.8.6</version>
</dependency>
Note: in QTZ-113 or the official Quartz Documentation 1.x 2.x there's no mention on how to activate this fix.
3) added the following to quartz.properties ("wm/default" was the JNDI of the already configured DefaultWorkManager in my WAS 8.5.5, see Resources -> AsynchronousBeans -> WorkManagers in WAS console):
org.quartz.threadExecutor.class=org.quartz.custom.WorkManagerThreadExecutor
org.quartz.threadExecutor.workManagerName=wm/default
Note: right class is org.quartz.custom.WorkManagerThreadExecutor for quartz-scheduler-1.8.6 (tested), or org.quartz.commonj.WorkManagerThreadExecutor from 2.1.1 on (not tested, but verified within actual quartz-commonj's jars on maven's repos)
4) moved the JNDI lookup in the empty constructor of the quartz job (thanks to m_klovre's "Thread outside of the J2EE container"); that is, the constructor was being invoked by reflection (newInstance() method) from the very same J2EE context of my application, and had access to java:global namespace, while the execute(JobExecutionContext) method was still running in a poorer context, which was missing all of my application's EJBs
Hope this helps.
Ps. as a reference, you can find here an example of the quartz.properties file I was using above
Check this article:
http://www.ibm.com/developerworks/websphere/techjournal/0609_alcott/0609_alcott.html
basically, set the taskExecutor property on SchedulerFactoryBean to use a org.springframework.scheduling.commonj.WorkManager TaskExecutor which will use container managed threads.
Just a note: the above QUARTZ-708's link is not valid anymore.
This new issue (in a new Jira) seems to be addressing the problem: http://jira.terracotta.org/jira/browse/QTZ-113 (fixVersion = 1.8.6, 2.0.2)
I have recently encountered this problem. Practically you need:
Implement thread pool by delegating work to Websphere Work Manager. (Quartz provides only SimpleThreadPool that run jobs on unmanaged threads). Tell quartz to use this thread pool by org.quartz.threadPool.class property
Tell quartz to use WorkManagerThreadExecutor (or implement custom one) by org.quartz.threadExecutor.class property
A bit patience with cumbersome legacy web containers :)
Here is github demo of using Quartz with Websphere (and also Tomcat).
Hope it helps someone..
You can check the below JIRA link raised on quartz regarding this.
http://jira.opensymphony.com/browse/QUARTZ-708
This has the required WebSphereThreadPool implementation which can be used with the changes in quartz.properties as mentioned to meet your requirements. Hope this helps.
Regards,
Siva
You will have to use websphere's managed thread pools. You can do this via spring and commonj. CommonJ can has a task executor that will create managed threads. You can even use a reference to a jndi managed thread resource. You can then inject the commonj task executor into the Spring based Quartz SchedulerFactoryBean.
Please see http://open.bekk.no/boss/spring-scheduling-in-websphere/ and scroll to "Quartz with CommonJ" section for more details.
The proposal from PaoloC for WAS85 ans Quartz 1.8.6 also works on WAS80 (and Quartz 1.8.6) and does not need Spring. (In my setup Spring 2.5.5 is present, but not in use in that context.)
That way I was able to override SimpleJobFactory by my own variant, using an InjectionHelper to apply CDI on every newly created job. Injection works for both #EJB (with JNDI lookup of the annotated EJB remote business interface) and #Inject (with JNDI lookup of the CDI BeanManager using a new InitialContext first, and then using this newly fetched BM to lookup the CDI bean itself).
Thank you PaoloC for that answer! (I hope this text will appear as an "answer to PaoloC" and not as an answer to the main topic. Found no way to differentiate between these.)