I'm working on a small servlet-based application with rapid build/deploy/test turn-around. I am finding that the wait time during development for Tomcat 8.0 to auto redeploy (with Host setting 'autoDeploy=true') is quite distracting: it varies between almost immediate and nearly 10 seconds.
I'd like (roughly in order of preference) to:
Configure Tomcat to check for redeployments on every request.
Reduce the time Tomcat waits before polling for redeployments (to, say, 1s).
Or, alternatively, have a convenient way to prompt Tomcat to redeploy immediately as soon as the .war is built. (I'm building/deploying with Ant.)
I've been searching the Tomcat documentation (and SO) for the last 30 minutes and don't see anything for any of these. Am I missing something?
Can anyone recommend an approach (re one of the above) that they found minimized the turn-around time (and distraction)?
Related
In our confluence server (tomcat 9 based), we added interruptThreshold to 30mins to kill long running threads (usually these are stuck threads).
However, interrupts for certain threads are not succesful (and it is okay, as it's mentioned in documentation that it doesn't work in all cases), but these specific threads gets killed after 8 hours.
These specific threads are related to a custom plugin and our developers say they don't have such 8 hours setting anywhere and I couldn't find anything in tomcat or in confluence.
We will be able to replicate the stuck threads, but how to findout what's killing them after 8 hours?
Such actions on Tomcat should be logged in the {CONFLUENCE_HOME}/{DATA_FOLDER}/logs folder, please find these logs files and look for this operation/timestamp.
If for some reason, logs are not fully reflected, you can set it as described in
https://confluence.atlassian.com/doc/configuring-logging-181535215.html.
Also, you can control loggng in -X... Java / Tomcat statements when running start.bin/start.sh.
I'm running openjdk 11.0.3 on a server. Whenever the server has been rebooted (every night): For the first initial launch of my Application, the users have to wait for 35 Seconds before the Application is even started. (Before the first System.out.println is written from main Method.) (subsequent launches are very fast though)
I have tried the following option to debug this:
-Xlog:class+load:file=classload.txt
Here are the most important finds:
...
[2.284s][info][class,load] jdk.internal.loader.URLClassPath$FileLoader source: jrt:/java.base
[5.032s][info][class,load] sun.security.rsa.RSASignature$SHA1withRSA source: jrt:/java.base
…
[5.051s][info][class,load] java.util.LinkedList$Node source: jrt:/java.base
[8.121s][info][class,load] pos.LFChangeable source: file:/C:/Users/rho/AppData/Roaming/edapp/pos.jar
…
[8.135s][info][class,load] java.io.FileNotFoundException source: jrt:/java.base
[10.584s][info][class,load] sun.reflect.misc.ReflectUtil source: jrt:/java.base
…
[11.744s][info][class,load] java.security.NoSuchAlgorithmException source: jrt:/java.base
[34.853s][info][class,load] jdk.internal.logger.DefaultLoggerFinder source: jrt:/java.base
Why is it hanging for 23 Seconds between loading java.security.NoSuchAlgorithmException and jdk.internal.logger.DefaultLoggerFinder? And what about the other seconds of slowdowns?
edit:
Based on the comments, I will clarify some.
This is a windows rdp server. Actually, it is more than one server, but the problem persists on all servers.
The Application is a standalone Application. So every morning there are problems as users who log in to start the Application, will try to start it multiple times when "nothing happens".
I have now tried restarting one of the servers quite a few times, and this is what I found:
Starting my Application with java11 after reboot takes on average 40 seconds before the first System.out.println. Then it is only 1-2 Seconds before my first JFrame shows.
Starting my Application with java8 (sun) after reboots takes on average 16 Seconds before the first System.out.println. But I then get a 25 second delay before my first JFrame shows.
Starting my Application with java11 after already started with java8 takes on average 4-6 seconds.
Your application might be suffering from the absence of a “class data-sharing (CDS) archive”. Such an archive allows much faster loading of standard classes and has been generated by default by some installers of previous versions, but OpenJDK 11 does not have an installer.
This is addressed by JEP 341:
Currently, a JDK image includes a default class list, generated at build time, in the lib directory. Users who want to take advantage of CDS, even with just the default class list provided in the JDK, must run java -Xshare:dump as an extra step. This option is documented, but many users are unaware of it.
So while this JEP is about JDK 12 doing the necessary steps automatically, it also mentions the fix for JDK 11: just run java -Xshare:dump on the command line once, to generate the archive.
Note that you can improve the startup time even further by including application classes in the CDS. See also the Class Data Sharing section of the JDK 11 documentation.
I have now tested extensively, and I am prepared to publish my results, along with the 2 different "solutions" that I made.
First, let me explain my application a bit. It is a swing enterprise-application that started it's life 13 years ago, and has been extended every since.
This application therefore is big, does a lot of different things (although most users uses only a portion of it), and has about 120 jar-files on it's classpath including all the third-party-jars.
As previously mentioned, after restart of the server it takes 35 seconds before my first login-JFrame is shown.
Solution 1:
This was my first solution, and is not a solution to the slow start, but more a solution to the user not starting multiple instances of the application.
I noticed that although my application was very slow on the first initial start, other applications were not.
A workaround was therefore to make a small standalone-application to display the splash screen, that I start like this in my program:
splashProcess = Runtime.getRuntime().exec("javaw -jar splash.jar");
Later I just kill it off with
splashProcess.destroy();
Note that if I should create a splashscreen with new JFrame() instead, it would take the usual 35 seconds before it is displayed.
Solution 2:
While testing, I found out I could simulate a restart by just deleting all the jar-files and copying them back.
In addition to reduced testing time, I found out that starting the application with just the 4-5 jar-files needed for the initial startup was very fast (although that would have lead to ClassNotFoundExceptions later),
this also ment that I could try to figure out which jar-file which led to the hang, by starting by copying all jar-files back, and then omitting one and one more.
However, I found out that it was not one jar-file that was to blame. The seconds it takes before the application start was steadily reduced a little bit each time I removed some jar-files.
So, it seems that the problem is that the first time I call new JFrame() in my application, java seems to build some sort of index or something of all classes in the classpath, although they are not used at this time.
I don't know why it does this, but this process takes quite some time with 120 jar-files on the classpath.
This led me to solution nr 2. When my application now starts, I check for an argument "startSilent".
If this is present, the only thing my application does is show a new JDialog with size 0,0 and then call System.exit(0);
I then made a script that runs my application with the "startSilent"-parameter that starts when the user logs in.
Now, if the user logs into the server and waits at least 35 seconds before starting our application, the start is now lightning fast, as the application has already started and exited once, so that the "classpath-index" or whatever it is has been built.
If the user starts the application after a shorter time, the start-time is reduced by how long the silent-script has already run.
(And the start will always be at least a fair degree faster than before, as the script starts before the desktop is ready).
So these are the results of my findings. I hope other will find them useful, and if someone can explain why what I call the "classpath-index" is created as it is, I would be welcome.
Is there any config in tomcat where in I can mention that start all application at same time when tomcat restarts. This is to avoid latency during tomcat restarts.
The best you can do is configure a suitably large startStopThreads setting for the <Host .../> element in server.xml so each Context starts in a parallel thread.
How successful this is will depend on how many cores you have on your system compared to the number of web applications and the relative start times of each web application.
Full details of the startStopThreads are available in the Tomcat Documentation.
They all start automatically, I'm not sure what else you need.
AFAIK they're started sequentially, one after the other - probably this is what you're aiming at. I'm not aware of a multi-threaded concurrent start, but even if this would be the case, they'd all have an individual startup time anyways, so you might shorten the time, but still have the same problems.
You can counter any problem by making your tomcat available to the outside world only when it has fully been started up (e.g. through proper loadbalancer configuration). If your applications take too long to inintialize, you might want to work on this issue as well.
We have an (Eclipse RCP) application of 90MB with 139 self-signed jars which starts in 8s without Web Start and in 10s in an older version of Java 7. We configured Java to not use the browser proxy, i.e. deployment.proxy.type=0.
With each update of Oracle's Java startup performance drops. It takes more and more time to fully start up:
7u60/7u65/8u25: 13s (starts after 5s of web start processing)
7u75: 23s
8u31: 20s
8u40: 29s
8u51/8u60ea: 32s
What can I do to solve this issue?
From the trace/logs I can see that it is very probable that this slowdown is completely due to validating the cached jars taking much more time. Note that this question is similar but doesn't provide the following details:
Diagnosis:
When cached, the update check runs in only 0.5s (server returns "304 Not
Modified"), but even with a full download it takes only a few seconds on the gigabit network.
After the update check, for each jar XXX there is a log entry:
validating cached jar XXX.jar
When this is done, com.sun.javaws.Main is started after which the same validation seems to happen again and takes about the same amount of time, then the application starts.
The time spent in validating the cached jars seems to correspond to
the extra time required before the application starts.
The web start splash screen always shows for about 2 seconds corresponding to the update check and is then hidden. Then after almost 20 seconds the Java console finally appears and my application actually starts.
During the delay, jp2launcher.exe uses about 16% processor time on a quad core with hyperthreading (8 logical cores). So it looks like it is fully using one of the logical processors.
What I have already tried but did not make any difference:
clearing web start cache (countless times)
configure deployment.properties to disable certificate revocation check (as well as blacklist.check and validation.ocsp, validation.crl)
running offline
using the version download protocol
add to site exceptions list
check web server logs for problems. None found, update check runs in about 500ms for all 138 jars.
use another web server
checked certificate expiration date 17 feb 2016
validated my jnlp with JaNela and found no serious issues
create a deployment rule set to allow the application to run unsigned in order to speed up validation. This should be possible and looks like a promising way to solve this, but I could not get it working. See also my answer on this post.
configured Java to "do not start the console"
Detail: some weird behavior on 7u60
In 7u60 the application is started after about 5 seconds, after which the Java console APPEARS to be doing the jar validation in the background while the application is already started. HOWEVER the .log file reports that the application gets started AFTER all the validations are done. It reports this as 25 seconds and then shows the first System.out of my application which actually happened after only 5 seconds or so. It also shows the jar update check with the server taking 10 times as long as reported by the server. So I guess this is an issue with the logging framework lagging behind! Haven't seen this on 8u51.
Not an answer per se (yet), but I found that Java 8u25 when tracing is enabled, only generate a single trace file. 8u51 generates two files, one from the JVM used to update the application and other to run it. This is new (two JVM startups) and I think is related to the new setting for using native Windows sandbox capabilities. The problem is that it shouldn't have to validate the signatures again on the second JVM. The separation on two JVM instances always happen no matter if the setting for using native sandbox is disabled (the default).
I reported a regression bug, will edit the answer if I get an answer from Oracle.
Note: Java 8u31 still runs everything on one JVM but have the same doubled startup time the question stated.
Here is the situation: I had an app with a cold start time of about 4 seconds. I was trying to improve the cold start time by removing a bunch of libraries and code I didn't really need. After doing that the cold start time was about 3 seconds latency, and 3 seconds CPU time used.
I changed the version number in appengine-web.xml, and nothing else. And now I have two versions of my app that have the exact same code, up and running.
For cold starts, the newer version uses 1800ms to 1900ms in CPU time.
For cold starts, the older version uses 2400ms to 3000ms in CPU time.
The exact same jsp page from each version is requested to test the cold start time. So far I have sampled 7 cold starts for each version.
hmmm, I think it is possible that there is some kind of caching of the look of your application, since gae upload is basically differential update (you send only changed files).
If you posted many changes on one version id, it is possible that GAE has many snapshots of your code.
Thus, if you do big changes (this is my rule of thumb) you should always change the version of your application, just to be sure. Additional commits I use only for bug fixes, never for big refactorings/adding or removing JARs. I think you also at that point you have new logs and simply "refreshing installation" of your application so GAE can do some optimizations...
Agree?
Sounds like a fluke, I don't see how changing the version number of your program could generate a change in speed. Unless there was a coincidental library update or some such.
Could the version number be changing an execution path somewhere? Perhaps in the XML parser or data binding that happens before your app is running?