I have a developed two small Java applications - a vanilla Java app and a Java Web application (i.e. Spring MVC, Servlets, JSP, etc.).
The vanilla application consists of several threads which read data continuously at varying rates (from once a second to twice a minute) from several websites, process the data and write it to a database.
The Web Application reads the data from the database and presents it using JSPs, etc.
I'd now like to deploy the applications to a Linux machine and have them run 24 x 7.
If the applications crash I would like them to be restarted.
What's the best way of doing this?
Your web container will run 24x7 by default. If your deployed application throws an exception, it's captured by the container. I wouldn't normally expect this process to not run. Perhaps if threads run away, then it may become unresponsive, so it's worth monitoring (perhaps by a separate process querying it via HTTP?).
Does your vanilla application need to run at regular intervals ? If so, then cron is a good bet. It'll invoke a new instance every 'n' minutes (or however you configure it). If your instance suffers a problem, then it'll simply bail out and a new instance will be launched at the next configured interval. Again, you should probably monitor this (capture log files?) in case some problem determines that it'll never succeed completely.
with Ubuntus upstart you can respawn processes automatically. A little bit more low-level is to put the respawn directly in /etc/inittab. Both work well, but upstart is more manageable (more tools), but requires a newer system (ubuntu, fedora, and debian is switching soon).
For inittab you need to add a line like this to /etc/inittab (from the inittab manpage):
12:2345:respawn:/path/to/myapp flags
For upstart you do something similar (this is a standard script in ubuntu 9.10):
user#host:/etc/init$ cat tty1.conf
# tty1 - getty
#
# This service maintains a getty on tty1 from the point the system is
# started until it is shut down again.
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -8 38400 tty1
Check out the ServletContextListener, this allows you to embed your java application inside your web application (by creating a background thread). Then you can have it all running inside the web container.
Consider investigating and using a web container supported by the operating system vendor so all the scripts to bring it up and down (including in case of problems) is written and maintained by somebody else but you.
E.g. Ubuntu has a Tomcat as a package
I have a crontab job running every 15 minutes to see if the script is still running. If not, it restarts the service. The script itself is a piece of Perl code:
#!/usr/bin/perl
use diagnostics;
use strict;
my %is_set;
for (#ARGV) {
$is_set{$_} = 1;
}
my $verbose = -1;
if ($is_set{"--verbose"}) {
$verbose = 1;
}
my #components = ("cdk", "qsar", "rdf");
foreach my $comp (#components) {
print "Checking component $comp\n" if ($verbose == 1);
my $bla = `ps aux | grep component | grep $comp-xws | grep -v "ps aux" | wc -l`;
$bla =~ s/\n|\r//g;
if ($bla eq "1") {
print " ... running\n" if ($verbose == 1);
} else {
print " ... restarting component $comp\n" if ($verbose == 1);
system "cd /home/egonw/runtime/$comp; sh runCDKcomponent.sh &";
}
}
First, when a problem occur, it is in general a good idea to have a human look at it to find the root cause as restarting a service without any action will in many cases not magically solve the issue. The common way to handle this situation is to use a monitoring solution offering some kind of alerting (by email, sms, etc) to let a human know that something is wrong and needs a human action. For example, have a look at HypericHQ, OpenNMS, Zenoss, Nagios, etc.
Second, if you want to offer some kind of highly available service, running multiple instances of the service (this is often referred to as clustering) would be a good idea. When doing so, if one instance goes down, the service won't be totally interrupted, obviously. Note that when using a cluster, if one node goes down because of too heavy load, it's very unlikely that the remaining part of the cluster will be able to handle the load so clustering isn't an absolute guarantee in all situations. Implementing this (at least for the web application) depends on the application server or servlet engine you are using.
But actually, if you are looking for something simple and pretty straight forward, I'd warmly suggest to check monit which is really a better alternative to a custom cron job (don't reinvent the wheel, monit is precisely doing what you want in a smart way). See this article for an introduction:
monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. For example, monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.
Java Service Wrapper may help with keeping the Java program up 24x7 (or very close).
Several years ago I worked on a project using Java 1.2 and our goal was to run 24x7. We never made it. The longest we managed to keep Java running was about 2-3 weeks. Twice it crashed after about 15 days. The first time we just restarted it, the second time a colleague did some research and found that the crash was due to an int variable overflowing in the Calendar class: the JdbcDriver had called new Date(year, month, day, hour minute, second) more than about 300 million times and each call had incremented the int 6 times. I think this particular bug may be fixed but you may find there are others that you encounter as you try to keep the JVM running for a long time.
So you may need to design your application to be restarted occasionally to avoid this kind of thing.
Related
I'm having trouble with a Jetty 9 server application that seems to go into some kind of resting state after a longer period of idleness. Normally the memory usage of the Java process is ~500 MB, but after being idle for some time it seems to drop down to less than 50MB. The first request that comes takes up to several seconds to respond whereas requests are normally on the scale of tens of milliseconds. But after one or two requests it seems like the application is back to it's normal responsive state.
I'm running on the 32-bit Oracle Java 8 JVM. My JVM configuration is very basic:
java -server -jar start.jar
I was hoping that this issue might be solvable through JVM configuration. Does anyone know if there's any particular parameter to disable this type of behavior?
edit: Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. See my own answer below for a description of my solution.
Based on the comment from Ivan, I was able to identify the source of the issue. Turns out Windows was swapping parts of the Java process out to disk. This was clearly visible when comparing the private working set to the commit size in the task manager.
My solution to this was two-fold. First, I made a simple scheduled job inside my server app that runs every minute and does a simple test run to make sure that the important services never go inactive for long periods. I'm hoping this should ensure that Windows doesn't regard the related pages as inactive.
Afterwards, I also noticed that the process was executing with "Below normal" priority. So I changed the script that starts the server to ensure that it's running with "High" priority going forward. This seems likely to affect swapping behavior and may very well also have been enough to resolve the issue on it's own, but I only found it after already deploying my first solution so that remains unclear. In any case, everything seems to be working as it should now.
I have a Java application that needs to run several times. Every time it runs, it checks if there's data to process and if so, it processes the data.
I'm trying to figure out what's the best approach (performance, resource consumption, etc.) to do this:
1.- Launch it once, and if there's nothing to process make it sleep (All Java).
2.- Using a bash script to launch the Java app, and when it finishes, sleep (the script) and then relaunch the java app.
I was wondering if it is best to keep the Java app alive (sleeping) or relaunching every time.
It's hard to answer your question without the specific context. On the face of it, your questions sounds like it could be a premature optimization.
Generally, I suggest you do what's easier for you to do (and to maintain), unless you have good reasons not to. Here are some possible good reasons, pick the ones appropriate to your situation:
For sleeping in Java:
The check of whether there's new data is easier in Java
Starting the Java program takes time or other resources, for example if on startup, your program needs to load a bunch of data
Starting the Java process from bash is complex for some reason - maybe it requires you to fiddle with a bunch of environment variables, files or something else.
For re-launching the Java program from bash:
The check of whether there's new data is easier in bash
Getting the Java process to sleep is complex - maybe your Java process is a complex multi-threaded beast, and stopping, and then re-starting the various threads is complicated.
You need the memory in between Java jobs - killing the Java process entirely would free all of its memory.
I would not keep it alive.
Instead of it you can use some Job which runs at defined intervals you can use jenkins or you can use Windows scheduler and configure it to run every 5 minutes (as you wish).
Run a batch file with Windows task scheduler
And from your batch file you can do following:
javac JavaFileName.java // To Compile
java JavaFileName // to execute file
See here how to execute java file from cmd :
How do I run a Java program from the command line on Windows?
I personally would determine it, by the place where the application is working.
if it would be my personal computer, I would use second option with bash script (as resources on my local machine might change a lot, due to extensive use of some other programs and it can happen that at some point I might be running out of memory for example)
if it goes to cloud (amazon, google, whatever) I know exactly what kind of processes are running there (it should not change so dynamically comparing to my local PC) and long running java with some scheduler would be fine for me
I have a Java application that uses the Apache Daemon service installer to register it as a Windows service. I am using Puppet to run an exec{} block to register the service, which works, and then chains a service{} block to start the service. Puppet uses "net.exe start" to run the service, but that command reports an error, even though the service starts correctly.
The output from running the command in a powershell shell is:
PS C:\ProgramData\PuppetLabs\puppet\etc\modules> net start myservice
The myservice_descriptive_name service is starting.....
The myservice_descriptive_name service could not be started.
More help is available by typing NET HELPMSG 3523.
As I refresh the Windows service panel while this command is running, I see the state change from:
blank field -> starting -> started
Is this a problem caused by the apache wrapper, which is starting a jvm in a separate shell or some other side effect? And, more importantly, can I get around this problem in Puppet while still using the service{} block? Is it possible to substitute sc.exe, which does not suffer the same problem, short of using an exec{} block?
To take the questions in order:
The net start command reports failure because the service appears to have hung.
Yes, the problem is caused by the Apache wrapper.
Specifically, the wrapper is telling Windows that it will reach the first checkpoint within two seconds. Since there does not appear to be any way for the Java code to implement a checkpoint, or to change the wait hint, this means that the service must start within two seconds to be compliant with the Windows service specification.
(In principle, Windows is entitled to terminate your service at this point. So far as I know, no current versions of Windows do so, though they may log error messages.)
Short of modifying Puppet or (preferably) the Apache wrapper, the only obvious workaround is to ensure that your service "starts" immediately, rather than waiting for initialization to complete.
This is less than ideal, since it means that the service can't provide feedback to Puppet if it really does fail to initialize, but no worse than your suggestion of using sc start instead of net start.
JPBlanc's answer explains why the net.exe times out waiting on the service to start, even though it does end up starting. You can definitely try swapping out net.exe calls for sc.exe (Service Control) instead.
I've created a ticket to address this - https://tickets.puppetlabs.com/browse/PUP-5475
If you find that it doesn't also timeout while waiting, please comment and/or file a pull request containing the change. At any rate, using something better than net.exe would be preferred.
The explanation is that the service takes too much time to start and does not communicate correctly with the starter.
When you write a service that initiate communications or DB connections you have to communicate with the Service Control Manager (SCM) to give the information that you are starting. Doing this kind of "I'am still starting message" the SCM can wait as mus time as you need to start. But much service writer or or tools to encapsulate exe files as services ignore that, so the SCM return "service could not be started". In Win32 this is handled by SetServiceStatus function, you will have much details there.
We have an application deployed on Tomcat 6. It's built on Spring/Struts 2, and has several Quartz tasks scheduled.
We'd like to move some tasks away from Quartz and onto Linux's cron, doing the very least amount of coding as possible. How do I run those Spring/Quartz tasks outside the Tomcat container and in a standalone Java application?
(UPDATE: Since someone wanted to know why we want to do this)
We wanted to move the scheduled tasks to their own Java applications because our Tomcat keeps dying on us. There are no errors logged. We suspect that this one huge Quartz task we have is the culprit, but whether it's because of a memory leak or our Tomcat seg-faulting due to being set-up incorrectly, we still don't know.
We wanted to isolate it by kicking it out of the Tomcat container, and see if Tomcat will still die intermittently. However, since the application is already live (though in closed beta), we wanted to troubleshoot this with the least amount of coding work, while still keeping it running (coz, you know, "new code, new problems" -- FYI, we're already considering a rewrite/re-engineering, but "firefighting" is a more urgent concern right now).
I'm not familiar with Quartz but I am familiar with stuts2 and cron.
Generally in linux you call separate processes with cron so I'd think it would be best to reduce the quartz jobs down into separate stand alone programs. Considering the Java EE nature of your project and the dependency on on aquiring services via spring I don't think this is a particularly attractive option.
A second route that I've seen with PHP but would work equally well with struts2, would be using lynx to call a specific url, which could trigger the job something like:
*/15 * * * * lynx -dump http://localhost/MyApp/MyAction
Which would call your action every 15 min (the dump option prevents lynx from entering interactive mode and just dumps output to stdout so the program will just run for a moment), which could then run your job. You would then want to look at iptables (or similar) to restrict access to those services you would not want accessed externally. You can do this within struts2 as well by putting all these actions in a single package and making an interceptor to check that the requester is the local host.
I think this second method would require the least amount of change.
Apparently, the simplest way to go about this is to create a standalone Java application that calls the bean method you're executing in Quartz:
import org.springframework.context.*;
import org.springframework.context.support.*;
public class SomeJob {
public static void main(String[] args) {
ApplicationContext ctx = new ClassPathXmlApplicationContext("applicationContext.xml");
MyBean myBean = (MyBean) ctx.getBean("myBean");
myBean.someMethod();
}
}
..then, running this in cron.
Meh.
[UPDATE: I forgot to add that this 30 sec. freezing problem only happens the first time I try to load a file from the server. Subsequent loads are very quick. Maybe some strange reverse DNS lookup? I am hosting on Google's appengine.]
I started a little project recently called http://www.chartle.net which is build around an applet.
Startup time is an important factor in the user's experience of an applet. I collect statistics and am shocked that I find often very long startup times (factor 50 to 100 higher then necessary)
The applet starts in 1-3 seconds depending on the speed of your computer and connection. Still for some users it takes up to 100 sec.
I have mixed results from my own tests. Mostly it is very fast but sometimes freezes the browser for a long time and the Java console doesn't tell me why. Best guess is, that it stalls when loading a saved chart.
Please help me figuring this out - best test by opening an already saved chart (click on one of the 'create' links at http://www.chartle.net/gallery)
Cheers,
Dieter
This is generic help rather than specific for your demo (which loaded fine for me in a few attempts).
Freezing applets
In the JDK bin directory there is a very handy programme called jstack. Refresh your browser window until it crashes and then run:
jstack *process_id*
This will give you the stack trace of any frozen Java process. If Java is not a separate process then you can use the browser's process (eg for Opera).
The following few problems were/are common for me:
I reccommend you use invokeLater rather than invokeAndWait on the init method (although you can't do this if you use start/stop methods)
Opera's custom java plugin acts very poorly...
Deadlocks caused by synch blocks and invokeAndWait's
Slow applets
Possibly the browser is fetching resources from the server, unable to use the jar file?
It may be that only the old plugin causes these problems. That means basically all people running on OSX and other users with Java prior to 1.6_update_10.
So, I would really appreciate people with such setups to watch their Java console and describe the first startup behaviour.
Cheers,
Dieter