Develop Reference

How to deploy a Java webapp on Amazon EC2 so that you can efficiently use autoscaling?

I'm trying to get my head around a simple WebApp Java deployment case on Amazon Web Services.
I have manually deployed on a single EC2 instance, manually installing Java, Tomcat, MySql on a vanilla RedHat image -- as if I owned the hardware.
Now I'm trying to set up an auto scalable, easy-to-manage, production proof environment:
1 -> n Tomcat instances with a standard Java WebApp deployed,
1 MYSQL database with 1 schema and a few tables (not much), the webapp connects to it.
I understand that autoscaling depends on some trigger (e.g. more than % of usage). However, when the new server is started, my app is not deployed because it's not in the VM image. Something needs to happen so that the new servers run my code.
What is the best option to mass deploy to 1-n autoscaling group of tomcat only servers on amazon?
What is the best option to upgrade your app ? I guess you don't want all the servers to go down in one go...
How about creating a VM image that automatically retrieves the latest build available from my CI (uploaded to a S3 or something else)? I could easily add this script to Tomcat bootstrap / linux startup...
How about commercial tools like Chef or Puppet? Any open-source equivalent?

I reviewed a lot of things on the web and yes, ElasticBeanStalk is the right thing to use as Georges said. Though, it feels like it's quite confusing at start.
First of all, you can run a load balanced / scalable environment on the Free tier using t2.micro instances. What matters is the number of hours in total. You can use 100 servers for 7 hours and it'll be fine. Be careful with Health Check because it pings your environnement all the time. If you run tests / evaluation and don't want to pay, make sure you save / backup and terminate your environment as soon as possible. Data Transfer is not free of charge apparently so make sure you don't use any multi-AZ and multi-zones if you don't want to pay.
That said, BeanStalk is a kind of "black box" that does a lot for you. It is intended to create all the configuration in the console for you. A kind of "super-wizard" for load balanced / scalable environments.
Configuration goes like this : you say you have a war file you want to deploy, you upload it, you pick a container (e.g. linux+tomcat), you set up a database and it will create a basic configuration with all you want : Load Balancing, autoscaling and monitoring. It's easy to change to settings to match your needs.
One tricky thing is database configuration : you need to set environnement variables and use them in your spring configuration as per : https://raymondhlee.wordpress.com/2013/06/01/migrating-a-java-web-app-for-deploy-to-aws-elastic-beanstalk/
Application update can occur in "rolling way" i.e. occurring on limited percentage of your fleet of servers (default is 30%) which means you don't have to worry too much about downtime.

Boxfuse does what you want.
For you Java web application you literally only have to execute:
boxfuse create my-tomcat-app -apptype=load-balanced
boxfuse scale my-tomcat-app -capacity=1-16:t2-micro:cpu25-75
boxfuse run my-tomcat-app-1.0.war -env=prod
This will
Configure your application to use an ELB
Set it to autoscale between 1 and 16 t2.micro instances based on CPU usage (scale in at 25% and below, scale out at 75% and above)
Create AMI with your app and Tomcat set up so they are ready to boot
Create an ELB
Create a security group with the correct ports
Create an auto-scaling group
Launch your instance(s)
Any subsequent update will be done as a zero downtime blue/green deployment.
More info: https://boxfuse.com/blog/auto-scaling

You could make use of the User Data on an EC2 launch configuration. You can use this to pass a shell script to your instance that will run automatically when it is launched for the first time. A common pattern is to have the shell script download and install a build from S3, as you suggested in your question.
With this pattern, upgrading the build on your auto-scaling group is easy. Simply upload a new build to S3. Then, manually terminate instances in the auto-scaling group. The auto-scaling will automatically spin up new EC2 instances, which will download the new build from S3, and presto, your auto-scaling group is updated. If you terminate each instance one at a time, and wait for the new instance to come online before terminating the next one, the impact on your system is minimized.

How to debug a Play framework program that works in debug mode not in production?

Having a problem with an app that runs fine in debug but not in production.
Error is java.util.concurrent.TimeoutException: No response received after 1000, after trying to download from an http link. Set ws.timeout higher and it works, but only in debug and not prod. Any ideas on how to debug what is going on in production mode?

I decided to put this answer as various cases because I thought there will be more cases which can be added in future and can be used as a check-list if someone comes across the same problem.
Is your Play! app running behind a front-end web-server like Nginx/Apache in PROD ?
This might be a very common scenario because the front-end web server stops accepting connections after a certain number of requests have not been completed by Play! In a DEV/DEBUG mode you may not usu. have a front-end web-server.
Is your PROD configuration same as DEV configuration ?
Cross-check your configuration for PROD and DEV, esp. the threadpool configuration (if you have any parallelism-factor, parallelism-max etc), memory (-Xmx), GC settings, Java environment etc.
Did you benchmark your Play! app ?
If you haven't done this yet, benchmark your Play! app in dev mode and prod mode using a HTTP benchmarking tool like wrk (and Gatling later). Once you get the numbers you can see if you want to tune your Play! app to receive more number of connections tweaking OS parameters or increasing Timeout (FWIW, increasing Timeouts without benchmarking is a bad idea.).
Though upgrading to Play 2.2 may not solve the mystery behind Debug vs. Prod mode, it is worth taking a look at it because of overall performance improvement which I have experienced.

Running webapps in separate processes

I'd like to run a web container where each webapp runs in its own process (JVM). Incoming requests get forwarded by a proxy webapp running on port 80 to individual webapps, each (webapp) running on its own port in its own JVM.
This will solve three problems:
Webapps using JNI (where the JNI code changes between restarts) cannot be restarted. There is no way to guarantee that the old webapp has been garbage-collected before loading the new webapp, so when the code invokes System.loadLibrary() the JVM throws: java.lang.UnsatisfiedLinkError: Native Library x already loaded in another classloader.
Libraries leak memory every time a webapp is reloaded, eventually forcing a full server restart. Tomcat has made headway in addressing this problem but it will never be completely fixed.
Faster restarts. The mechanism I'm proposing would allow near-instant webapp restarts. We no longer have to wait for the old webapp to finish unloading, which is the slowest part.
I've posted a RFE here and here. I'd like to know what you think.
Does any existing web container do this today?

I'm closing this question because I seem to have run into a dead end: http://tomcat.10.n6.nabble.com/One-process-per-webapp-td2084881.html
As a workaround, I'm manually launching a separate Jetty instance per webapp.

Can't you just deploy one app per container and then use DNS entries and reverse proxies to do the exact same thing? I believe Weblogic has something like this in the form of managed domains.

No, AFAIK, none of them do, probably because Java web containers emphasize following the servlet API - which spins off a thread per http request. What you want would be a fork at the JVM level - and that simply isn't a standard Java idiom.

If I understand correctly you are asking for the standard features for enterprise quality servers such IBM's WebSphere Network Deployment (disclaimer I work for IBM) where you can distribute applications across many JVMs, and those JVMs can in fact be distributed across many physical machines.
I'm not sure that your fundamental premise is correct though. It's not necessary to restart a whole JVM in order to deploy a new version of an application. Many app servers will use a class-loader strategy that allows them to discard a version of an app and load a new one.

Where to host a periodically running Python or Java service?

I'm going to build a little service which monitors an IMAP email account and acts on the read messages. For this it just has to run every say 10 min, no external trigger required, but I want to host this service externally (so that I don't need to worry about up times.)
To be machine independent I could write the service in Java or Python. What are good hosting providers for this? and which of the two languages is better supported?
The service has either to run the whole time (and must do the waiting itself) or it has to be kicked off every 10 min. I guess most (web) hosts are geared towards request driven code (e.g. JSP) and I assume they shut down processes which run forever. Who offers hosting for user-written services like the one mentioned above?

Depending on what actions you require, and your requirements for resources, Google App Engine might be quite suitable for both Python and Java services (GAE supports both languages quite decently). cron jobs can be set to run every 10 minutes (the URL I gave shows how to do that with Python) and you can queue more tasks if the amount of work you need to perform on a certain occasion exceeds the 30-seconds limit that GAE supports.
GAE is particularly nice to get started and experiment since it has reasonably-generous free quotas for most all resources your jobs could consume (you need to enable billing, provide a credit card, and set up a budget, to allow your jobs to consume more than their free quota, though).
If you decide that GAE has limitations you can't stand, or would cost you too much for billed use of resources over the free quotas, any hosting provider supporting a Unix-like cron jobs scheduler should be acceptable. Starting up from scratch a Python script every 10 minutes may be faster than starting up from scratch a JVM, but that depends on what it is that you have to do every 10 minutes (for some kinds of tasks Python will be just as fast, or maybe even faster -- for others it will be slower, and we have no way to guess what kinds of tasks you require or at what "tipping point" the possibly-faster JVM will "pay for its own startup" wrt the possibly-slower Python... basically you'll need to assess that for yourself!-).

You are lucky, since Google AppEngine provides CRON jobs both for Python and Java.
GAE - Python
GAE - Java

Check out Google App Engine. You can set up a cron job for your Java or Python script.

How to execute a Java program 24 x 7 on linux

I have a developed two small Java applications - a vanilla Java app and a Java Web application (i.e. Spring MVC, Servlets, JSP, etc.).
The vanilla application consists of several threads which read data continuously at varying rates (from once a second to twice a minute) from several websites, process the data and write it to a database.
The Web Application reads the data from the database and presents it using JSPs, etc.
I'd now like to deploy the applications to a Linux machine and have them run 24 x 7.
If the applications crash I would like them to be restarted.
What's the best way of doing this?

Your web container will run 24x7 by default. If your deployed application throws an exception, it's captured by the container. I wouldn't normally expect this process to not run. Perhaps if threads run away, then it may become unresponsive, so it's worth monitoring (perhaps by a separate process querying it via HTTP?).
Does your vanilla application need to run at regular intervals ? If so, then cron is a good bet. It'll invoke a new instance every 'n' minutes (or however you configure it). If your instance suffers a problem, then it'll simply bail out and a new instance will be launched at the next configured interval. Again, you should probably monitor this (capture log files?) in case some problem determines that it'll never succeed completely.

with Ubuntus upstart you can respawn processes automatically. A little bit more low-level is to put the respawn directly in /etc/inittab. Both work well, but upstart is more manageable (more tools), but requires a newer system (ubuntu, fedora, and debian is switching soon).
For inittab you need to add a line like this to /etc/inittab (from the inittab manpage):
12:2345:respawn:/path/to/myapp flags
For upstart you do something similar (this is a standard script in ubuntu 9.10):
user#host:/etc/init$ cat tty1.conf
# tty1 - getty
#
# This service maintains a getty on tty1 from the point the system is
# started until it is shut down again.
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -8 38400 tty1

Check out the ServletContextListener, this allows you to embed your java application inside your web application (by creating a background thread). Then you can have it all running inside the web container.
Consider investigating and using a web container supported by the operating system vendor so all the scripts to bring it up and down (including in case of problems) is written and maintained by somebody else but you.
E.g. Ubuntu has a Tomcat as a package

I have a crontab job running every 15 minutes to see if the script is still running. If not, it restarts the service. The script itself is a piece of Perl code:
#!/usr/bin/perl
use diagnostics;
use strict;
my %is_set;
for (#ARGV) {
$is_set{$_} = 1;
}
my $verbose = -1;
if ($is_set{"--verbose"}) {
$verbose = 1;
}
my #components = ("cdk", "qsar", "rdf");
foreach my $comp (#components) {
print "Checking component $comp\n" if ($verbose == 1);
my $bla = `ps aux | grep component | grep $comp-xws | grep -v "ps aux" | wc -l`;
$bla =~ s/\n|\r//g;
if ($bla eq "1") {
print " ... running\n" if ($verbose == 1);
} else {
print " ... restarting component $comp\n" if ($verbose == 1);
system "cd /home/egonw/runtime/$comp; sh runCDKcomponent.sh &";
}
}

First, when a problem occur, it is in general a good idea to have a human look at it to find the root cause as restarting a service without any action will in many cases not magically solve the issue. The common way to handle this situation is to use a monitoring solution offering some kind of alerting (by email, sms, etc) to let a human know that something is wrong and needs a human action. For example, have a look at HypericHQ, OpenNMS, Zenoss, Nagios, etc.
Second, if you want to offer some kind of highly available service, running multiple instances of the service (this is often referred to as clustering) would be a good idea. When doing so, if one instance goes down, the service won't be totally interrupted, obviously. Note that when using a cluster, if one node goes down because of too heavy load, it's very unlikely that the remaining part of the cluster will be able to handle the load so clustering isn't an absolute guarantee in all situations. Implementing this (at least for the web application) depends on the application server or servlet engine you are using.
But actually, if you are looking for something simple and pretty straight forward, I'd warmly suggest to check monit which is really a better alternative to a custom cron job (don't reinvent the wheel, monit is precisely doing what you want in a smart way). See this article for an introduction:
monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. For example, monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.

Java Service Wrapper may help with keeping the Java program up 24x7 (or very close).
Several years ago I worked on a project using Java 1.2 and our goal was to run 24x7. We never made it. The longest we managed to keep Java running was about 2-3 weeks. Twice it crashed after about 15 days. The first time we just restarted it, the second time a colleague did some research and found that the crash was due to an int variable overflowing in the Calendar class: the JdbcDriver had called new Date(year, month, day, hour minute, second) more than about 300 million times and each call had incremented the int 6 times. I think this particular bug may be fixed but you may find there are others that you encounter as you try to keep the JVM running for a long time.
So you may need to design your application to be restarted occasionally to avoid this kind of thing.