Develop Reference

What is the recommended way to develop wars for an embedded jetty server in eclipse?

I'm working on a set of independent applications that share some common endpoints. The way I have it setup is to have each application be an independent war, and to have an embedded jetty server which provides those common endpoints. The embedded jetty server sets up the endpoints and loads the wars upon start. Each of our customers will have different applications installed, but will always have the common part, hence the need to have independently installable application wars.
This basic architecture is working ok, but in the development cycle I need to build each of the applications war to then run the whole thing with the embedded jetty server. I would like for the whole thing just to auto deploy when changes are made to any of the application files.
Any thoughts on how to do this?

Use a DeploymentManager with a WebAppProvider to find the webapps and deploy them (it will auto-redeploy on change).
Then setup your deployment to use exploded webapp directories, not war files.
You can even opt to setup deployment XML's in a single place, pointing to the contents of each webapp project's target/${project.build.finalName} directory (if using maven).
This works for many things, but not ALL things.
If you change classes or libs then there's a category of issues around memory leaks, and pinned classloaders, that can result in your reloaded webapp not behaving as you expect.
See
https://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html

How to move the Liferay hooks into a directory inside webapps?

I have created many hooks for Liferay with Eclipse Java EE IDE for Web Developers.
The problem is that when I create a hook is deployed inside the webapps folder.
My webapps folder tree:
There are many hooks. How can how can I move them into a new folder inside webapps?
in order to have something like this:
webapps
calendar-portlet
hooks
blogs-hook
breadcrumb-hook
dockbar-hook
login-hook
blogs-hook
sitepages-hook
truefalsetocheckbox-hook
userinfo-hook
wiki-hook
marketplace-portlet
my-theme
resources-importer-web
ROOT
Any help is greatly appreciated! Thank you so much!

I suppose you're speaking about Tomcat's (or whatever appserver you're using) webapp folder. You cannot merge different apps into one folder: it has to keep the structure on deploy.
So what you can do is to aggregate those hooks on source level (i.e., create only one hook project in Eclipse with the Liferay IDE so you can put everything there). So you could put them into the same plugin.
Btw I would discourage using several hooks, because they might occasionally overwrite each other's behaviour. Imagine you have some sort of property defined in a-hook and b-hook in a properties file. You cannot tell which one will be used after the deployment.

Adding a bit of reasoning to rlegendi's answer, here's what you might consider:
If you use Liferay's auto-deploy mechanics, e.g. through the plugins-sdk deployment, Liferay places the hooks in tomcat's webapps folder (in case you use tomcat) and you'd also have to override this.
You might be able to configure tomcat to monitor different directories, even on the same virtual host (I've only tried this for different virtual hosts and don't know if this will work properly with Liferay), but the auto-deploy problem would persist, you'd basically have to change the whole deployment logic, or deploy manually to tomcat as you'd do with "big iron" appservers.
If you're happy about the development-time separation between the hooks, but would like some better runtime overview, you might want to merge different hooks into one for deployment. However, you'd have to do this on your own.
But my final recommendation is to not care too much about what's in the webapps directory anyway - you shouldn't rely on directory listings of this directory for knowing what's running on your appserver. Rather use proper deployment techniques that monitor your implementations, and limit access for manual changes. E.g. deploy your applications scripted and install a system that alerts you should required components not be available and running.

Tomcat parent webapp shared by configurable children webapps

Currently, we support many clients using the same web app, but each client has a different configuration for accessing their database, setting files etc. As the client list grows, updating the web apps is becoming increasingly arduous, and the duplication of resources is a waste of memory, file space, etc..
What we'd like to do is have a parent web app which is shared by all children web apps. Then have each child web app carry only files specific to them. When the child web app starts up, Tomcat loads the web app from the parent web app and then overrides any files defined in the child web app following an identical package structure.
We've been googling around and haven't found a ready or complete solution. Solutions we've looked at:
Tomcat common/share - could handle class and JAR files, but we don't see a way to handle static and JSP resources residing above the WEB-INF dir.
CATALINA_BASE appears to be more suited for running multiple instances of Tomcat which we'd rather avoid
A Maven possible solution, but we are not big fans of Maven, so would rather avoid it also.
Anybody have suggestions or ideas on how to solve this? If Tomcat configuration is not possible, what about a different application server (such as Glassfish) or a tool for doing dynamic file updated (such as OSGi, rsync). Would like to remove the resource duplication if possible.
Thank you.

There is no such thing as "parent" or "child" webapps. It's not part of J2EE spec and AFAIK it's not supported by any application server.
That said, your problem is twofold:
1) Having shared resources. This part is pretty easy assuming "resources" means static resources (images / CSS / javascript / etc...).
If they are truly shared (e.g. you don't need to have a separate version in some of your webapps), host them elsewhere (separate "common" webapp or put Apache in front of your Tomcat and host them there.
If you do need to have "local" versions of some of those resources you may be able to do some clever conditional URL rewriting or simply write a servlet that would check whether particular resource exists locally and, if not, take it from "common" location.
Precompile your JSPs so you only have to deal with JARs.
If your Tomcat instance only hosts your apps, you can indeed put your JARs in shared (or lib in the latest version); otherwise you can deploy them with each application .
2) Simplifying deployment. I'm not really sure what the big problem is here... It's rather trivial to write an Ant (batch, shell, what have you) script that would assemble and deploy WARs based on "common" and "per-app" directory structures.
Alternatively, you may want to take a look at using JNDI to severely reduce the number of files that have to be deployed (in theory, to a single context.xml for each application).

You can build parent-child hierarchy if you use Spring at your web-apps - Using a shared parent application context in a multi-war Spring application.
I.e. you can define all shared stuff at the 'parent' context and have 'child' contexts just to use it.

If all you had was setting file and configuration changes you could manage these through the context.xml and then you can point the docBase of each application context at a common directory for all the applications to share the same source.
the drawback to this is changes to the application will require a tomcat restart.
This does not however solve your problem if you want to override logic.
A option that I am exploring for a similar scenario is to move the client custom portion into ajax widgets / gadgets. Then have it be part of the configuration files to tell the application which version of the gadget to pull for which client.
you can review documentation for having applications share a docbase here http://tomcat.apache.org/tomcat-5.5-doc/config/context.html

How do you manage embedded configuration files and libraries in java webapps?

I'm currently working on a j2ee project that's been in beta for a while now. Right now we're just hammering out some of the issues with the deployment process. Specifically, there are a number of files embedded in the war (some xml-files and .properties) that need different versions deploying depending on whether you are in a dev, testing or production environment. Stuff like loglevels, connection pools, etc.
So I was wondering how developers here structure their process for deploying webapps. Do you offload as much configuration as you can to the application server? Do you replace the settings files programmatically before deploying? Pick a version during build process? Manually edit the wars?
Also how far do you go in providing dependencies through the application servers' static libraries and how much do you put in the war themselves? All this just to get some ideas of what the common (or perhaps best) practice is at the moment.

I think that if the properties are machine/deployment specific, then they belong on the machine. If I'm going to wrap things up in a war, it should be drop-innable, which means nothing that's specific to the machine it's running on. This idea will break if the war has machine dependent properties in it.
What I like to do is build a project with a properties.example file, each machine has a .properties that lives somewhere the war can access it.
An alternative way would be to have ant tasks, e.g. for dev-war, stage-war, prod-war and have the sets of properties part of the project, baked in in the war-build. I don't like this as much because you're going to end up having things like file locations on an individual server as part of your project build.

I work in an environment where a separate server team performs the configuration of the QA and Production servers for our applications. Each application is generally deployed on two servers in QA and three servers in Production. My dev team has discovered that it is best to minimize the amount of configuration required on the server by putting as much configuration as possible in the war (or ear). This makes server configuration easier and also minimizes the chance that the server team will incorrectly configure the server.
We don't have machine-specific configuration, but we do have environment-specific configuration (Dev, QA, and Production). We have configuration files stored in the war file that are named by environment (ex. dev.properties, qa.properties, prod.properties). We put a -D property on the server VM's java command line to specify the environment (ex. java -Dapp.env=prod ...). The application can look for the app.env system property and use it to determine the name of the properties file to use.
I suppose if you have a small number of machine-specific properties then you could specify them as -D properties as well. Commons Configuration provides an easy way to combine properties files with system properties.
We configure connection pools on the server. We name the connection pool the same for every environment and simply point the servers that are assigned to each environment to the appropriate database. The application only has to know the one connection pool name.

wrt configuration files, I think Steve's answer is the best one so far. I would add the suggestion of making the external files relative to the installation path of the war file - that way you can have multiple installations of the war in the one server with different configurations.
e.g. If my dev.war gets unpacked into /opt/tomcat/webapps/dev, then I would use ServletContext.getRealPath to find the base folder and war folder name, so then the configuration files would live in ../../config/dev relative to the war, or /opt/tomcat/config/dev for absolute.
I also agree with Bill about putting as little as possible in these external configuration files. Using the database or JMX depending on your environment to store as much as it makes sense to. Apache Commons Configuration has a nice object for handling configurations backed by a database table.
Regarding libraries, I agree with unknown to have all the libs in the WEB-INF/lib folder in the war file (self-packaged). The advantage is that each installation of the application is autonomous, and you may have different builds of the war using different versions of the libraries concurrently.
The disadvantage is that it will use more memory as each web application will have its own copy of the classes, loaded by its own class loader.
If this poses a real concern, then you could put the jars in the common library folder for your servlet container ($CATALINA_HOME/lib for tomcat). All installations of your web application running on the same server have to use the same versions of the libraries though. (Actually, that's not strictly true as you could put overriding versions in the individual WEB-INF/lib folder if necessary, but that's getting pretty messy to maintain.)
I would build an automated installer for the common libraries in this case, using InstallShield or NSIS or equivalent for your operating system. Something that can make it easy to tell if you have the most up to date set of libraries, and upgrade, downgrade, etc.

I usually make two properties files:
one for app specifics (messages, internal "magic" words) embedded in the app,
the other for environment specifics (db access, log levels & paths...) exposed on each server's classpath and "sticked" (not delivered with my app). Usually I "mavenise" or "anttise" these one to put specific values, depending on the target env.
Cool guys use JMX to maintain their app conf (conf can be modified in realtime, without redeploying), but it's too complex for my needs.
Server's (static ?) libraries: I strongly discourage server library use in my apps as it adds dependency to the server:
IMO, my app must be "self-packaged": dropping my war, and that's all. I have seen wars with 20 Mbs of jars in it, and that's not disturbing for me.
A common best-practice is to limit your external dependencies to what is offered by the J2EE dogma: the J2EE API (use of Servlets, Ejbs, Jndi, JMX, JMS...). Your app has to be "server agnostic".
Putting dependencies in your app (war, ear, wathever) is self-documenting: you know what libraries your app depends on. With server libs, you have to clearly document these dependencies as they are less obvious (and soon your developers will forget this little magic).
If you upgrade your appserver, chances that the server lib you depends on will also change. AppServer editors are not supposed to maintain compatibility on their internal libs from version to version (and most of the time, they don't).
If you use a widely-used lib embedded in your appServer (jakarta commons logging, aka jcl, comes to mind) and want to ugrade it's version to get the latest features, you take the huge risk that your appServer will not support it.
If you relies on a static server object (in a static field of a server class, e.g. a Map or a log), you'll have to reboot your appserver to clean this object. You loose the ability to hot-redeploy your app (old server object will still exists between redeployments). Using appServer-wide objects (other than those defined by J2EE) can lead to subtle bugs, especially if this object is shared between multiple apps. That's why I strongly discourage the use of objects which resides in a static field of an appServer lib.
If you absolutely need "this object in this appserver's jar", try to copy the jar in your app, hoping there's no dependency on other server's jar, and checking your app's classloading policy (I take the habit to put a "parent last" classloading policy on all my apps: I'm sure I won't be "polluted" by server's jars - but I don't know if it is a "best practice").

I put all configuration in the database. The container (Tomcat, WebSphere, etc) gives me access to the initial database connection and from then on, everything comes out of the database. This allows for multiple environments, clustering, and dynamic changes without downtime (or at least without a redeploy). Especially nice is being able to change the log level on the fly (although you'll need either an admin screen or a background refresher to pick up the changes). Obviously this only works for things that aren't required to get the app started, but generally, you can get to the database pretty quickly after startup.

Place MySQL connector JAR in WEB-INF/lib of my WAR instead of in $CATALINA_HOME/common/lib?

I'm about to use MySQL with Hibernate on a Tomcat 5.5.x server.
Do I have to put mysql-connector-java-[version]-bin.jar in $CATALINA_HOME/common/lib/ or could I place it in WEB-INF/lib inside my WAR file with my other library dependencies?
It would be easier to have it in my WAR in WEB-INF/lib, as I could get it using the Maven repository that way. Are there any big draw backs to having it there instead of in the common libraries directory of Tomcat?

If your jar is in common then it's loaded globally in tomcat. Every webapp sees it. If you put it in your webapp only your webapp sees it. What I assume Boris is referring to is if some part of your global tomcat config loads something that needs a mysql connection (like a connection pool) then it's going to need the jdbc driver, so it'll need it in common. Otherwise, no.
I believe there's a security configuration you can set up in the tomcat config that stores its config in a db. If you use this, it'll need a driver.
Why would you want stuff in webapp/WEB-INF/lib? well, because it's modular, it's part of your webapp, if you move your webapp somewhere else it doesn't need an extra part of it that's part of your tomcat installation. Maintenance is much easier if you can drop your webapp directly in a stock tomcat installation. Another reason- if you have multiple webapps, they're all going to use the jars in common, which could cause library conflicts and version issues.
In general, put as little in common as you can get away with. Even if you only have a single app.