Develop Reference

Is there a typical way to handle separate configurations for different tomcat instances in a Java app?

Our system is split up into different environments, each one as a separate Tomcat instance.
Development (Windows)
QA: Accessed by our QA department (Linux)
Production: Live site, accessible to customers. (Linux)
Each of these environments rely on separate databases, and several other web services. This means we need to keep track of the various URLs, usernames, and passwords that are all different. Many of these settings are shared across several apps, so it would be ideal to have them all in one place to remove duplication.
Right now we have config files in the application itself. We use Maven profiles to fill in the different config settings when we build the app. But this is clumsy because we have to build a different WAR for each environment.
Where is a good place to store the config files so that we can deploy the same WAR file to each server?
I've done a fair amount of research on this already. But I haven't found an solution that completely makes sense to me yet.
Separate Config Directory
Define a directory to hold config files. Such as /opt/config on linux.
I like this idea, but how do I tell Tomcat where this directory is? I see references to context.xml, but every example I've seen puts the context.xml in the META-INF folder inside the WAR. Is there a way to configure this outside the WAR?
System Property to define environment
This involves setting a system property, and then using some sort of if/else or switching logic to load the appropriate config file. This seems workable, but a bit messy. But where/how do you set this property? I typically start tomcat with ./startup.sh. Do I add arguments to that command or is there another configuration somewhere?
JNDI
I don't think this is an option for us. Every tutorial I've looked at for this seems to be dependent on LDAP or something similar. To my knowledge we don't have that available to us, and it seems like too much overhead to get set up for only a half-dozen config files.

Use system property that refers to the location where your configuration file or directory is located. In this case you can manage different environment easily and no if/else logic is needed.
You application can have hard coded value of config file path, that will allow running application without any additional system property. The application however should fail to start if mandatory data is not found.
Concerning to partial sharing of data among environments.
You can split your data into several files by categories. Some files will be shared, some other different for different environments. You can even develop your own mechanism of references between data files.
However better approach is using some ready-to-use packages. For example Spring framework supports very flexible configuration mechanism. However if you already have Spring-less application introducing this framework for configuration only seems like an overkill. In this case take a look on Apache Commons Configuration package.

Worked with my team on this and we came up with what we feel is a cleaner approach. While every tutorial I found put the context.xml inside the WAR, it can also be placed in the conf folder of the Tomcat directory.
This works for us as all our servers are Tomcat based. So each server can have it's own context.xml which has a property pointing to the config folder on that particular server.

Purpose of storing variables in web.xml?

A lot of the advice on the web on storing variables which may change depending on the env/other conditions is to put them in web.xml, but isn't the web.xml within the war file? even if you find the exploded war and change it, wouldn't it get overriden if you update the war file? Or does the webcontainer provide any method to configure the web.xml without tinkering with the war file?

The web.xml variables are of very limited use, in my experience - the only advantage is that it's a standard location to look for hard-coded "configuration".
There are several common solutions to get a more sensible way to configure web apps, none of which is standard:
Use system properties (which usually involves fiddling around with startup scripts, and it can be hard to get a good overview of your entire config)
Use environment variables (same drawbacks as system properties)
Read a config file from a predefined location; often from the classpath by using getResourceAsStream (IIRC that usually means putting the config files in Tomcat's lib directory)
You can also use JNDI, which has the disadvantage of being rather heavy-weight both to set up and read (if you're using vanilla Java, anyways - Spring for example has rather good support for reading from JNDI). However, JNDI is rather good because it's per-application, and not a process-global setting. If you need to run several instances of the same app on the same server, JNDI is pretty much the only option (although you can use it to just point out a config file somewhere, which makes things easier to work with).
This may be relevant to your interests: How can I store Java EE configuration parameters outside of an EAR or WAR?

Advantages of specifying Parameter Values in web.xml
Using your own settings file requires additional coding and management.
Hard-coding parameter values directly into your application code makes them more difficult to change in the future, and more difficult to use different settings for different deployments (eg: JDBC settings, mail server address).
Other developers using your code will be able to find any relevant parameters more easily, as this is a standard location for such parameters to be set.
See also:
Advantages of specifying Parameter Values in web.xml
Web.xml.EnvEntry
Referencing Environment Variables in web.xml

As far as I know web.xml does not provide ability to store custom variables. Typical way to configure your web application is to store configuration in database, separate properties/xml/json/other file, get configuration from separate web service or provide it through environment variables.
Often a mixture of all these is used. For example you can add system variable using -D switch when running your container. This variable will contain path to file or URL where your configuration can be found.
You can supply parameters using OS environment.
You choice should depend on how many parameters do you have, what kind of application are you developing and how can you configure application server or computer OS. For example if you a hosting application on server you cannot configure these ways are not for you, so DB or web service are your only ways.

The folks that work on the Tomcat container recognize the irony that you have identified and have implemented a way to work-around the issue.
The solution that they implemented for the issues that you have alluded to is to create another xml file... the context.xml file, which is read by the server.
It appears that you can edit this file and have the new values read by the Tomcat without a restart... as long as you keep the elements out of the server.xml.
I do not use Tomcat so I might be mis-interpreting the docs
The GlassFish web container supports a similar feature, but does it via a couple admin cli command (asadmin):
set-web-env-entry
set-web-context-param
There is probably web admin console support and you can set them up by editing the domain.xml. It seems like it isn't as flexible as the Tomcat implementation... but it does make it really easy to use.
You need to disable and then enable your application for the changed values to 'take'. Do not redeploy you app, since that will delete the value that you just set.

Centralized Application properties for multiple system

I am looking for a open-source solutions which allow hosting different properties for different applications and allow changes. On any change of properties it should notify or push the change to the application appropriately.
So, instead every application managing the properties in physical file and deploying physically; these properties can be pushed to a single system. User will have GUI to load and change the properties as per right. Should allow push as mentioned.
If you have already similar open source solutions in mind please advice.

Is this something that Puppet can manage for you?

I don't think what you've described (as nice as it would be) will be likely to exist in an app server. If a program is looking for a file, it's either going to load it with a FileReader (or similar), or it will use ClassLoader.getResourceAsStream(). It might be looking for data that is returned in properties, format, XML properties format, or even something completely different like RDF with which to configure itself. Also many programs read their config on start-up and then hold the values in memory, so you would still need to reboot them to get them to change.
To get something like this to work there would need to be a standard for configuration provisioning and live updates. Once that existed the webapp authors and server vendors would each need to add support for the standard.
If you are the one writing the programs to be managed however, then you can write your programs to request configuration from a service, and have a configuration push feature.... there may be packages out there that can speed up adding that to your code, but I get the impression you are looking to manage programs written by others.

Have you considered to use JMX? I think he could be a good starting point to implement your requirements.
MBeans's attributes can store your application's properties, the MBeanServer will allow you to make them available from remotting, JConsole offers you an GUI to update properties values.
You also can write within the MBean some code that notify the corrrespondig application when a user change any properties using the GUI.

How to Manage Client Specific Configuration

For a product that is used by multiple clients where different clients ask for different customizations both user interface wise and functionality wise, how to accommodate those changes without getting the code cluttered with client specific code?
Are there any frameworks(for any programming language) that help with this?
To add more detail the UI is web based and written using JSP.

This is one of the most difficult business requirement to manage different versions of same app, so do not expect open frameworks for that case, however each company involved develops its own system for sth like that.
As for business logic modifications, you would benefit for strong interfacing and IoC (such as Spring). You would override the services for your specific case and change the required methods, then inject into IoC the modified version of the service.
As for UI, it's more difficult because you've chosen JSP, which has little flexibility. When you'd be programming in Swing or GWT, than you could do UI modification same way - override needed UI classes, change them, inject modified versions. With JSP - propably there will be lot of modifications to .jsp files in your customized version.
Now the change modification/bug fixing - there is fully usage of version controll system. Of course, your customer-specific versions are branches, while main, standard version is trunk. Bug fixes are made to trunk, then merged to customer-specific branches. With interfacing/overriding implementations most of the merges would be the easy way, however, with JSP, I would expect conflicts to be often...
Generally, code changes merge easier than anything XML-based.

How about simple OOP? Set up a realistic interface/base class and depending on some sort of configuration, instantiate either child class A or B, depending on the client. It's hard to provide more detail for a language-agnostic question like this, but I think it's very realistic.

One solution to this problem, common in the Win32/.NET world, is to move client-specific "code" into resource files. Many .NET projects (.NET has built-in support for this pattern through the System.Resources namespace) use this pattern for internationalization, by placing the UI strings into one file per language, and then loading UI strings from the appropriate file at runtime.
How does this pattern apply to a JSP application? Well, here you can keep one resources file per client (or, instead of files, use a database), and load the user-specific customizations from the resources file whenever you serve a page.
Say for example that your biggest customer wants to have their logo overlaid on some part of each webpage in your site. Your page could load the CustomerLogo property, and use that as the src attribute for the HTML image at that part of the page. If you are serving the page to the important customer, you load the URL "/static/images/importantCustomerLogo.png," and otherwise you fall back to the default resources file, which specifies the URL "/static/images/logo.png."
This way, you can abstract out the code for loading properties into one or two Java files, and just use those properties throughout the website. The only part of your codebase that is customer-specific will be the set of resources files, and those can be in a clean XML format that is easy to read and modify. The upshot is that people who didn't develop the application in the first place can modify XML without having to read the code first, so you won't have to maintain the resources files - the sales department can do that job for you.

GWT does this out of the box via a feature called deferred binding
When compiling a GWT application the compiler actually generates different versions of the code targeted for each different browsers. this is done automatically out of the box with the GWT components taking care of the different browser gory details.
This feature can be expanded to product arbitrary compilations based on custom properties. here is a simplified example: assume you have different view definitions for a normal and a detailed view
public abstract class AbstractView { ....}
public abstract class NormalView extends AbstractView { ... }
public abstract class DetailedView extends AbstractView { ....}
you can create a module definition that will generate two different versions, one using the NormalView class the other using the DetailedView (in your gwt.xml file)
<define-property name="customMode" values="normal,detailed" />
<replace-with class="com.example.NormalView">
<when-type-is class="com.example.AbstractView" />
<when-property-is name="customMode" value="normal" />
</replace-with>
<replace-with class="com.example.DetailedView">
<when-type-is class="com.example.AbstractView" />
<when-property-is name="customMode" value="detailed" />
</replace-with>
using
AbstractView view = GWT.create(AbstractView.class);
will provide the appropriate instance at runtime.
It's up to you to encapsulate your client specific code into specific classes, and to expose common interfaces for the different implementations.
You will also need to select the appropriate compiled version according to the client currently viewing (you can use jsp for this.)
please don't take the code samples above as tested, there might be problems with the syntax, it is just intended to convey the general idea
A JSP backend is an ideal hosting environment for a GWT app, you will be able to take advantage of the requestfactory mechanism for easy communication between client and server.
obviously there is a learning curve here, IMO the official documentation is a good place to start.

I guess that you may try to read OSGi related articles (or books)...This platform would give you a very pragmatic answer to your modularity issues.It's especially designed to be able to handle different modules living all together with dependencies and versioning.
As mentionned early in an answer , dependency injection through the OSGi Declarative Services is a very valuable alternative to Spring , with dynamic capabilities.. Deploying a bundle providing a service and your references will be updated automatically , dropping it and they will be refreshed too...
Have a look to this technology and ask some questions after ?
Regards
jerome

What is the proper way to store app's conf data in Java?

Where do you store user-specific and machine-specific runtime configuration data for J2SE application?
(For example, C:\Users\USERNAME\AppData\Roaming</em> on Windows and /home/username on Unix)
How do you get these locations in the filesystem in platform-independent way?

First on the format:
Java property files are good for key/value pairs (also automatically handle the newline chars). A degree of structure is possible by using 'dot notation'. Drawback is that the structure does not allow you to easily enumerate top-level configuration entities and work in drill-down manner. Best used for a small set of often tweakable environment-specific settings
XML files - quite often used for more complex configuration of various Java frameworks (notably J2EE and Spring). I would advice that you at least learn about Spring - it contains many ideas worth knowing even if you decide not to use it. If you decide to roll your own XML configuration, I'd recommend using XStream with customized serialization options or if you just need to parse some XML, take a look at XOM. BTW Spring also allows you to plug your custom XML configuration language, but it's a relatively complex task. XML configuration is best used for more complex 'internal' configuration that is not seen or tweaked by the end user.
Serialized Java objects - a quick and easy way to persist the state of an object and restore it later. Useful if you write a configuration GUI and you don't care if the configuration is human readable. Beware of compatibility issues when you evolve classes.
Preferences - introduced in Java 1.4, allow you to store typed text, numbers, byte arrays and other primitives in platform-specific storage. On Windows, that is the registry (you can choose between /Software/JavaSoft/Prefs under HKLM or HKCU). Under Unix the same API creates files under the user home or /etc. Each prefs hive can be exported and imported as XML file. You can specify custom implementation of the PreferencesFactory interface by setting the "java.util.prefs.PreferencesFactory" JVM property to your implementation class name.
In general using the prefs API can be a good or a bad thing based on your app scenario.
If you plan to have multiple versions of the same code running on the same machine with different configuration, then using the Preferences API is a bad idea.
If you plan using the application in a restricted environment (Windows domain or tightly managed Unix box) you need to make sure that you have proper access to the necessary registry keys/directories. This has caught me by surprise more than once.
Beware from roaming profiles (replicated home dirs) they make up for some funny scenarios when more than one active machines are involved.
Preferences are not as obvious as a configuration file under the application's directory. most of the desktop support staff doesn't expect and doesn't like them.
Regarding the file layout of the prefs it again depends on your application. A generic suggestion is:
Package most of your XML files inside application's JAR either in the root or under /META-INF directory. These files will be read-only and are considered private for the application.
Put the user modifiable configuration under $APP_HOME/conf . It should consist mainly of properties files and occasionally a simple XML file (XStream serialization). These files are tweaked as part of the installation process and are usually not user serviceable.
Under the user-home, in a dot-directory (i.e. '~/.myapplication') store any user configuration. The user configuration may override the one in the application conf directory. Any changes made from within the application go here (see also next point).
You can also use an $APP_HOME/var directory to store any other mutable data which is specific to this application instance (as opposed to the user). Another advantage of this approach is that you can move and backup the whole application and it's configuration by simple copy of one directory.
This illustrates some standard techniques for managing configuration. You can implement them using different libraries and tools, starting from raw JRE, adding Spring/Guice or going for a full J2EE container (possibly with embedded Spring)
Other approaches for managing configuration are:
Using multiple base directories for running multiple instances of the application using different configurations.
Using lightweight registries for centralized configuration management
A centrally managed Configuration Management Database (CMDB) file, containing the host-specific values for each machine is rsync-ed every night to all production hosts. The application uses a templated configuration and selects from the CMDB during runtime based on the current hostname.

That depends on your kind of J2SE Application:
J2SE executable JAR file (very simple): use user.home System property to find home-dir. Then make a subdir accordingly (like e.g. PGP, SVN, ... do)
Java Web Start provides very nice included methods to safe properties. Always user-specific
Finally Eclipse RCP: There you have the notion of the workspace (also derived from user.home) for users and configuration (not totally sure how to access that tricky in Vista) for computer wide usage
All these approaches are, when used with care -- use correct separatorChar -- OS neutral.

Java has a library specifically for doing this in java.util.prefs.Preferences.
Preferences userPrefs = Preferences.getUserNodeForPackage(MyClass.class); // Gets user preferences node for MyClass
Preferences systemPrefs = Preferences.getSysteNodeForPackage(MyClass.class); // Gets system preferences node for MyClass
Preferences userPrefsRoot = Preferences.getUserRoot(); // Gets user preferences root node
Preferences systemPrefsRoot = Preferences.getSystemRoot(); // Gets system preferences root node

I use this
String pathFile = null;
if(OS.contains("win")){
pathFile = System.getenv("AppData");
}else{
pathFile = System.getProperty("user.home");
}
I save the settings of my application here
C:\Users\USERNAME\AppData\ on windows
user.home (/home/USERNAME) on other platfroms

For user specific config, you could write a config file to the folder pointed to by the "user.home" system property. Would only work on that machine of course.

You might want to look at Resource Bundles.