Develop Reference

How to prevent the configuration files change by mistake

We have quite large system consists of some parts - some webapps deployed on application server, some standalone processes (working in the background as daemon or system services), some standalone applications.
All of them use custom configuration retrieved from external files with different format. Webapps usually use standard Java property pattern - name=value. Other processes rather use custom format usually based on XML but not always.
My question is how to make technical prevention for any change made by mistake in such configuration files?
The first (and maybe little immature) solution would be to install any VCS - Version Control System (Git, Mercurial) on the server. And make local repository in the place where configuration files are located.
This of course doesn't prevent any mistake, but provides configuration change versioning and helps find changes.
Does it exist any better solution? I prefer simple solutions.
More info:
Windows Server; Tomcat as web application container; standalone processes written in Java; configuration files located externally in single, common directory;

how to deal with shared-libraries for many web applications using the same libraries

We have a web application made in Java, which uses struts2, spring and JasperReport. This application runs on glassfish 4.0.
The libraries of the application are in the WEB-INF/lib folder, and also in glassfish are installed 4 more than uses the same libraries.
Glassfish is configured to use 1024mb for heapspace and 512m for permgen, and the most of the memory consumption when i use libraries per application is in the struts actions and spring aop classes (using netbeans profiler).
The problem we are having is the amount of memory consumed by having libraries in the classloader per application because is to high and generates PermGen errors and we have also noticed that the application run slower with more users.
because of that we try to use shared-libraries, put it in domain1/lib folder and found that with a single deployed application the load time and memory consumption is much lower, and the application works faster in general. But when we deploy the rest of the applications on the server only the first application loaded works well and the rest has errors when we calls struts2 actions.
We believe that is because each application has slightly different settings on struts2 and log4j.
We have also tried to put only certain libraries on glassfish and leaving only struts2 in the application but it shows InvocationTargetException errors because all libraries depend the lib from apache-common and it dont matter if we put those lib on one place or another. Also if we put it in both places the application don’t start.
there any special settings or best practices for using shared-libraries?
Is there a way to use shared-libraries but load settings per application? or we have to change the settings to make them all the same?

Is there any special settings or best practices for using shared-libraries? Is there a way to use shared-libraries but load settings per application? or we have to change the settings to make them all the same?
These are actually interesting questions... I don't use GlassFish but, according to the documentation :
Application-Specific Class Loading
[...]
You can specify module- or application-specific library classes [...] Use the asadmin deploy command with the --libraries option and specify comma-separated paths
[...]
Circumventing Class Loader Isolation
Since each application or individually deployed module class loader universe is isolated, an application or module cannot load classes from another application or module. This prevents two similarly named classes in different applications or modules from interfering with each other.
To circumvent this limitation for libraries, utility classes, or individually deployed modules accessed by more than one application, you can include the relevant path to the required classes in one of these ways:
Using the Common Class Loader
Sharing Libraries Across a Cluster
Packaging the Client JAR for One Application in Another Application
Using the Common Class Loader
To use the Common class loader, copy the JAR files into the domain-dir/lib or as-install/lib directory or copy the .class files (and other needed files, such as .properties files) into the domain-dir/lib/classes directory, then restart the server.
Using the Common class loader makes an application or module accessible to all applications or modules deployed on servers that share the same configuration. However, this accessibility does not extend to application clients. For more information, see Using Libraries with Application Clients. [...]
Then I would try:
Solution 1
put all the libraries except Struts2 jars under domain1/lib ,
put only Struts2 jars under domain1/lib/applibs,
then run
$ asadmin deploy --libraries struts2-core-2.3.15.2.jar FooApp1.war
$ asadmin deploy --libraries struts2-core-2.3.15.2.jar FooApp2.war
To isolate Struts2 libraries classloading while keeping the rest under Common Classloader's control.
Solution 2
put all the libraries except Struts2 jars under domain1/lib ,
put only Struts2 jars under domain1/lib/applibs, in different copies with different names, eg appending the _appname at the jar names
then run
$ asadmin deploy --libraries struts2-core-2.3.15.2_FooApp1.jar FooApp1.war
$ asadmin deploy --libraries struts2-core-2.3.15.2_FooApp2.jar FooApp2.war
To prevent sharing of the libraries by istantiating (mock) different versions of them.
Hope that helps, let me know if some of the above works.

You can try to create what is known as a skinny WAR. Pack all your WARs inside an EAR and move all the common JARs from WEB-INF/lib to the lib/ folder in the EAR (don't forget to set <library-directory> in the application.xml).

I'd bet that placing the libs under lib/ or lib/ext won't resolve your performance issues. You did not write anything about the applications or server settings, like size of application, available Heap and PermGen space, but nonetheless I would recommend to stay with separate libs per app.
If you place the libs in server dirs, they will be shared among all apps. You will loose the option to upgrade only one of your applications to a new framework or to get rid away of any of them. Your deployment will be bound to a specific server architecture.
And you wrote it did not solve your problems, it even may raise new ones.
I would recommend to invest some hours into tuning the server. If it runs with defaults, allocate more PermGen and HeapSpace.
If this does not help, you should analyze in deep what's going wrong. Shared libs might be a solution, but you don't know the problem, yet. IBM offer some cool and free tools to analyze heap dumps, this could be a good starting point.

I came here in search of guidance about installing libraries that are shared among multiple applications or projects. I am deeply disappointed to read that the accepted practice favors installing a copy of every shared library into each project. So, if you have ten Web application, all of which use, e. g., httpcomponents-client, mysql-connector-java, etc., then your installation contains ten copies of each.
This behavior reminds me, painfully, of the way of thinking that motivated me to abandon the mainframe in favor of the PC; the thinking seemed to be "I don't care how many resources my application consumes. In fact, I'd like to be able to brag about what a resource hog it is." Excuse me, please, while I hurl.
The interface exposed by a library is an immutable contract that is not subject to change at the developer's whim.
There is this concept called backwards compatibility. If you break it, you create a new interface.
I know of at least two types of interfaces that adhere to the letter and spirit of these rules.
By far the oldest is the IBM System/370 system libraries. You might have Foo and Foo2, where the latter extends and/or breaks the contract made by the Foo interface in some way that made it incompatible.
From its beginnings in the Bell Labs Unix project, the standard C runtime library has adhered to the above rules.
Though it is much newer, the Microsoft COM interface specification enforces the same rule.
To their credit, Microsoft generally adheres to those rules in the Win32 API, too, although there are a handful of exceptions in that API. To a degree, they went backwards with the .NET Framework, which seems slavishly to follow in the footsteps of the Java environment that it so eagerly seeks to replace.
I've been using libraries since 1978, and my understanding was and is that the goal of putting code into a library was to make it reusable. While maintaining copies of the library code in each application eliminates the need to implement it again for each new project, it severely complicates upgrading, since you now have ten (or more) copies of the library, each of which must be updated.
If libraries adhere to the rule that an interface is an immutable contract, why shouldn't they live in a shared library directory, as do the Unix system libraries that live in its /lib directory, from which everything that runs on the host shares a single copy of the standard C runtime library, Zlib, and so forth.
Color me seriously disappointed.

How to improve our build and deployment process?

Our build/deploy process is very tedious, sufficiently manual and error-prone. Could you give proposals for improvement?
So let me describe our deployment strategy and build process.
We are developing system called Application Server (AS for short). It is essentially servlet-based web application hosted on JBoss Web server. AS can be installed in two "environments". Each environment is a directory with webapp's code. This directory is placed on network storage. Storage is mounted to several production servers where JBoss instances are installed. Directory is linked to JBoss' webapps directory. Thus all JBoss instances use the same code for environment. Configuration of JBoss is separate from environment and updated on per instance basis.
So we have two types of patches: webapp patches (for different environments) and configuration patches (for per instance configuration)
Patch is an executable file. In fact it is bash script with embedded binary rpm package. Installation is pretty straight-forward: you just execute file and optionally answer some questions. Important point is that the patch is not a system as a whole - it contains only some classes with fixes and/or scripts that modify configuration files. Classes are copied into WEB-INF/classes (AS is deployed as exploded directory).
The way we build those pathes is:
We take some previous patch files and copy them.
We modify content of patch. The most important part of it is RPM spec. There we change name of patch, change its prerequisite rpm packages and write down actual bash commands for backing up, copying and modifying files. This is one of the most annoying parts because we not always can get actual change-set. That is especially true for new complex features which are spanned among multiple change requests and commits. Also, writing those commands for change-set is tedious and error-prone.
For webapp patches we also modify spec for other environment. Usually they are identical excepting rpm package name.
We put all rpm related files to VCS
We modify build.xml by adding a couple of targets for building new patch. Modification is done by copypasting and editing.
We modify CruiseControl's config by copypasting project and changing ant targets in it
At last, we build a system
Also, I'm interested in any references on patch preparation and deployment practices, preferably for Java applications. I haven't succeed googling that.

The place I work had similar problems, but perhaps not as complex.
We responded by eliminating the concept of patch altogether. We stopped patching, and started simply installing the whole app (even if we do a just a small change).
We now have Cruise Control build complete install kits that happen to contain the build timestamp in the install-kit name. This is a Cruise Control build artifact.
Cruise Control autoinstalls them on a test server, and runs some automated smoke tests. We then run manual tests on the test server. Then we install the artifact on a staging, then production server.
Getting rid of patching caused some people to splutter, "isn't that wasteful if you're just changing a couple of things?" and "why would you overwrite all the software just to patch something?"
But the truth is that good source control, automated install-kit building, and one-step installation has saved us tons of time. It does take a few seconds longer to install, but we can do it far more repeatedly and with less developer labor.

How do you manage embedded configuration files and libraries in java webapps?

I'm currently working on a j2ee project that's been in beta for a while now. Right now we're just hammering out some of the issues with the deployment process. Specifically, there are a number of files embedded in the war (some xml-files and .properties) that need different versions deploying depending on whether you are in a dev, testing or production environment. Stuff like loglevels, connection pools, etc.
So I was wondering how developers here structure their process for deploying webapps. Do you offload as much configuration as you can to the application server? Do you replace the settings files programmatically before deploying? Pick a version during build process? Manually edit the wars?
Also how far do you go in providing dependencies through the application servers' static libraries and how much do you put in the war themselves? All this just to get some ideas of what the common (or perhaps best) practice is at the moment.

I think that if the properties are machine/deployment specific, then they belong on the machine. If I'm going to wrap things up in a war, it should be drop-innable, which means nothing that's specific to the machine it's running on. This idea will break if the war has machine dependent properties in it.
What I like to do is build a project with a properties.example file, each machine has a .properties that lives somewhere the war can access it.
An alternative way would be to have ant tasks, e.g. for dev-war, stage-war, prod-war and have the sets of properties part of the project, baked in in the war-build. I don't like this as much because you're going to end up having things like file locations on an individual server as part of your project build.

I work in an environment where a separate server team performs the configuration of the QA and Production servers for our applications. Each application is generally deployed on two servers in QA and three servers in Production. My dev team has discovered that it is best to minimize the amount of configuration required on the server by putting as much configuration as possible in the war (or ear). This makes server configuration easier and also minimizes the chance that the server team will incorrectly configure the server.
We don't have machine-specific configuration, but we do have environment-specific configuration (Dev, QA, and Production). We have configuration files stored in the war file that are named by environment (ex. dev.properties, qa.properties, prod.properties). We put a -D property on the server VM's java command line to specify the environment (ex. java -Dapp.env=prod ...). The application can look for the app.env system property and use it to determine the name of the properties file to use.
I suppose if you have a small number of machine-specific properties then you could specify them as -D properties as well. Commons Configuration provides an easy way to combine properties files with system properties.
We configure connection pools on the server. We name the connection pool the same for every environment and simply point the servers that are assigned to each environment to the appropriate database. The application only has to know the one connection pool name.

wrt configuration files, I think Steve's answer is the best one so far. I would add the suggestion of making the external files relative to the installation path of the war file - that way you can have multiple installations of the war in the one server with different configurations.
e.g. If my dev.war gets unpacked into /opt/tomcat/webapps/dev, then I would use ServletContext.getRealPath to find the base folder and war folder name, so then the configuration files would live in ../../config/dev relative to the war, or /opt/tomcat/config/dev for absolute.
I also agree with Bill about putting as little as possible in these external configuration files. Using the database or JMX depending on your environment to store as much as it makes sense to. Apache Commons Configuration has a nice object for handling configurations backed by a database table.
Regarding libraries, I agree with unknown to have all the libs in the WEB-INF/lib folder in the war file (self-packaged). The advantage is that each installation of the application is autonomous, and you may have different builds of the war using different versions of the libraries concurrently.
The disadvantage is that it will use more memory as each web application will have its own copy of the classes, loaded by its own class loader.
If this poses a real concern, then you could put the jars in the common library folder for your servlet container ($CATALINA_HOME/lib for tomcat). All installations of your web application running on the same server have to use the same versions of the libraries though. (Actually, that's not strictly true as you could put overriding versions in the individual WEB-INF/lib folder if necessary, but that's getting pretty messy to maintain.)
I would build an automated installer for the common libraries in this case, using InstallShield or NSIS or equivalent for your operating system. Something that can make it easy to tell if you have the most up to date set of libraries, and upgrade, downgrade, etc.

I usually make two properties files:
one for app specifics (messages, internal "magic" words) embedded in the app,
the other for environment specifics (db access, log levels & paths...) exposed on each server's classpath and "sticked" (not delivered with my app). Usually I "mavenise" or "anttise" these one to put specific values, depending on the target env.
Cool guys use JMX to maintain their app conf (conf can be modified in realtime, without redeploying), but it's too complex for my needs.
Server's (static ?) libraries: I strongly discourage server library use in my apps as it adds dependency to the server:
IMO, my app must be "self-packaged": dropping my war, and that's all. I have seen wars with 20 Mbs of jars in it, and that's not disturbing for me.
A common best-practice is to limit your external dependencies to what is offered by the J2EE dogma: the J2EE API (use of Servlets, Ejbs, Jndi, JMX, JMS...). Your app has to be "server agnostic".
Putting dependencies in your app (war, ear, wathever) is self-documenting: you know what libraries your app depends on. With server libs, you have to clearly document these dependencies as they are less obvious (and soon your developers will forget this little magic).
If you upgrade your appserver, chances that the server lib you depends on will also change. AppServer editors are not supposed to maintain compatibility on their internal libs from version to version (and most of the time, they don't).
If you use a widely-used lib embedded in your appServer (jakarta commons logging, aka jcl, comes to mind) and want to ugrade it's version to get the latest features, you take the huge risk that your appServer will not support it.
If you relies on a static server object (in a static field of a server class, e.g. a Map or a log), you'll have to reboot your appserver to clean this object. You loose the ability to hot-redeploy your app (old server object will still exists between redeployments). Using appServer-wide objects (other than those defined by J2EE) can lead to subtle bugs, especially if this object is shared between multiple apps. That's why I strongly discourage the use of objects which resides in a static field of an appServer lib.
If you absolutely need "this object in this appserver's jar", try to copy the jar in your app, hoping there's no dependency on other server's jar, and checking your app's classloading policy (I take the habit to put a "parent last" classloading policy on all my apps: I'm sure I won't be "polluted" by server's jars - but I don't know if it is a "best practice").

I put all configuration in the database. The container (Tomcat, WebSphere, etc) gives me access to the initial database connection and from then on, everything comes out of the database. This allows for multiple environments, clustering, and dynamic changes without downtime (or at least without a redeploy). Especially nice is being able to change the log level on the fly (although you'll need either an admin screen or a background refresher to pick up the changes). Obviously this only works for things that aren't required to get the app started, but generally, you can get to the database pretty quickly after startup.

How do you maintain java webapps in different staging environments?

You might have a set of properties that is used on the developer machine, which varies from developer to developer, another set for a staging environment, and yet another for the production environment.
In a Spring application you may also have beans that you want to load in a local environment but not in a production environment, and vice versa.
How do you handle this? Do you use separate files, ant/maven resource filtering or other approaches?

I just put the various properties in JNDI. This way each of the servers can be configured and I can have ONE war file.
If the list of properties is large, then I'll host the properties (or XML) files on another server. I'll use JNDI to specify the URL of the file to use.
If you are creating different app files (war/ear) for each environment, then you aren't deploying the same war/ear that you are testing.
In one of my apps, we use several REST services. I just put the root url in JNDI. Then in each environment, the server can be configured to communicate with the proper REST service for that environment.

I just use different Spring XML configuration files for each machine, and make sure that all the bits of configuration data that vary between machines is referenced by beans that load from those Spring configuration files.
For example, I have a webapp that connects to a Java RMI interface of another app. My app gets the address of this other app's RMI interface via a bean that's configured in the Spring XML config file. Both my app and the other app have dev, test, and production instances, so I have three configuration files for my app -- one that corresponds to the configuration appropriate for the production instance, one for the test instance, and one for the dev instance.
Then, the only thing that I need to keep straight is which configuration file gets deployed to which machine. So far, I haven't had any problems with the strategy of creating Ant tasks that handle copying the correct configuration file into place before generating my WAR file; thus, in the above example, I have three Ant tasks, one that generates the production WAR, one that generates the dev WAR, and one that generates the test WAR. All three tasks handle copying the right config file into the right place, and then call the same next step, which is compiling the app and creating the WAR.
Hope this makes some sense...

We use properties files specific to the environments and have the ant build select the correct set when building the jars/wars.
Environment specific things can also be handled through the directory service (JNDI), depending on your app server. We use tomcat and our DataSource is defined in Tomcat's read only JNDI implementation. Spring makes the lookup very easy.
We also use the ant strategy for building different sites (differeing content, security roles, etc) from the same source project as well.
There is one thing that causes us a little trouble with this build strategy, and that is that often files and directories don't exist until the build is run, so it can make it difficult to write true integration tests (using the same spring set up as when deployed) that are runnable from within the IDE. You also miss out on some of the IDE's ability to check for the existence of files, etc.

I use Maven to filter out the resources under src/main/resources in my project. I use this in combination with property files to pull in customized attributes in my Spring-based projects.
For default builds, I have a properties file in my home directory that Maven then uses as overrides (so things like my local Tomcat install are found correctly). Test server and production server are my other profiles. A simple -Pproduction is all it then takes to build an application for my production server.

Use different properties files and use ant replace filters which will do the replacement based on environment for which the build is done.
See http://www.devrecipes.com/2009/08/14/environment-specific-configuration-for-java-applications/

Separate configuration files, stored in the source control repository and updated by hand. Typically configuration does not change radically between one version and the next so synchronization (even by hand) isn't really a major issue.
For highly scalable systems in production environments I would seriously recommend a scheme in which configuration files are kept in templates, and as part of the build script these templates are used to render "final" configuration files (all environments should use the same process).

I recently also used Maven for alternative configurations for live or staging environments. Production configuration using Maven Profiles. Hope it helps.

I use Ant's copy with a filter file.
In the directory with the config file with variables I have a directory with a file for each environment. The build script know the env and uses the correct variable file.

I have different configuration folders holding the configurations for the target deployment, and I use ANT to select the one to use during the file copy stage.

We use different ant targets for different environments. The way we do it may be a bit inelegant but it works. We will just tell certain ant targets to filter out different resource files (which is how you could exclude certain beans from being loaded), load different database properties, and load different seed data into the database. We don't really have an ant 'expert' running around but we're able to run our builds with different configurations from a single command.

One solution I have seen used is to configure the staging environment so that it is identical to the production environment. This means each environment has a VLAN with the same IP range, and machine roles on the same IP addresses (e.g. the db cluster IP is always 192.168.1.101 in each environment). The firewalls mapped external facing addresses to the web servers, so by swapping host files on your PC the same URL could be used - http://www.myapp.com/webapp/file.jsp would go to either staging or production, depending on which hosts file you had swapped in.
I'm not sure this is an ideal solution, it's quite fiddly to maintain, but it's an interesting one to note.

Caleb P and JeeBee probably have your fastest solution. Plus you don't have to setup different services or point to files on different machines. You can specify your environment either by using a ${user.name} variable or by specifying the profile in a -D argument for Ant or Maven.
Additionally in this setup, you can have a generic properties file, and overriding properties files for the specific environments. Both Ant and Maven support these capabilities.

Don't forget to investigate PropertyPlaceholderConfigurer - this is especially useful in environments where JNDI is not available