Related
My company is trying to determine the best strategy for implementing batch Java programs. We have a few hundred (and growing) separate Java programs. Most of them are individual Jasper Reports but some are bigger batch Java jobs. Currently, each Java Project is packaged an independent JAR file using Eclipse's export option. Those JARs are then deployed to our Linux server manually where they are tested. If they pass testing, they are then migrated up through QA and onto Production through a home grown source code control system.
Is this the best strategy for doing batch Java? Ongoing maintenance can be a hassle since searching Jar files is not easy and different developers are creating new Java Projects (new reports) every week.
Importing existing projects from the Jar files into Eclipse is a tricky process as well. We would like these things to be easier. We have thought about packaging all the code into 1 big project and writing an interface to be able to execute the desired "package" (aka program) maybe using a Web Server.
What are other people/companies doing out there with their batch Java programs? Are there any best practices out there on this stuff? Any help/ideas/working models would be appreciated.
I would say that you should be able to create one web based app for access Jasper reports, rather than a bunch of batch processes. Then, when you need to deploy a new report, just deploy a minor update that accesses a new compiled Jasper report file.
That said, you should be checking your code, not your binaries, into a Subversion or Git repository. Dump the "home grown" source control repository. Life is too short to try to home grow stuff like that. Just use Git or Subversion, they're proven, simple, and functional. When you import a new project, just pull it down from Subversion, don't try to import the JAR file from your Eclipse IDE.
Put your JAR files into a Maven repository such as Nexus, and deploy to QA and Production from there. Create automated builds for every project (be that with Maven or something else). Don't depend upon an IDE to export your JAR files. IDE's change and exporting from an IDE introduces more opportunity for human error. Also, different developers will prefer different IDE's. By standardizing on something like Maven, you're a bit more IDE agnostic.
Mhy company has standardized Java Batch execution using IBM Websphere Extended Deployment.
Here http://www.ibm.com/developerworks/websphere/techjournal/0801_vignola/0801_vignola.html is an article introducing techniques for programming and deploying java batch.
Introduction to batch programming using WebSphere Extended Deployment Compute Grid
Christopher Vignola, WebSphere
Architect, IBM
Commonly thought of as a
legacy "mainframe" technology, batch
processing is showing itself to be a
venerable workload style with growing
demand in Java™ and distributed
environments. This article introduces
an exciting new capability for Java
batch processing from IBM®, the leader
in batch processing systems for the
last 40 years. This content is part of
the IBM WebSphere Developer Technical
Journal.
WebSphere Extended Deployment Compute
rid provides a simple abstraction of
a batch job step and its inputs and
outputs. The programming model is
concise and straightforward to use.
The built-in checkpoint/rollback
mechanism makes it easy to build
robust, restartable Java batch
applications.
The Batch Simulator utility provided
with this article offers an
alternative test environment that runs
inside your Eclipse (or Rational
Application Developer) development
environment. Its xJCL generator can
help jump start you to the next phase
of testing in the Compute Grid unit
test server.
But even if you are not interested in the product, the article is a must read anyway.
Our company is using eclipse since several years now (we are using WTP since release 0.7)
I am currently evaluating eclipse 3.6.2 with WTP 3.2.3 which should replace eclipse 3.4.2 with WTP 3.0.4 as being our main IDE.
And I have to say that once again I am quite disappointed in concerns of performance:
WTP 3.2.3 seems to be much slower than 3.0.4.
In fact I am really wondering why WTP gets slower with each release.
One of our applications (dynamic web project) contain about 4000 java classes and 700 jsps/jsp fragments. We only need basic WTP functionality for developing jsps, xmls and xsd. We don't need high sophistic features like Dali (should JPA tools really covered by a webtools project?), Libra or a visual xml editor in the first place.
Another interesting point is that WTP seems to slow down the whole IDE.
SWT is non-reponsive for some fraction of seconds, cpu usage is very high (especially after a built took place - if you look at the system jobs, several jsp/javascript indexers are doing work for some minutes, even if all WTP build validators have been disabled), opening new files are slower, navigating through the project etc.
This can be especially seen on older machines which do contains only a single core cpu.
The worst thing is that I've got the feeling that the WTP dev team does not care much about performance
(e.g. have a look at the http://wiki.eclipse.org/WTP_Performance_Tests page - last update took place in 2008).
Bug reports and Newsgroup posts regarding to performance of basic features (e.g. jsp editing/validating) are often ignored or closed after some time, some examples: here, here, and here.
Quo vadis, WTP?
Please don't get me wrong:
I don't want to blame WTP.
In fact I believe that WTP is a good open-source project developed by a talented team.
But obviously the project has a problem with its quality assurance, especially in terms of performance which affects usability and user acceptance.
I just want to point out that the team should focus on the things which are essential to most of the users in the first place and afterwards work on implementing super-duper-features.
My Questions
What are your experiences with WTP, especially the most recent releases?
Can you confirm or disprove my observations?
Are there better alternatives?
Did you switch from or to WTP and why?
Do you have some best practices to speed it up, especially for upper-mid-sized like ours?
UPDATE
I'd like to make an update on this question to reflect the current answers and
to sum up the current results:
Many users complain about more or less on the same issues so I see those issues as confirmed.
BTW, this question is also mentioned on a news post on theserverside.com with additional comments.
The responsible WTP project lead, nitind, made a notable post on the current situation of WTP, which I like to cite:
"The simple fact is that we don't spend much time on performance tests because we lack resources for doing that."
"Of course we'd like to be proactive about it rather than reactive, but we tend to allocate our time to functional problems first."
So this question turns a little bit into some kind of an open letter from the community to the WTP team:
Dear WTP team,
it's obvious that WTP is suffering from major quality/performance issues
which you try to play down or to ignore.
Please invest some time to improve the current situation
at the cost of new features and do everything what's required
to solve the current problems.
E.g. revive the performance team, do some regression tests between
previous releases or ask the community for (precise defined) help.
I am sure that they are enough people willing and able to help here.
If you like, do some kind of poll to get a feeling what should be
the most important scopes of future's WTP releases.
Please, please, listen to your community.
To respond, I'm the lead for the projects that supply the JSP, XML, and JavaScript source editing functionality in WTP. The simple fact is that we don't spend much time on performance tests because we lack resources for doing that. Of course we'd like to be proactive about it rather than reactive, but we tend to allocate our time to functional problems first. We do have an adopter product running performance regression tests regularly, but I expect that tests are run on multi-core machines by now--and we haven't had any new red flags reported to us for some time.
Of the 3 bugs you linked to, 2 predate the 3.0.4 version you laud and the third is either a formatting performance issue (since addressed) or one with as-you-type validation specific to XML files (the fixing of which would have triggered a memory leak in Xerces, iirc, hence us not putting it in at that time). If you have concrete projects that you can attach to a bug and say "doing X is slower in 3.2 by Y amount", we'll do what we can to figure out where there's a regression.
As for the indexers, they should at least eventually complete. The on-disk information that is stored has changed between WTP versions, and those files need to be reprocessed so they're again included in search and (where implemented) refactoring operations. Once the initial indexing is complete, it should act incrementally and be virtually unnoticeable. One architectural change you may be running into is that for JSPs, the entire workspace needs to be indexed in a single workbench session for that index to be considered "up to date". Shutting down Eclipse out of frustration will only prolong the impact of that reprocessing.
It sounds like your company's standard install includes the entirety of WTP rather than rolling your own custom distribution. I urge you to check the Startup and Shutdown preference page and turn off early startup of any functionality you're not interested in using. Nothing you've mentioned interest in makes use of that facility, but there are other areas of WTP and the Platform that do. Anything you're not interested in validating is fair game on the Validation preference page, as well as the setting to validate JSP fragments by default on the Web / JSP Files / Validation preference page.
We have the same problem with WTP 3.2.3 here, too. We use it in our product for several years, too but the acceptance of our developers and customers in this tool is decreasing every year because in every newer release it is slower and slower.
I would like to use it if I could disable all "advanced" features but as you mentioned you cannot disable the indexers at all. Also you cannot stop the validator of JSP files if it is already running (you can test this if you have that many files as you have and we also have around 1000 JSP files and many tag files in our project).
I also can prove that increasing the memory does not help. It only prevents crashes of the whole eclipse but it does not reduce the UI blocking internal operations of WTP.
In the newest version 3.2.3 I got many hangs when I start Tomcat from within the servers view. The UI just hangs about 1 minute. It's not only me who has the hangs, it's all my colleagues who work on Windows have the same problem. On Linux I do not know about this problem.
Also there are problems in WTP when you have no access to the internet. It seems there are request to some registries to download schemas or such things and if you do not have a connection then it just hangs and waits for the time out.
I don't know who's to blame: WTP or JBoss Tools.
The fact is that, as I work with GWT (minimal JSP), I went the opposite way: No WTP at all!!! I'm now using plain Eclipse for Java, and use a run configuration to deploy (programatically invoking ANT) and start the server, and I never looked back!!!
Eclipse used to take ~1.5GB and crashed several times. Now, it sits on ~800MB, and the whole environment became more stable.
I have seen similar effects, here's one solution that might be suitable in some project environments...
To guarantee fast and responsible Eclipse Web project environment, consider this:
Use Eclipse IDE for Java Developers
98MB version is leaner than 200MB EE version
from General / Startup and Shutdown, disable all but "Eclipse UI"
from Validation, disable validators which you do not need
these probably prevent some performance issues (YMMW)
Use Jetty
you do not need WTP or any other plugins, it's pure Java
like Tomcat but very quick & simple to install into any IDE / environment
embed directly into your project files, configure once for every developer
works in any IDE (Eclipse, IDEA, JDeveloper..)
start Servlet Container with "Debug As" / "Run As"
shutdown Servlet Container by clicking red box on Eclipse console
IDE Console, debugging & hot code replacement & JRebel works fine
Result: fast & responsive Eclipse when compared to many other Eclipse installations that use Eclipse EE version with WTP stuff.
Why? It might be that some Eclipse feature or plugin contains bugs or simply uses resources in a bad way and this makes Eclipse UI sluggish.
Non-Java EE Eclipse is good enough even for many Java EE project environments, it all depends on your architecture and what tools you are using..
Here's a quick tutorial to get started in case you wish to try Jetty Servlet Container out with Eclipse. See https://vaadin.com/web/jani/home/-/blogs/vaadin-for-everyone-how-to-get-started . Download VaadinProjectForAnyIDE.zip, it's an Eclipse project. Just ignore that Vaadin stuff and replace HelloWorldApplication.java with your own servlet and edit web.xml accordingly.
One more thing. With Eclipse EE version, you might also wish to try J2EE preview server which actually is Jetty embedded into Eclipse bundle. However, this also uses WTP mechanism.
I also think the stability and performance of Eclipse/WTP is somewhat alarming. I'm using Eclipse since mid 2003 and have been trying WTP since its very first releases.
At first the quality was absolute abysmal, but for a 0.x version I couldn't complain of course. While waiting for WTP to mature, I used MyEclipse, which was sort of okay but had its flaws too (and being partially based on WTP, inherited some of WTP).
When MyEclipse became heavier and heavier, slower and slower and we ran into several stability issues, we switched to 'pure WTP'. All we were using was really the basic JSP/JSF editor and deployer.
Since WTP doesn't do incremental deployment (at least not for the JBoss server runtime) we added the separate server runtime from JBoss tools. When we adopted Facelets, we also switched to the editor from JBoss tools.
We do however run into a lot of problems we also had with MyEclipse. There are unexplainable slowdowns, but much worse are various stability problems. There are lots of awkward exceptions and crashes. A typical .log file on many different workstations I examined is chockfull of exceptions. A small selection of the last 10 exceptions in my log:
1.
java.lang.NullPointerException
at org.eclipse.jst.jsp.core.internal.validation.JSPActionValidator.isElIgnored(JSPActionValidator.java:147)
2.
java.lang.RuntimeException
at org.eclipse.jface.viewers.ColumnViewer.checkBusy(ColumnViewer.java:763)
3.
java.lang.ArrayIndexOutOfBoundsException: 38
at org.eclipse.debug.internal.ui.viewers.model.FilterTransform$Node.addFilter(FilterTransform.java:67)
4.
org.eclipse.jdi.TimeoutException: Timeout occurred while waiting for packet 302111.
at org.eclipse.jdi.internal.connect.PacketReceiveManager.getReply(PacketReceiveManager.java:171)
5.
java.lang.NullPointerException
at org.eclipse.jst.jsf.facelet.core.internal.cm.ElementCMAdapter.getLength(ElementCMAdapter.java:109)
6.
Caused by: java.lang.NullPointerException
at org.eclipse.jst.jsf.facelet.core.internal.registry.taglib.WebappConfiguration.getConfigFilesFromContextParam(WebappConfiguration.java:184)
7.
org.eclipse.emf.ecore.resource.Resource$IOWrappedException: Feature 'span' not found. (file:///mysystem/Eclipse.app/Contents/MacOS/com
/sun/faces/metadata/taglib/facelets_jsf_core.taglib.xml, 453, 52)
at org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.handleErrors(XMLLoadImpl.java:83)
...
at org.eclipse.jst.jsf.facelet.core.internal.registry.taglib.TagModelLoader.loadFromInputStream(TagModelLoader.java:100)
8.
java.lang.NullPointerException: No IModelProvider exists for project P/my_project of version: Utility Module
at org.eclipse.jst.j2ee.model.ModelProviderManager.getModelProvider(ModelProviderManager.java:101
9.
java.lang.NullPointerException
at org.eclipse.jst.j2ee.internal.deployables.J2EEDeployableFactory.clearCache(J2EEDeployableFactory.java:238)
10.
org.eclipse.jst.jee.model.internal.mergers.ModelException: java.util.ConcurrentModificationException
at org.eclipse.jst.jee.model.internal.mergers.EjbJarMerger.process(EjbJarMerger.java:79)
Note that these are just the last 10, there are many, many more exceptions.
The casual reaction would be: "Your Eclipse install is corrupted! You have a local problem!" Yes, I might have a local problem, but this "local problem" seems widespread as many Eclipse installs I inspected seem to have this stuff in their logs.
I'm also having problems with deployments like reported at the following link in various incarnations: http://community.jboss.org/thread/158611 It may be JBoss tools specific or it may be based on the underlying WTP or even Eclipse code. I don't know, but I do know it's a nasty problem. Every WTP and JBoss tools version there is 'something' fixed, and every version a problem like that resurfaces in a slightly different form.
Between the stability problems I'm able to get some work done and I love the auto completion and navigate-into features the editors offer me (which keeps me from switching to a text editor and building completely on the command line), but I sure would love some increased stability.
By far the best way for speeding up my projects has been to precompile code that I am not currently using. We have about 20 projects that make up our system and when working on any specific problem I'm only touching a specific subset of those java files. Compiling most of the code that I won't be touching and throwing it into some .jar's, then using that as the source instead of including the projects has proven to speed up things by quite a bit. I imagine it will help you as well if you have 4k+ files.
Each project just has a little build.xml that will make a jar out of it to include.
As for the mind numbing slowness in the JSP editing. I have the same problems, it's just so dam slow. I don't have many more than 100 jsp files but I have the same issues as you. My solution has just been to throw money at hardware, which I must admit I enjoy doing :P.
To answer the following question:
Do you have some best practices to speed it up, especially for upper-mid-sized like ours?
Turning off validation and auto-building after file-saving is a good start to increase performance.
I've disabled the WTP JSP editor for the reasons you mentioned above: It just needs too many resources. Some more things you should consider:
Edit JSPs in the normal HTML editor. That means you don't get code completion which is a good thing. IMO, Mixing Java and HTML is a mistake in the first place and an editor can't fix that. Put all Java code into helper beans (which you can then test easily) and just access the beans from JSP. This should get rid of 99% of all the <% %> tags and solve most of your problems already.
Consider using Spring to be able to build more complex beans and inject them into your JSPs using these patterns:
How to inject spring beans into a jsp 2.0 SimpleTag?
Use SpringBeanAutowiringSupport:
<%!
#Autowired
private Bean bean;
public void jspInit() {
SpringBeanAutowiringSupport
.processInjectionBasedOnServletContext(
this, getServletContext()
);
}
%>
Try a different VM. WTP editors creates huge amounts of objects and now all VMs (GC implementations) can handle that equally well. If you use Sun's Java, try JRockit or IBMs J9. Also play with the GC settings. Increasing RAM won't help because if you have GC issues, more RAM usually only makes it worse (since the GC will have to process more data).
Precompile as much code as possible. You don't need 4000 classes open at all times on your workspace. Cut your huge project into manageable chunks.
Replace JSPs with plain Java servlets and use HTML rendering libraries like rendersnake or use a programming language which plays more nice with HTML (like Groovy).
Get some decent hardware. A new PC with a quad core and 8GB RAM costs $1000. If you save ten minutes every day, the investment will be paid up in 50 days (at the rate of 1 person costs $1000/day all in all).
Try MyEclipse which has much better web editors. The JSP editor is better than WTP (code completion works most of the time, for example) but it's still sluggish.
WTP (3.2.3) is slow for me too. I belive I have found some ways to make it not so slow:
We use maven, so there is a target directory which contains a copy of all JSP's and some other XML's. I have recognized that they are sometimes scanned by the WTP validators. But this is not necessary, so I have excluded them from validation (Project/Properties/Validation/XXX/Exclude Group/). (You should be able to gain the same effect when marking the target directory as derived, but that does not work for me some times ;-( )
I have made the (not scientifically proven) observation, that WTP seems to be faster with JSPX than JSP files.
If you need barebones Java EE then you are better off with Netbeans, if you need everything but just working you are better off with IDEA. It is as simple as that.
Couldn't comment so I will just put my comment in to this answer.
Have you tried upping Eclipse's memory allocation, I know it used to crash all the time when I used it on Mac OS X awhile ago. There is a config file that has the base RAM allocation in it, once I modified that file and gave it an extra 128 megabytes of memory it behaved better. Not sure if this will affect WTP, I am commenting more in regards to the core Eclipse application itself.
I'm not so experienced but i worked on some big Java EE projects (using maven2) with very distinct ways to handle the installation / delivery on the different platforms.
1) One of them was to use snapshots for development and then make a maven release, of components and main webapplications. Thus the delivery is:
war/ear files
List item
properties files
sgdb files
some others
And teams will use that files to put the new application versions in the different platforms.
I think this process is strict and permits you to always keep easily the different configurations passed in production, but it's not really flexible, the process is a bit heavy and it conducted us to sometimes do some dirty things like overriding a class of a war to patch a regression...
This is an e-commerce website with 10million unique visitors per month and a 99.89% availability.
2) Another i saw is to checkout the sources on each platform and then install the snapshot artifacts in a local repository. Then the application server will use these snapshots of the .m2 folder.
There is not a real delivery process since to put a new version in production, we just have to update the sources of the components / webapps, do some maven clean install and restart the application server.
I think it's more flexible but i see some drawbacks and this approach seems dangerous for me.
This website has a frontoffice, i don't know the numbers but it's far less than the 1st one. It also has a big backoffice available for most employees of a 130 000 people company.
I guess depending on the website, its exposition to the public and the availability required, we have to adapt the delivery strategy to the needs.
I'm not here to ask which solution is the best but wonder if you have seen different things, and which strategy you would use in which case?
Without dealing dealing web sites, I had to participate in release management process for various big (Java) projects in heterogeneous environment:
development on "PC", meaning in our case Windows -- sadly still Windows Xp for now -- (and unit testing)
continuous integration and system testing on linux (because they are cheaper to setup)
pre-production and production on Solaris (Sun Fire for instance)
The common method I saw was:
binary dependency (each project uses the binaries produced by the other project, not their sources)
no recompilation for integration testing (the jars produced on PC are directly used on linux farms)
full recompilation on pre-production (meaning the binary stored on the Maven repo), at least to make sure that everything is recompiled with the same JDK and the sale options.
no VCS (Version Control System, like SVN, Perforce, Git, Mercurial, ...) on a production system: everything is deployed from pre-prod through rsynch.
So the various parameters to take into account for a release management process is:
when you develop your project, do you depend directly on the sources or the binaries of the other projects?
where do you store your setting values?
Do you parametrize them and, if yes, when do you replace the variables by their final values (only at startup, or also during runtime?)
do you recompile everything on the final (pre-production) system?
How do you access/copy/deploy on your production system?
How do you stop/restart/patch your applications?
(and this is not an exhaustive list.
Depending on the nature of the application release, other concerns will have to be addressed)
The answer to this varies greatly depending on the exact requirements and team structures.
I've implemented processes for a few very large websites with similar availability requirements and there are some general principles I find have worked:
Externalise any config such that the same built artifact can run on all your environments. Then only build the artifacts once for each release - Rebuilding for different environments is time consuming and risky e.g. it not the same app that you tested
Centralise the place where the artifacts get built. - e.g. all wars for production must be packaged on the CI server (using the maven release plugin on hudson works well for us).
All changes for release must be traceable (version control, audit table etc.), to ensure stability and allow for quick rollbacks & diagnostics. This doesn't have to mean a heavyweight process - see the next point
Automate everything, building, testing, releasing, and rollbacks. If the process is dependable, automatable and quick the the same process can be used for everything from quick fixes to emergency changes. We use the same process for a quick 5 minute emergency fix and for a major release, because it is automated and quick.
Some additional pointers:
See my answer property-placeholder location from another property for a simple way to load different properties per environment with spring.
http://wiki.hudson-ci.org/display/HUDSON/M2+Release+Plugin If you use this plugin and ensure that only only the CI server has the correct credentials to perform maven releases, you can ensure that all releases are performed consistently.
http://decodify.blogspot.com/2010/10/how-to-build-one-click-deployment-job.html A simple way of deploying your releases. Although for large sites you will probably need something more complicated to ensure no downtime - e.g. deploying to half the cluster at a time and flip-flopping web traffic between the two halves - http://martinfowler.com/bliki/BlueGreenDeployment.html
http://continuousdelivery.com/ A good website and book with some very good patterns for releasing.
Hope this helps - good luck.
To add to my previous answer, what you are dealing with is basically a CM-RM issue:
CM (Change Management)
RM (Release Management)
In other words, after the first release (i.e. the main initial development is over), you have to keep making release, and that is what CM-RM is supposed to manage.
The implementation of the RM can be either 1) or 2) in your question, but my point would be to add to that mechanism:
proper CM in order to track any change request, and evaluate their impact before committing to any development
proper RM in order to be able to realize the "release" tests (system, performance, regression, deployment tests), and then to planify, schedule, perform and then monitor the release itself.
Without claiming it's a best solution, this is how my team currently does staging and deployment.
Developers initially develop at their local machine, the OS is free to choose, but we strongly encourage using the same JVM as will be used in production.
We have a DEV server where frequently snapshots of the code is being pushed to. This is simply a scp from the binary build produced from the IDE. We plan to build directly on the server though.
The DEV server is used for stakeholders to continuously peek along with development. By its very nature it's unstable. This is well known with all users of this server.
If the code is good enough, it's branched and pushed to a BETA server. Again, this is a scp of a binary build from the IDE.
Testing and general QA takes place on this BETA server.
Mean while, if any emergency changes should be necessary for the software currently in production, we have a third staging server called the UPDATE server.
The UPDATE server is initially only used to stage very small fixes. Here too we use scp to copy binaries.
After all testing is conducted on UPDATE, we copy the build from UPDATE to LIVE. Nothing ever goes to the live servers directly, it always goes via the update server.
When all testing is finalized on BETA, the tested build is copied from the beta server to the UPDATE server and a final round of sanity testing is performed. Since this is the exact build that was tested on the beta server, it is very unlikely that problems are found in this stage, but we uphold the rule that everything deployed to the live server should go via the update server and that everything on the update server should be tested before moving it on.
This sliding strategy allows us to develop for 3 versions in parallel. Version N that's currently in production and staged via the update server, version N+1 that will be the next major release that's about to be released and is staged on the beta server, and version N+2 that is the next-next major release for which development is currently underway and is staged on the dev server.
Some of the choices that we made:
A full application (an EAR) typically depends on artifacts from other projects. We choose to include the binaries of those other projects instead of building the whole thing from source. This simplifies building and gives greater assurance that a tested application is bundled with exactly the right versions of all its dependencies. The cost is that a fix in such a dependency has to be manually distributed to all applications that depend on it.
Configuration for every staging is embedded in the EAR. We currently use a naming convention and a script copies the right version of each configuration file to the right location. Parameterizing the path for each configuration file, e.g. by using a single {stage} placeholder in a root config file is currently being considered. The reason we store the config in the EAR, is because the developers are the ones who introduce and depend on configuration, so they should be the ones responsible for maintaining it (adding new entries, removing unused one, tweaking existing ones, etc).
We use a DevOps strategy for a deployment team. It consists of a person who is purely a developer, two persons who are both developer and operations and two persons who are purely operations.
Embedding the configuration in the EAR might be controversial, since traditionally operations needs to have control about e.g. the DB data sources being used in production (to what server it points to, how many connections a connection pool is allowed to have, etc). However, since we have persons on the development team who are also in operations, they are easily able to sanity check the changes made by other developers in the configuration while the code is still in development.
Parallel to the staging we have the continuous build server server doing a scripted (ANT) build after every check-in (with a maximum of once per 5 minutes), and runs unit tests and some other integrity tests.
It remains difficult to say whether this is a best-of-breed approach and we're constantly trying to improve our process.
I am a big advocate of a single deployable containing everything (Code, Config, DB Delta, ...) for all environments, built and released centrally on the CI server.
The main idea behind this is that Code, Config & DB Delta are tightly coupled anyway. The code is dependent on certain properties being set in the config and some objects (tables, views, ...) being present in the DB. So why split this and spend your time tracking everything to make sure it fits together, when you can just ship it together in the first place.
Another big aspect is minimizing differences between environments, to reduce failure causes to the absolute minimum.
More details in my Continuous Delivery talk on Parleys: http://parleys.com/#id=2443&st=5
I recently have a problem that my java code works perfectly ok on my local machine, however it just wouldn't work when I deploy it onto the web server, especially the DB part. The worst part is that the server is not my machine. So I had to come back and forth to check the versions of softwares, the db accounts, the settings, and so on...
I have to admit that I did not do a good job with the logging mechanism in the system. However as an newbie programmer with little experience, I had to accept my learning curves. Therefore, here comes a very general but important question:
According to your experience, where would it be most likely to go wrong when it is working perfectly on the development machine but totally surprises you on the production machine?
Thank you for sharing your experience.
The absolute number one cause of problems which occur in production but not in development is Environment.
Your production machine is, more likely than not, configured very differently from your development machine. You might be developing your Java application on a Windows PC whilst deploying to a Linux-based server, for example.
It's important to try and develop against the same applications and libraries as you'll be deploying to in production. Here's a quick checklist:
Ensure the JVM version you're using in development is the exact same one on the production machine (java -version).
Ensure the application server (e.g. Tomcat, Resin) is the same version in production as you're using in development.
Ensure the version of the database you're using is the same in production as in development.
Ensure the libraries (e.g. the database driver) installed on the production machine are the same versions as you're using in development.
Ensure the user has the correct access rights on the production server.
Of course you can't always get everything the same -- a lot of Linux servers now run in a 64-bit environment, whilst this isn't always the case (yet!) with standard developer machines. But, the rule still stands that if you can get your environments to match as closely as possible, you will minimise the chances of this sort of problem.
Ideally you would build a staging server (which can be a virtual machine, as opposed to a real server) which has exactly (or as close as possible to) the same environment as the production server.
If you can afford a staging server, the deployment process should be something like this:
Ensure application runs locally in development and ensure all unit and functional tests pass in development
Deploy to staging server. Ensure all tests pass.
Once happy, deploy to production
You're most likely running under a different user account. So the environment that you inherit as a developer will be vastly different from that a a production user (which is likely to be a very cut down environment). Your PATH/LD_LIBRARY_PATH (or Windows equivalents) will be different. Permissions will have changed etc. Plus the installed software will be different.
I would strongly recommend maintaining a test box and a test user account that is set up with the same software, permissions and environments as the production user. Otherwise you really can't guarantee anything. You really need to manage and control the production and test servers wrt. accounts/installed software etc. Your development box will always be different, but you need to be aware of the differences.
Finally a deployment sanity check is always a good idea. I usually implement a test URL that can be checked as soon as the app is deployed. It will perform database queries or whatever other key functions are required, and report unambiguously as to what's working/not working via a traffic light mechanism.
Specifically you can check all the configuration files (*.xml / *.properties) in your application and ensure that you are not hard coding any paths/variables in your app.
You should maintain different config files for each env. and verify the installation guide from env admin. (if exists)
Other than that versions of all softwares/dependency list etc as described by others.
A production machine will likely miss some of the libraries and tools you have on your development machine. Or there may be older versions of them. Under circumstances it may interfere with the normal software function.
Database connection situation may be different, meaning users and roles and access levels.
One common (albeit easy to detect) problem is conflicting libraries, especially if you're using Maven or Ivy for dependency management and don't double check all the managed dependencies at least once before deploying.
We've had numerous incompatible versions of logging frameworks and even Servlet/JSP API .jar:s a few times too many in our test deployment environment. Also it's always a good idea to check what the shared libraries folder of your tomcat/equivalent contains, we've had some database datasource class conflicts because someone had put postgre's jdbc jar to the shared folder and project came with its own jar for jdbc connectivity.
I always try to get an exact copy of the Server my product is running. After some apps and of course a lot of Bugs i vreated myself a List of common Bugs/Hints. Another Solution i tested for my last project was to get the running Software on that Server and try to configure it. Strange effects can happen with that^^
Last but not least..i always test my apps on different machines.
In my experience there is no definite answer to this question. Following are some of the issues I faced.
Automatic updates was not turned on in dev server (windows) and it was turned on in the production server(which in first place is wrong!). So one of my web application crached due to a patch applied.
Some batch jobs were running in the production app server which changed some data on which my application was using.
It is not me who does the deployment for my company so most of the time people who deploy miss some registry entries, or add wrong registry entries. Simple but very hard to detect (may be for me ;-) ) once I took hours to identify a space in one of the registry values. Now We have a very long release document which has all the details about all servers used by the application and there is a check list for "current release" which the engineers who deploy the application fill in.
Willl add more if I remeber any.
Beyond just a staging server another strategy for making sure the environments you deploy into are the same is to make sure they are set up automatically. That is you use a tool like Puppet to install all the dependencies that the server has and run your install process before every installation so that all the configuration is reset. That way you can ensure the configuration of the box is what you have set it to during the development process and have the configuration of the production environment in source control.
Is it possible to set up continuous build of projects written in .NET and Java on single build server with single set of applications?
I've seen CruiseControl has support for both world but as far as I know these are two separate applications. Should I go with a separate machine to build Java projects (a machine for .NET projects is already set up)?
Hudson has support for both Ant (out of the box IIRC) and NAnt (through plugin) so that's an option for you.
CruiseControl supports several different build options include Ant, Maven, NAnt, Phing (php), Rake, XCode, and then the generic "exec" which you can use to invoke any command-line script/tool you want. Not a problem mixing Java and .NET on the same server.
Of course this is not unique to CruiseControl. There are lots of CI tools that support multiple build technologies as you can see on this matrix of features.
Disclaimer: I work on CruiseControl. OTOH since I don't make money on it I don't much care which tool people choose. I care more about advancing the state of CI practices which is why I organize the Continuous Integration and Testing Conference (CITCON).
Ant and NAnt can reasonably easily execute arbitrary processes, including each other, so the actual build part shouldn't be too hard.
I'd expect the tricky bit to be getting the reports (including unit test results) into an appropriate format. How's your XSLT? ;)
EDIT: Now that I think about it, my first agile project had a continuous build server (just CruiseControl, I believe) which must have been doing some of this... I suspect at the time we directly invoked Visual Studio to build the code and NUnit to test it. If I were at the same company I'd check, but that was two jobs ago :(
You could checkout Atlassian Bamboo. Unfortunately, its not free, unless you are applying for an opensource/community license for use with opensource software.
You can use two different products on the same machine. Or you can run a single builds system across multiple machines. It is really up to the load you place on your CI.