I recently have a problem that my java code works perfectly ok on my local machine, however it just wouldn't work when I deploy it onto the web server, especially the DB part. The worst part is that the server is not my machine. So I had to come back and forth to check the versions of softwares, the db accounts, the settings, and so on...
I have to admit that I did not do a good job with the logging mechanism in the system. However as an newbie programmer with little experience, I had to accept my learning curves. Therefore, here comes a very general but important question:
According to your experience, where would it be most likely to go wrong when it is working perfectly on the development machine but totally surprises you on the production machine?
Thank you for sharing your experience.
The absolute number one cause of problems which occur in production but not in development is Environment.
Your production machine is, more likely than not, configured very differently from your development machine. You might be developing your Java application on a Windows PC whilst deploying to a Linux-based server, for example.
It's important to try and develop against the same applications and libraries as you'll be deploying to in production. Here's a quick checklist:
Ensure the JVM version you're using in development is the exact same one on the production machine (java -version).
Ensure the application server (e.g. Tomcat, Resin) is the same version in production as you're using in development.
Ensure the version of the database you're using is the same in production as in development.
Ensure the libraries (e.g. the database driver) installed on the production machine are the same versions as you're using in development.
Ensure the user has the correct access rights on the production server.
Of course you can't always get everything the same -- a lot of Linux servers now run in a 64-bit environment, whilst this isn't always the case (yet!) with standard developer machines. But, the rule still stands that if you can get your environments to match as closely as possible, you will minimise the chances of this sort of problem.
Ideally you would build a staging server (which can be a virtual machine, as opposed to a real server) which has exactly (or as close as possible to) the same environment as the production server.
If you can afford a staging server, the deployment process should be something like this:
Ensure application runs locally in development and ensure all unit and functional tests pass in development
Deploy to staging server. Ensure all tests pass.
Once happy, deploy to production
You're most likely running under a different user account. So the environment that you inherit as a developer will be vastly different from that a a production user (which is likely to be a very cut down environment). Your PATH/LD_LIBRARY_PATH (or Windows equivalents) will be different. Permissions will have changed etc. Plus the installed software will be different.
I would strongly recommend maintaining a test box and a test user account that is set up with the same software, permissions and environments as the production user. Otherwise you really can't guarantee anything. You really need to manage and control the production and test servers wrt. accounts/installed software etc. Your development box will always be different, but you need to be aware of the differences.
Finally a deployment sanity check is always a good idea. I usually implement a test URL that can be checked as soon as the app is deployed. It will perform database queries or whatever other key functions are required, and report unambiguously as to what's working/not working via a traffic light mechanism.
Specifically you can check all the configuration files (*.xml / *.properties) in your application and ensure that you are not hard coding any paths/variables in your app.
You should maintain different config files for each env. and verify the installation guide from env admin. (if exists)
Other than that versions of all softwares/dependency list etc as described by others.
A production machine will likely miss some of the libraries and tools you have on your development machine. Or there may be older versions of them. Under circumstances it may interfere with the normal software function.
Database connection situation may be different, meaning users and roles and access levels.
One common (albeit easy to detect) problem is conflicting libraries, especially if you're using Maven or Ivy for dependency management and don't double check all the managed dependencies at least once before deploying.
We've had numerous incompatible versions of logging frameworks and even Servlet/JSP API .jar:s a few times too many in our test deployment environment. Also it's always a good idea to check what the shared libraries folder of your tomcat/equivalent contains, we've had some database datasource class conflicts because someone had put postgre's jdbc jar to the shared folder and project came with its own jar for jdbc connectivity.
I always try to get an exact copy of the Server my product is running. After some apps and of course a lot of Bugs i vreated myself a List of common Bugs/Hints. Another Solution i tested for my last project was to get the running Software on that Server and try to configure it. Strange effects can happen with that^^
Last but not least..i always test my apps on different machines.
In my experience there is no definite answer to this question. Following are some of the issues I faced.
Automatic updates was not turned on in dev server (windows) and it was turned on in the production server(which in first place is wrong!). So one of my web application crached due to a patch applied.
Some batch jobs were running in the production app server which changed some data on which my application was using.
It is not me who does the deployment for my company so most of the time people who deploy miss some registry entries, or add wrong registry entries. Simple but very hard to detect (may be for me ;-) ) once I took hours to identify a space in one of the registry values. Now We have a very long release document which has all the details about all servers used by the application and there is a check list for "current release" which the engineers who deploy the application fill in.
Willl add more if I remeber any.
Beyond just a staging server another strategy for making sure the environments you deploy into are the same is to make sure they are set up automatically. That is you use a tool like Puppet to install all the dependencies that the server has and run your install process before every installation so that all the configuration is reset. That way you can ensure the configuration of the box is what you have set it to during the development process and have the configuration of the production environment in source control.
Related
I understand the concept of source version control and how it applies to self-contained projects like a Windows application. But for web development, most files are stored on the web server. This has become a headache for development with many people just copying and renaming files and then pushing files over to production is another mess.
I need some kind of source version control that is relatively not too difficult to learn and must be GUI-based or have a GUI as an option. The people who will use this have little or no knowledge of the command line.
How can I integrate source version control with web server files? What software is available for such an endeavor? And is it possible to have the source version control software administer both the production and development web servers or I may only have two separate source version control installs for each web server and manually push over changes?
The web servers are Windows-based and also use Tomcat for Java/JSP.
Any help would be appreciated. Thank you.
I think you are not clear on the idea of version control. Version control is about managing your code. It is about putting your code in a remote server (may be in a central location) and accessing it using a client tool. This way a number of people can work on different part of the code and than push their work to version control server. It has nothing do with the type of the project.
The project can be a windows application, web server application or any application.
While using version control, in regular intervals or whenever needed you build your code from the version control server and deploy it to the web server which means you are deploying code that is already build (a .war for a web application).
You first deploy to your development server and later deploy the same war to the production server.
You can use SVN server for your version control server and Tortoise SVN as client.
You have to split in mind two different but interacting things - Version Control and Deploy Tools:
VCS has to do with any evolving over time items, which you want to have under control
Deploy just deliver correct object into the correct place at the correct time and convert "set of something" into Product.
Deploy isn't a problem per se (almost any job can be automated), main problem in multiDEV environment (2+) with central STAGE (less with PROD) server is question of communication between Devs and synchronizing of their operations, i.e. - workflow and management:
just imagine 2 (or more) devs, performing diferent unrelated tasks, which want to test latest own (and only own) changes on common STAGING server (because they haven't functional local environment). If 1-st deploy "some WIP" on server, he don't want to have own tests be interrupted and code poisoned by deploying third-party changes. They must to communicate and coordinate actions, it can't be dumb "copy to..." in post-commit hook
And is it possible to have the source version control software administer both the production and development web servers
Yes. But VCS does not "administer" web-servers in common sense, rather it's "communicates" or "take into account"
Problem: I have a standalone Java app (henceforth known as "the agent") that runs as a service on internal company servers. It acts as a remote agent for some central servers. As the agent gets deployed in more places, managing them is getting more complicated. Specifically: pushing updates is painful because it's a fairly manual process, and getting access to the logs and other info about the environments where the agents are running is problematic, making debugging difficult. The servers under discussion are headless and unattended, meaning that this has to be a fully automated process with no manual intervention, hence Java Web Start isn't a viable solution.
Proposed solution: Make the agent phone home (to the central servers) periodically to provide agent status and check for updates.
I'm open to other suggested solutions to the problem, but I've already got a working prototype for the "status and self-updates" idea, which is what this question is focused on.
What I came up with is actually a separate project that acts as a wrapper for the agent. The wrapper periodically calls the central server via HTTP to check for an updated version of the agent. Upon finding an update, it downloads the new version, shuts down the running agent, and starts the new one. If that seems like an odd or roundabout solution, here are a few other considerations/constraints worth noting:
When the wrapper gets a new version of the agent, there may be new JAR dependencies, meaning class path changes, meaning I probably want to spawn a separate Java process instead of fiddling with ClassLoaders and running the risk of a permanent generation memory leak, which would require manual intervention--exactly what I'm trying to get away from. This is why I ended up with a separate, "wrapper" process to manage the agent updates in my prototype.
Some servers where the agents are deployed are resource-limited, so any solution needs to be low on CPU and memory usage. That makes me want a solution that doesn't involve spinning up a new JVM and is a stroke against having a separate wrapper process.
The agent is already deployed to both Windows and RHEL servers, so the solution must be cross-platform, though I wouldn't have a problem duplicating a reasonable amount of the process in batch and bash scripts to get things rolling.
Question: As stated, I want to know how to make a self-updating Java app. More specifically, are there any frameworks/libraries out there that would help me with this? Can someone with experience in this area give me some pointers?
If your application is OSGi based, you could let OSGi handle bundle updates for you. It is similar to the wrapper approach you suggest, in that the OSGi container itself is "the wrapper" and some of it won't be updated. Here's a discussion on this
Different solution: use (and pay for) install4j. Check out the auto-update features here
No need for wrapper (save memory) or java web start (adds more restrictions on your application), simply let a thread in you application check periodically for updates (e.g. from cloud) and download updates if available, then code these two calls in you application:
launch a shell script (.sh or .cmd) to update your artifacts and launch your application after few seconds pause in the script(to avoid having two instances of your application at the same time).
Terminate your application (first instance)
The script can overwrite needed artifacts and re-launch your application.
enjoy !
Have a look at Java Web Start.
It is technology that's been part of Java since... 1.5? maybe 1.4? and allows deployment and install of standalone Java-based apps through a web browswer. It also enables you to always run the latest app.
http://www.oracle.com/technetwork/java/javase/overview-137531.html
http://en.wikipedia.org/wiki/JNLP#Java_Network_Launching_Protocol_.28JNLP.29
also see this question: What's the best way to add a self-update feature to a Java Swing application?
It appears as though Webstart is the only built in way to do this at the moment.
We have a java application hosted on JBoss with a Posgres DB, and we've traditionally been selling it as an appliance (full server with application installed). Now, we need to allow clients to be able to download and install it on their servers. What is the best way to approach this? Ideally, I'd like it to be a one packaged installation file that they can run and it checks for dependencies, deploys the war file, executes the postgres sql to setup the database and start up jboss.
JBoss and Postgres will be installed by the client prior to installation.
The simplest way is to use a bash script for Linux and possible bat/cmd files for Windows, though that is not ideal. Are there any libraries available to accomplish something like this?
install4j can be used to let users install applications. The installation package will contain everything needed (application, JBoss, postgres). Furthermore, it has ant and maven tasks, too, and you can even allow the users to do some basic configuration on-the-fly.
The latest version of JBoss is OSGi based. Have you consider to use this solution ?
If JBoss and Postgres are already preinstalled and configured by users as they wish then it would be very difficult to make a silver-bullet automatic installer that takes into account and correctly handles whatever incompatibilities it can face in real life.
Maybe a detailed install instruction would be enough. Especially for advanced users. For the others - bundle some diagnoctic scripts in case they face problem.
Also consider using liquibase to do automatic database initialization and migration on application's startup. This would greatly simplify the rest of install procedure: just check deps, make datasource and deploy app.
I'm not so experienced but i worked on some big Java EE projects (using maven2) with very distinct ways to handle the installation / delivery on the different platforms.
1) One of them was to use snapshots for development and then make a maven release, of components and main webapplications. Thus the delivery is:
war/ear files
List item
properties files
sgdb files
some others
And teams will use that files to put the new application versions in the different platforms.
I think this process is strict and permits you to always keep easily the different configurations passed in production, but it's not really flexible, the process is a bit heavy and it conducted us to sometimes do some dirty things like overriding a class of a war to patch a regression...
This is an e-commerce website with 10million unique visitors per month and a 99.89% availability.
2) Another i saw is to checkout the sources on each platform and then install the snapshot artifacts in a local repository. Then the application server will use these snapshots of the .m2 folder.
There is not a real delivery process since to put a new version in production, we just have to update the sources of the components / webapps, do some maven clean install and restart the application server.
I think it's more flexible but i see some drawbacks and this approach seems dangerous for me.
This website has a frontoffice, i don't know the numbers but it's far less than the 1st one. It also has a big backoffice available for most employees of a 130 000 people company.
I guess depending on the website, its exposition to the public and the availability required, we have to adapt the delivery strategy to the needs.
I'm not here to ask which solution is the best but wonder if you have seen different things, and which strategy you would use in which case?
Without dealing dealing web sites, I had to participate in release management process for various big (Java) projects in heterogeneous environment:
development on "PC", meaning in our case Windows -- sadly still Windows Xp for now -- (and unit testing)
continuous integration and system testing on linux (because they are cheaper to setup)
pre-production and production on Solaris (Sun Fire for instance)
The common method I saw was:
binary dependency (each project uses the binaries produced by the other project, not their sources)
no recompilation for integration testing (the jars produced on PC are directly used on linux farms)
full recompilation on pre-production (meaning the binary stored on the Maven repo), at least to make sure that everything is recompiled with the same JDK and the sale options.
no VCS (Version Control System, like SVN, Perforce, Git, Mercurial, ...) on a production system: everything is deployed from pre-prod through rsynch.
So the various parameters to take into account for a release management process is:
when you develop your project, do you depend directly on the sources or the binaries of the other projects?
where do you store your setting values?
Do you parametrize them and, if yes, when do you replace the variables by their final values (only at startup, or also during runtime?)
do you recompile everything on the final (pre-production) system?
How do you access/copy/deploy on your production system?
How do you stop/restart/patch your applications?
(and this is not an exhaustive list.
Depending on the nature of the application release, other concerns will have to be addressed)
The answer to this varies greatly depending on the exact requirements and team structures.
I've implemented processes for a few very large websites with similar availability requirements and there are some general principles I find have worked:
Externalise any config such that the same built artifact can run on all your environments. Then only build the artifacts once for each release - Rebuilding for different environments is time consuming and risky e.g. it not the same app that you tested
Centralise the place where the artifacts get built. - e.g. all wars for production must be packaged on the CI server (using the maven release plugin on hudson works well for us).
All changes for release must be traceable (version control, audit table etc.), to ensure stability and allow for quick rollbacks & diagnostics. This doesn't have to mean a heavyweight process - see the next point
Automate everything, building, testing, releasing, and rollbacks. If the process is dependable, automatable and quick the the same process can be used for everything from quick fixes to emergency changes. We use the same process for a quick 5 minute emergency fix and for a major release, because it is automated and quick.
Some additional pointers:
See my answer property-placeholder location from another property for a simple way to load different properties per environment with spring.
http://wiki.hudson-ci.org/display/HUDSON/M2+Release+Plugin If you use this plugin and ensure that only only the CI server has the correct credentials to perform maven releases, you can ensure that all releases are performed consistently.
http://decodify.blogspot.com/2010/10/how-to-build-one-click-deployment-job.html A simple way of deploying your releases. Although for large sites you will probably need something more complicated to ensure no downtime - e.g. deploying to half the cluster at a time and flip-flopping web traffic between the two halves - http://martinfowler.com/bliki/BlueGreenDeployment.html
http://continuousdelivery.com/ A good website and book with some very good patterns for releasing.
Hope this helps - good luck.
To add to my previous answer, what you are dealing with is basically a CM-RM issue:
CM (Change Management)
RM (Release Management)
In other words, after the first release (i.e. the main initial development is over), you have to keep making release, and that is what CM-RM is supposed to manage.
The implementation of the RM can be either 1) or 2) in your question, but my point would be to add to that mechanism:
proper CM in order to track any change request, and evaluate their impact before committing to any development
proper RM in order to be able to realize the "release" tests (system, performance, regression, deployment tests), and then to planify, schedule, perform and then monitor the release itself.
Without claiming it's a best solution, this is how my team currently does staging and deployment.
Developers initially develop at their local machine, the OS is free to choose, but we strongly encourage using the same JVM as will be used in production.
We have a DEV server where frequently snapshots of the code is being pushed to. This is simply a scp from the binary build produced from the IDE. We plan to build directly on the server though.
The DEV server is used for stakeholders to continuously peek along with development. By its very nature it's unstable. This is well known with all users of this server.
If the code is good enough, it's branched and pushed to a BETA server. Again, this is a scp of a binary build from the IDE.
Testing and general QA takes place on this BETA server.
Mean while, if any emergency changes should be necessary for the software currently in production, we have a third staging server called the UPDATE server.
The UPDATE server is initially only used to stage very small fixes. Here too we use scp to copy binaries.
After all testing is conducted on UPDATE, we copy the build from UPDATE to LIVE. Nothing ever goes to the live servers directly, it always goes via the update server.
When all testing is finalized on BETA, the tested build is copied from the beta server to the UPDATE server and a final round of sanity testing is performed. Since this is the exact build that was tested on the beta server, it is very unlikely that problems are found in this stage, but we uphold the rule that everything deployed to the live server should go via the update server and that everything on the update server should be tested before moving it on.
This sliding strategy allows us to develop for 3 versions in parallel. Version N that's currently in production and staged via the update server, version N+1 that will be the next major release that's about to be released and is staged on the beta server, and version N+2 that is the next-next major release for which development is currently underway and is staged on the dev server.
Some of the choices that we made:
A full application (an EAR) typically depends on artifacts from other projects. We choose to include the binaries of those other projects instead of building the whole thing from source. This simplifies building and gives greater assurance that a tested application is bundled with exactly the right versions of all its dependencies. The cost is that a fix in such a dependency has to be manually distributed to all applications that depend on it.
Configuration for every staging is embedded in the EAR. We currently use a naming convention and a script copies the right version of each configuration file to the right location. Parameterizing the path for each configuration file, e.g. by using a single {stage} placeholder in a root config file is currently being considered. The reason we store the config in the EAR, is because the developers are the ones who introduce and depend on configuration, so they should be the ones responsible for maintaining it (adding new entries, removing unused one, tweaking existing ones, etc).
We use a DevOps strategy for a deployment team. It consists of a person who is purely a developer, two persons who are both developer and operations and two persons who are purely operations.
Embedding the configuration in the EAR might be controversial, since traditionally operations needs to have control about e.g. the DB data sources being used in production (to what server it points to, how many connections a connection pool is allowed to have, etc). However, since we have persons on the development team who are also in operations, they are easily able to sanity check the changes made by other developers in the configuration while the code is still in development.
Parallel to the staging we have the continuous build server server doing a scripted (ANT) build after every check-in (with a maximum of once per 5 minutes), and runs unit tests and some other integrity tests.
It remains difficult to say whether this is a best-of-breed approach and we're constantly trying to improve our process.
I am a big advocate of a single deployable containing everything (Code, Config, DB Delta, ...) for all environments, built and released centrally on the CI server.
The main idea behind this is that Code, Config & DB Delta are tightly coupled anyway. The code is dependent on certain properties being set in the config and some objects (tables, views, ...) being present in the DB. So why split this and spend your time tracking everything to make sure it fits together, when you can just ship it together in the first place.
Another big aspect is minimizing differences between environments, to reduce failure causes to the absolute minimum.
More details in my Continuous Delivery talk on Parleys: http://parleys.com/#id=2443&st=5
Just wondering if anyone has ever converted a Websphere project to run under tc server? I run on a Mac and would love to be able to run my application locally without having to run Websphere in a VM.
I realize there are differences which would have to be accounted for, and that's really my question - what would I have to change? I also realize that even if this is possible, one couldn't depend on the tc server configuration before going to production - it would have to be tested in Websphere first.
As we know, application server provides lot more features than a tomcat. Hence first, you would have to check if your application is using any of those features. If so, then you would have to see if that missing functionality could be plugged-in somehow. To exemplify, you could be using the transactional capability of the application server. Tomcat doesn't come with transaction capability. You would need to plug-in a a third party component for that. For this to happen less intrusively, the code should be configurable so that based on the environment it would know whether to make use of the capabilities of the application server or use the capabilities of the plugged-in components in case of tomcat.
Advantage of this: It would certainly be boosting to the productivity if you use tomcat to develop rather than any application server, as the time it takes to start and stop tomcat if far less than the coffee time (To me ,one coffee time is the amount it takes for an application server to stop and start even when you have the minimal hardware requirements mentioned by it)
Warning:: Again this approach has a negative point that you would miss to see how your component interacts with the classes in the production environment, while you develop. Hence it becomes mandatory to have your QA environment and above to make use of the application server. This way you could avoid any surprises in production.