We already have a good build server in Hudson but we want something that would let us startup and shutdown servers, push out new deployments of software (which is much more involved than just a single WAR or EAR going somewhere, there is copying, extracting, renaming, etc.), and various other tasks we would like to automate.
I've looked at SmartFrog (seems largely academic rather than commonly used), ControlTier (a dose of heavy complexity to go with your complexity), Capistrano (Ruby and Rails focused), and Func (no docs at all, their MediaWiki has been wiped). Is there nothing that is a good 80/20 solution for this kind of thing. Surely you could have a tool that would offer a lot of what ControlTier offers without all the overhead.
The alternative is Ant scripts to do everything (possibly available through our existing Hudson server) or even a Hudson plugin of some type but that feels kludgy to me and I'm just not liking it as a solution (plus we might be rebuilding something when we don't need to).
Please, tell me about a good alternative.
We are using Ubuntu only and use Debian packages to distribute builds across our servers and control them with init scripts. It's also great to setup a new server. Just add dependencies to Java, MySql and whatever you need and you'll be up and running a new deployment within seconds.
You might want to try Cargo. You can use ant and maven for the deployment definition you want and then set Hudson to point at your ant's build.xml or maven's pom.xml.
Chef was mentioned here but I have no experience with it.
There is also Puppet that you need to check out. You might need to read this great whitepaper on achieving fully automated provisioning
-Ken
LiveRebel might be a suitable tool for this task. It provides CLI API and also a Hudson/Jenkins plugin for automation
Related
What exactly do Gradle and Maven "do" that eclipse or sts doesn't? From what I've read it builds, runs, etc which can all be done in eclipse easily.
If I have an existing project I've created, built, and currently runs in eclipse via tomcat, what would I use gradle for?
There's not a lot of benefit to using Maven or Gradle on a small project that you never share with anyone; an IDE can do the build just as well. But as the number of developers increases and the complexity of the build increases, it becomes very useful to separate the build instructions from the IDE. Let's drill into these a little bit.
With the increase in developers, you don't want everyone to have to come by and use your IDE to get a build done. That would be really annoying! So they're on their own machines, but then they tend to have different setups (how dare they have different user account names!) and probably have their IDE installations set up a little differently too. So we need some kind of build instructions that people can use to get things going, and it helps if everyone can use the same build instructions repeatably so that you don't get too many instances of “but it works on my machine!”. It's also very helpful if those instructions are simple enough to use that a new programmer to the team can get up to speed rapidly.
But the other thing that often happens as projects grow in scope is that their builds become more complicated. They very often gain additional dependencies (they didn't start out needing a high-performance date parser and MIME-type identifier, buit they've become required since and you don't want to have to write all those from scratch) and that means you've got to make sure that when the build is done, the right version of those dependencies is used. But that's not the only way that complexity increases. It's also very often the case that you find you're using more automatically-generated code. You might find yourself working with XML schemas or WSDL a lot, or maybe your using Hibernate, or Spring, or … well, there's lots of ways in which things can get complicated, OK? Getting the various steps to do all the build right, reliably, in these sorts of scenarios can be a bit tricky, but encoding them as instructions to something like Maven makes life a lot easier once you've taken the jump in the first place. (It gets even more important when you start trying to deal with projects which need many different sub-programs that work in concert; some of those are plain hard to build even with Maven or Gradle or any other tool.)
And then there's the possibility of offloading work to a build server, running tests automatically, managing dependencies cleanly, etc. IDEs don't handle these all that well by themselves; where they do a reasonable job of it, it's usually because they're using a tool like Maven under the covers to do the heavy lifting.
tl;dr
You don't have to make your code work with a build system, but it helps if you do and in many ways.
Maven and Gradle can do many things that Eclipse doesn't. However, the most important thing they do, is to decouple the bulding and testing processes from the IDE you choose (i.e. Eclipse). When you work on a large environment, with many programmers, usually you can not control the IDE they use. So, it's better to use a tool like Maven and Gradle to standardize these tasks. The same happens with the code examples of a book: instead of the authors having to provide the instructions for configuring any IDE to execute them, they provide the Maven or Gradle files, so the reader can build and test them on any IDE he's using.
Another very important feature that Maven and Gradle give you, is the fact that dependencies are managed without the need of having the executable code under source version control. Instead of having the executable code you depend on as part of the project, you declare the dependencies on a text file (which is under source version control), and then get them from a repository.
However, you may only see the real advantages of using tools like Maven or Gradle (and even Jenkins or Hudson), when you think in large scale projects, developed along many months by teams composed of many developers).
Gradle and Maven are build tools. Maven was first and is a bit older, Gradle is newer and has redefined a way of how projects are built and maintained. In my opinion it's also much easier to use, more readable and easier to maintain. I prefer Gradle ;)
You use eclipse or STS (any other IDE) for development. And while You finish this process You need to provide a configured artifact (war, ear, whatever...) to production and deploy it there. These artifacts have well defined format and the application won't be run from eclipse or STS at the production environment. It's tiresome and error prone to prepare such artifacts by hand.
Gradle or Maven can take responsbility of building and preparing these artifacts (in fact such tools can do much more) off Your shoulders, they make this process automated.
I have a Java application (a quite large one with many external .jar dependencies as well as dependencies on images) and I need to package it up so that someone can double click to run, for example. Or something easy like that.
It uses Java Persistence, so it requires a sql connection which is specified in the Persistence.xml file in the Java Project.
How can I package this up? I was thinking:
the installation process should validate that the user has MySQL installed and if not, direct them to install it
the installation process could ask the user to enter credentials for any database and then I could update the Persistence.xml at run time
These were two ideas I had...but I wasn't sure if there was a known solution to this problem. Any help would be much appreciated!
I think you should take a look at embedded database solutions, like H2. Also, you can package your application using maven's shadowing or jar plugin, having the jar-with-dependencies profile activated.
This will nicely rid you of checking for database servers running on the client machine, and also will give you the proper means of bundling the application in one nice JAR, albeit a little large.
Maven is a build ecosystem and toolset especially designed for building Java applications and executing the code -- and generally doing whatever else you can imagine that's possible to do with and to your code.
It has a rich API for developing plugins and many developers have exploited this feature. There are numerous plugins for building -- and launching -- and packaging your application as well as helping you manage your applications dependencies.
Maven's shadowing comes in the form of maven-shade-plugin, available here. What it does is that it helps you create a single JAR file from all your dependencies. Also, there is the maven-jar-plugin which offers a profile jar-with-dependencies. It is also accessible from here.
H2, on the other hand is a full-fledged RDBMS. This is the website: http://www.h2database.com/html/main.html, and here is a tutorial.
You can find information on embedding the database here:
How to embed H2 database into jar file delivered to the client?
Embedding the Java h2 database programmatically
h2 (embedded mode ) database files problem
I would also suggest you use a combination of H2/Hibernate/Spring which is a very easy setup and provides you with really rich features and an easy-to-use API.
I hope this helps you :)
Building a sophisticated installer that checks lots of dependencies, and runs on lots of different platforms (which I assume you want) is complicated.
I suggest that you look at an installer generator; see What is the best installation tool for java?
Another alternative that I've seen in a few products is to write a (non-GUI) installer or configurer in a scripting language like Perl.
I wrote an installer using ANT, but has no GUI. Also, I used Iz Pack (good option), so I think that depends on how smart do you want it to be, if you are supposed to use it, or a non-technical person, etc.
Currently we have a Java Restlet API with dependencies controlled via Maven. When we update the API we run maven assembly:assembly which does the unit tests etc and produces a single jar file. We then upload this to the production server and run it using nohup.
Is there a better or more automated way of doing this? Is this where something like Hudson would come in?
Thanks
My experience goes with webapp-deployment. But same should hold true here. Use Maven, Cargo, Nexus (or Artifactory), Hudson and probably, Jira in conjunction of product release.
Automated release process are more reliable because there is no human factor involved that may forget a step.
We also use Liquibase for database versioning. And, if you are dealing with database changes in your application deployment. You'll realize Liquibase boosts so much confidence while running alter scripts.
I would suggest to go through the following resources
Automated Deployment with Maven - going the whole nine yards If you can, literally follow this pattern.
Maven 2 Effective Implementation -- this book really helped us a lot.
There are several Maven plugins to help deployment. The most general of them is Cargo, but there are also app server specific plugins for some concrete servers like JBoss.
Most companies I have worked for (actually, all) have had some sort of custom in-house built deployment system; even if build was done using a standard framework (like Maven in use at my current company).
Part of this is because there are many aspects that tie closely to company-specific infrastructure, capacity management and monitoring systems; and so even though there are open-source systems, there is usually something that needs to be tweaked.
It sounds like you are running your app on its own--it isn't part of any application server. If you aren't using an application server, there are probably some ways to get cargo and maven to deploy it for you, but you may be better off just using some shell scripts to deploy and run the application.
However, as your application grows, you may find a need for an application server like Jetty, JBoss, Glassfish, Tomcat, etc. When this happens, take a look at the cargo plugin for Maven because it will allow you to do something like:
mvn cargo:redeploy
That will package up your application, send it to the server and restart the app. If you want Hudson to do this for you automatically you can add it as a target to build.
Cargo can save you a lot of time when you have to frequently update an application server.
We are currently using JDeveloper to build our production EARs. One problem with this is that if the developer doesn't add new files to a VCS, then that developer is the only one capable of making EARS, and therefore he can use unversioned files as well.
What would be a good system that seperates this, so that EAR files can be correctly produced without depending on the local developers workspace (this would also ensure that they add their files to a VCS before allowing to make a deployment/check-in).
One problem with this is that if the developer doesn't add new files to a VCS, then that developer is the only one capable of making EARS,
If the developer doesn't use the VCS, this is not your only problem:
You cannot reproduce things in another environment, you're tied to the developer machine (but you're aware of that). What if he is sick?
Not versioning files means you don't have any history of modifications and that you don't know what you put into production ("Hmm, what is in this version? Wait, let's open the EAR to check that.").
And last but not least, in case of hardware failure (e.g. a hard drive crash), you can say good bye to everything that is not in the VCS.
So, the first thing to fix is to ALWAYS version files, even if there is only one developer as working alone doesn't save you from the mentioned problems. These points should be reminded (the developer needs to be aware of them to understand their importance).
To make sure this happens, you should actually not rely on the developer machine to build the EAR but rather rely on an "external" process that would be the reference. You want to avoid this syndrome:
alt text http://img716.imageshack.us/img716/9901/worksonmymachinestarbur.png
To put such a process in place, you need to automate the build (i.e. your whole build can be run in one command) and to break the dependency with your IDE. In other words, do not use the IDE to build your EAR but rather use a tool like Maven or Ant (which are IDE agnostic). That would be the second thing to fix.
Once you'll have automated your build process, you could go one step further and run it continuously: this is called Continuous Integration (CI) and allows to get frequent, ideally immediate, feedback about changes (to avoid big bang integration problems). That would be the third thing to fix.
Given your actual toolset (which is far from ideal, there is not much community support for the tools you are using), my recommendation would be to use Ant (or Maven if you have some knowledge of it) for the build and Hudson for the continuous integration (because it's extremely easy to install and to use, and it has a Dimensions plugin).
Here's my recommendation:
Get a better IDE - IntelliJ, Eclipse, or NetBeans. Nobody uses JDeveloper
Have developers check into a central version control system like Subversion or Git.
Set up a continuous integration facility using Ant and either Cruise Control or Hudson to automate your builds.
What we do is use cruisecontrol. It does two things, it lets us do continuous integration builds, so that we have nightly builds as well as lightweight builds that get built every time a change is checked it.
We also use it to more specifically address your issue. When we want to ship, we use cruisecontrol to kick off a build, that is tagged with the proper production build version. It will grab the code from our version control system (we use SVN) and will build on that, so it is not dependent on developers local environments.
One thing you might also want to consider is creating a production branch to build out of. So, production ears for a particular release are always built from that branch. This way you have even have a bit more control over what goes into the build.
Instead of doing builds from developer workspaces, setup Maven, and have something like Hudson run your Maven build. The artificats of this build (your ear) gets deployed.
Is it possible to set up continuous build of projects written in .NET and Java on single build server with single set of applications?
I've seen CruiseControl has support for both world but as far as I know these are two separate applications. Should I go with a separate machine to build Java projects (a machine for .NET projects is already set up)?
Hudson has support for both Ant (out of the box IIRC) and NAnt (through plugin) so that's an option for you.
CruiseControl supports several different build options include Ant, Maven, NAnt, Phing (php), Rake, XCode, and then the generic "exec" which you can use to invoke any command-line script/tool you want. Not a problem mixing Java and .NET on the same server.
Of course this is not unique to CruiseControl. There are lots of CI tools that support multiple build technologies as you can see on this matrix of features.
Disclaimer: I work on CruiseControl. OTOH since I don't make money on it I don't much care which tool people choose. I care more about advancing the state of CI practices which is why I organize the Continuous Integration and Testing Conference (CITCON).
Ant and NAnt can reasonably easily execute arbitrary processes, including each other, so the actual build part shouldn't be too hard.
I'd expect the tricky bit to be getting the reports (including unit test results) into an appropriate format. How's your XSLT? ;)
EDIT: Now that I think about it, my first agile project had a continuous build server (just CruiseControl, I believe) which must have been doing some of this... I suspect at the time we directly invoked Visual Studio to build the code and NUnit to test it. If I were at the same company I'd check, but that was two jobs ago :(
You could checkout Atlassian Bamboo. Unfortunately, its not free, unless you are applying for an opensource/community license for use with opensource software.
You can use two different products on the same machine. Or you can run a single builds system across multiple machines. It is really up to the load you place on your CI.