We've used Hudson for quite a while to implement a CI server with all the bells and whistles. The setup is quite straight forward, when installing from the provided RPMs and Debs, but through googling I haven't been able to figure out whether the plugins are installable using apt/rpm or some other package manager?
The reason is that I would like to create a (meta)package for Ubuntu which would install and also update both Hudson and all the plugins through the normal upgrade mechanism. At the same time I could create a template setup for other projects, say JavaEE project needs git, cobertura and Chuck Norris plugins, while my Python project needs plugins XXX and YYY.
Anybody got such a setup? As a workaround I figured setting up a number of Maven POMs, which would do the init, and later upgrades, but I feel this would require more scripting on the side, which I'm not very eager to do.
Any other suggestions for this would also be appreciated.
I haven't been able to figure out whether the plugins are installable using apt/rpm or some other package manager?
They are installed "manually".
The reason is that I would like to create a (meta)package for Ubuntu which would install and also update both Hudson and all the plugins through the normal upgrade mechanism. (...)
I'm not aware of an existing solution for this (very likely because packaging plugins is a big mistake in general: it's a lot of work for the linux distros and plugins go much faster than the release cycles). So I guess you'll have to package them yourself.
And be careful with upgrades of Hudson, they often break plugins (in other words, I recommend to test the upgrade of Hudson in a test environment, like a virtual machine).
Anybody got such a setup? (...)
I bundle plugins in an archive and unzip them in <HUDSON_HOME>/plugins/.
Related
I have an Eclipse project (which is generated via a maven archetype and requires m2e) that I want to deploy to many users. Currently the way I'm telling them to do it is extremely manual, (i.e. download this plugin, download that plugin, download this m2e connector). Is there a way to write down plugin dependencies somewhere in my project so that when the project is imported Eclipse will tell the user "This project depends on you having these Eclipse plugins installed, would you like Eclipse to auto-fetch the plugins for you?"
Please read about deploying eclipse plugins first, all of that is available in the basic infrastructure of eclipse.
Whenever you install something via the update manager in eclipse, you don't install plugins as such, but instead you install features, which are groups of plugins plus dependencies to other features. So you have to do 2 things when creating a feature for your plugin:
List all dependent features in the feature editor.
On the tab "Sites to visit" you have to list the update site URLs of the dependent features.
P2 update manager will auto-magically discover everything needed during the installation process of your plugin afterwards.
Yes there are several tools created by Eclipse member companies (both with free to use and for-pay editions) that will do exactly what you want. Both products have the notion of "profiles" that is a pre-set configuration of Eclipse plugins (basically one of the available packages plus a bunch of plugins). Then once you have this profile configured you can freely share it with our team via email or whatever sharing mechanism the products provide. Once your team members accept and start "using" the profile those software products will provision exactly what dependencies everyone needs to run your Eclipse projects.
Pulse from Genuitec
They have a Community edition and a Managed Team edition based on your team's needs.
Yoxos from EclipseSource
They have a free plan if you want to share your profiles publicly. If you want to keep them private they have a pricing plan for that.
Not sure how much help this is, but why not use the Eclipse Marketplace if it is available. Saves you the effort to manually download and install the plugin.
If I were using python I would probably like to use pip as a nice installer for continuous delivery with its nice repository integration and scripting capabilities.
Do I have anything similar in java which would be useful for me in continuous deployment?
Can someone recommend me how they do full continuous deployment in java?
I'm going to have multiple servers with complex configurations and huge multiple clusters with databases, NOSQL's (and using maven for the some of the projects while others are just downloaded pacakges) etc etc... anyone has recommendation for that?
Again I think pip is a very nice installer and could help me, anyone has experience maybe with ubuntu juju?
However if I use ubuntu juju that would mean I would have to use ubuntu based servers and not centos.
There's a kind of bright line between Java app build and Java app deployment. Build CI in Java is pretty straightforward with a variety of tools available - build scripting (Ant, Maven, Gradle, etc), continuous builds (Jenkins, Go, Anthill, etc), and repositories (Nexus, Artifactory, etc). Dependency management for libraries is a hairball for Java, so definitely use Maven or Ivy for it.
Deployment is a much wilder and less mature world. The environments are potentially far more complex, and often include messy non-Java things like relational databases. You can hand-roll scripts, or use ControlTier or Capistrano or something like that (which will still involve some hand-rolling).
I'm not completely clear what pip does, but here is my toolchain for CI/CD
You need a build tool:
Maven (does a lot of stuff, including downloading dependencies and driving you crazy)
ANT (will poke you until you die with xml brackets)
Gradle and others (pretty much everybody including ANT uses/can use Ivy for downloading dependencies from repositories)
You need an CI server
jenkins
various commercial options (Teamcity, Bamboo ...)
For the deploying part you need something to deploy your apps.
This really depends on the build tool you use (which should be able to do the deployment). Maven has some plugins for this afaik, but I think you will have to google for your app server and the build tool to find a solution for your specific need.
Probably what you are looking for is building a deployment pipeline. Check a video example here: http://www.youtube.com/watch?v=6CEQOuHM86Y
There are multiple ways to achieve it. Ill tell you my preferred one.
Components you will need:
VCS server (SVN, Git)
CI Server (Jenkins, Hudson, TeamCity)
Build Tool (Maven, Ant, Gradle)
Artifact Repository (Artifactory, Nexus)
Deployment Tool (Rundeck, Puppet, Deployinator, Capistrano)
Target Environment/s (Application Server like Tomcat, JBoss)
Workflow:
1) CI Server polls VCS Server for changes
2) When a change is found (i.e., a commit), starts job execution, getting an artifact (CI Server will compile and run tests). CI Server internally will use a Build tool like Maven.
3) CI Server uploads artifact to an Artifact Repository
4) Deployment Tool reads Artifact Repository and provides a list of artifacts to be deployed, plus the list of Target Environments where Developer/Ops can select a combination of both and deploy the artifact in the selected server.
Take into consideration some criteria at the moment of picking the tools. If you have a really big infrastructure (like 200+ Target Environments), robust solutions like Puppet make sense. If you have a modest one (let say, 10 Target Environments) then Rundeck may be enough. Also, take into consideration that some of the listed tools are not free (Puppet Enterprise is not free beyond ten nodes, for example).
I'm looking for a complete java development infrastructure with an integration of:
an IDE like eclipse,
a build system like Maven,
a version control system like subversion,
a continuous integration server like Hudson,
a repository manager like Nexus,
an automated release plugin like Maven Release Plugin.
Further I'd like to have:
a predefined multi component project structure
...and optionally:
an issue manager like Jira,
an integration with an open source hoster like sourceforge.
Evaluating all these systems could take a long time, making the setup of a running infrastructure a job of a month or longer.
At work I'm ready to setup each system individually, but for my private development at home I'd like to have something like Devware - A development environment virtual appliance (unfortunately I didn't find a download link) where everything is already installed and functional.
So, could you please give me some advice which combinations create a working infrastructure? Or even better where to find a preconfigured development infrastructure?
PS: I'm not committed to any of the named products, so feel free to suggest alternatives if they match better.
If you're looking to avoid some integration pains, check out SecureCI, which is a free download of various open-source products in an integrated stack.
Blurb from the site:
SecureCI™ provides organizations with an integrated suite of open source tools that assist in the automation of the software build, test, and deploy process. By integrating secure code analysis and web security testing technologies, SecureCI extends continuous integration to incorporate automated security analysis and security testing of applications as well.
The current version bundles these tools:
Project Management
Trac (0.11.5)
Sonar (1.10.1)
Security Scanning
ratproxy (1.58)
Development
Subversion (1.4.6)
Hudson (1.322)
Apache Ant (1.7.1)
Apache Maven (2.2.1)
Sonatype Nexus (1.3.6)
Testing
JUnit (4.6)
Cobertura (1.9.2)
Selenium Remote Control (1.0.1)
Code Analysis
Checkstyle (5.0)
FindBugs (1.3.8)
PMD (4.2.5)
JDepend (2.9)
As far as IDE integrations are concerned, later versions of Eclipse now come integrated with Mylyn, this provides a task focused API for connecting to various tools.
The Trac integration can be installed with the Connector Install Wizard.
Other Eclipse integrations to use:
Subversion integration: Subclipse and the Subclipse Mylyn integration)
Hudson plugin
Maven integration: M2Eclipse (also has some Mylyn connectors)
Code Coverage: ECLEmma (I find the Cobertura plugin doesn't work too well myself)
Selenium plugin
Checkstyle plugin
Findbugs plugin
PMD plugin
JDepend plugin or CAP
Or just use IntelliJ from JetBrains.
You might want to look into http://www.poweredbypulse.com. It provides easy to install pre-configured Eclipse instances. You can also build/configure your own.
It has been a while since I actually used this so it may have changed since then.
We have several products which have a lot of shared code and which must be maintained several versions back.
To handle this we use a lot of Eclipse projects, some contain library jars, and some contain shared source code (in several projects to avoid getting a giant heap with numerous dependencies while being able to compile everything from scratch to ensure that source and binaries are consistent). We manage those with projectSet.psf's as these can directly pull all projects out from CVS and leave a fully prepared workspace. We do not do ant builds directly or use maven.
We now want to be able to put all these projects and their various versions in a Continous Integration tool - I like Hudson but this is just a matter of taste - which essentially means that we need to get an automatic way to check out the projects to a fresh workspace, and compile the source folders as described in the project-files in each project. Hudson does not provide such an approach to build a project, so I have been considering what the best way to approach this would be.
Ideas have been
Find or write an ant plugin/converter that understands projectSet.psf's and map to cvs-checkout and compile tags.
Create the build.xml files from within Eclipse and use those. I tried this, and found the result to be verbose and with absolute locations which is not good with automatic tools putting files where they want to.
Write a Hudson plugin which understands projectSet.psf's to derive a configuration and build it.
Just bite the bullet and manually create and update the CI configuration whenever stuff breaks - I don't like this :)
I'd really like to hear about other peoples experiences so I can decide how to approach this.
Edit: Another option might be using a CI which knows better about Eclipse projects and/or project sets. We are not religious - this is just a matter of getting stuff running without having to do everything ourselves. Would Cruise Control be a better option perhaps? Others?
Edit: Found that ant4eclipse has a "Team Project Set" facility. http://ant4eclipse.sourceforge.net/
Edit: Used the ant4eclipse and ant-contrib ant extensions to build a complete workspace as a sjgned runnable jar file similar to the Runnable Jar facility in Eclipse 3.5M6. I am still depending on Eclipse to create the initial empty workspace, and extract the ProjectSet, so that is the next hurdle.
Edit: Ended up with a dual configuration, namely that Hudson extracts the same set of modules as listed in the ProjectSet.pdf file from CVS (which needs to have the same tag) causing them to be located next to each other. Then ant4eclipse works well with the projectSet.psf file embedded in the main module. Caveat: Module list in Hudson must be manually updated, and it appears that a manual workspace cleanup is needed afterwards to let Hudson "discover" that there is more projects now than earlier. This has now worked well for us for a couple of months, but it was quite tedious to get everything working inside the ant file.
Edit: The "Use Team Projects" with ant4eclipse and a Ctrl-A, Ctrl-C in Project Panel with a Ctrl-V in the CVS projects in Hudson has turned out to work well enough for us to live with (for mature projects this is very rarely changed). I am awaiting the release of ant4eclipse 1.0 - http://www.ant4eclipse.org/, currently in milestone 2 - to see how much homegrown functionality can be replaced with ant4eclipse things.
Edit: ant4eclipse is as of 20100609 in M4 so the schedule at http://www.ant4eclipse.org/node?page=1 is slipping somewhat.
Edit: My conclusion after using our ant4eclipse approach for a longer period is that the build script get very gnarly and is hard to maintain. Also the Team ProjectSet facility (which ant4eclipse use to locate the projects) which works well for CVS based repositories, but not after we migrated to git (which is a big thing in itself). New projects will most likely be based on maven, as this has good support in Jenkins.
I'm not completely sure I understand the problem, but it sounds like the root issue is that you have many projects, some of which are dependent on others. Some of the projects that are closer to the "leaf" of the dependency tree need to be able to use "stable" (or previously "released") versions of the more "core" projects.
I solve exactly this problem using Hudson, ant, and ivy. I follow a pattern demonstrated by Clark in Pragmatic Project Automation (he doesn't demonstrate the dependency problems and solutions, and he uses CruiseControl rather than hudson.)
I have a hand-written ant build file (we call it "cc-build.xml", because of our CruiseControl roots.) This file is responsible for refreshing the working space for the project from the CM repository and labeling the contents for future reference. It then hands off control to another hand-written ant build file (build.xml) that is provided by each project's developers. This project is responsible for the traditional build steps (compile, packaging, etc.) It is required to spit out the installable artifacts, unit test reports, etc, to the Hudson artifacts directory. It is my experience that automatically generated build files (by Eclipse or other similar IDE's) will never get close to getting this sufficiently robust for use in a CI scenario.
Additionally, it uses ivy to resolve its own dependencies. Ivy supports precisely-specified dependency versions (e.g. "use version 1.1") and it supports "fuzzy versions" (e.g. "use version 1.1+" or "use the latest version in integration status.") Our projects typically start out specifying a very "fuzzy" version for internal projects under ongoing development, and as they get close to a release point, they "freeze" the dependency version so that stuff stops moving underneath them.
The non-leaf projects (projects that are dependents for other projects) also use ivy to publish their artifacts to our internal ivy repository. That repository keeps all past builds of the dependents, so that any project can always depend on any other previous version.
Lastly, each project in Hudson is configured to have a build trigger that causes a rebuild when any of its dependent projects successfully build. This causes them to get built again with the (possibly) new ivy dependent version.
It is worth noting that once you get this up and running, consistent automated "labeling" or "tagging" of an automated build's inputs is going to be critical for you - otherwise troubleshooting post-build problems is going to result in having to untangle a hornet's nest to find the original source.
Getting all of this setup for our environment took quite a bit of effort (primarily in setting up the ivy repository and ant build files,) but it has paid for itself many times over in saved headaches in manually managing the dependencies and decreased troubleshooting effort.
Write a Hudson plugin which
understands projectSet.psf's to derive
a configuration and build it.
That seems like the winning answer to me.
I work with CruiseControl rather than Hudson but in my experience if you can create a plugin that solves your problem it will quickly payoff. And it is generally pretty easy to write a plugin that is custom fit for your solution as opposed to one that needs to work for everyone in a similar situation.
I have tried both Cruise Control (CC) and Hudson for our CI solution. We (as a company) decided on Hudson. But for your question "Does CC support Eclipse project build" the answer is no as far as I know. CC supports many more different build tools and Source Control systems but it is a bit more difficult to configure and use. As for Hudson, it is more simple to configure and use it. We developed our custom plugins for both CC and Hudson for the parts of our build cycle that they do not provide as is. As for plugin development, if you know / use Maven, Hudson is simpler too. But if you are not familiar to Maven, first you need to learn the basic usage of maven to successfully develop a Hudson plugin. But once you understand the basic usage of maven, plugin development, test and even debug is simpler in Hudson.
For your specific problem, I can think of a solution that makes use of Eclipse plugins as well. You can develop your own Eclipse plugin that for instance gets the psf files from a (configurable) folder, and use Eclipse internals to process these psf's. I mean you can use existing Eclipse source codes that takes a psf file, check-outs it's project definitions and compile these projects. This Eclipse plugin of yours may have a preference page (which you can access by Eclipse -> Window -> Preferences) and configure which folder it will use to look for psf files. Your Eclipse plugin should also have a way to start psf processing without user interaction. For this, you can use ipc to trigger your process. I mean your Eclipse plugin can listen for a port, and you can write another java application that will connect to your plugin through this port and trigger its process. As for CI part, you can use either CC or Hudson and use their external process execution support. If you are using Windows, you can write a bat file (for Linux sh file) that first launchs Eclipse that has your plugin installed. Then it launches your java application that will communicate with your Eclipse plugin to trigger your process. From your CI tool you will need to run your bat / sh file to trigger your process.
I have to organize a development environment where I can run Maven projects with JBoss Seam, IDE eclipse 3.4.x and deploying to JBoss 5.
The projects that will run on this environment are based in Java 6, EJB3 and JSF1.2. The environment has to support hot-deploy.
I used to work in a development environment with Sysdeo Plugin to make Tomcat run all my applications - I've rarely used EJB (only for MDB's).
So I would prefer an environment similar to this.
I'd like to know what you guys use for the kind of architecture (what kind of eclipse plugins - if they work fine, things like that)
The thing I really didn't get right is why my Maven2 project with SEAM as a dependency packaged as EAR doesn't appear in my server (in Eclipse Ganymede - tab servers) for me to make deploy (right click - option Add and Remove Projects...).
Do I have to include an specific project nature to make my Maven2 EAR project visible to my JBoss AS included in my Eclipse Ganymede?
Seam doesn't appear to go well with Maven2. I'm facing some problems to make they work together - some dependencies appear to be missing and I have to put some extra files in some special places like seam.properties and components.xml with some special contents.
I feel like forced to use seam-gen and Ant. Too bad!
Not sure if this is helpful to you, but but we run the following
eclipse as IDE
mercurial for source code management
merclipse mercurial eclipse plugin http://goldenhammers.com/merclipse/
maven for builds (and m2eclipse)
mylyn with bugzilla for issue tracking
tomcat as application server
hudson for continuous integration http://hudson-ci.org/
reviewboard for code reviews http://www.review-board.org/
sonar for code quality metrics http://sonar.codehaus.org/
proxmox VE for virtualization http://pve.proxmox.com/wiki/Main_Page
Most things run on separate virtual machines to keep interference to a minimum. Proxmox VE is a breeze to setup (15 mins and you are running).
Hudson monitors the repository and automatically builds and tests each push.
If the war build is successful it is automatically (re)deployed (using a hudson plugin) into Tomcat and restarted.
I cannot recommend these tools enough.
HTH
I am currently working on the same environment you asked for, with the only difference I am running the app on a tomcat 6.0.18. I prefer to use tomcat 'cause it's so faster to run, and I don't use EJB for now.
Eclipse plugins :
maven : m2eclipse.codehaus.org
jboss tools : www.jboss.org/tools
web tools platform for hot deploying : www.eclipse.org/webtools/
I took the Eclipse Java EE version, I don't use seam-gen to create the basic architecture.
I don't have so many problems with this environment, sometimes the hot-deploy doesn't work and I have to manually clean files.
The only problem I had was with the separation of my app in two modules : eclipse wasn't doing the job well (not taking the last package of one module while building the other one), and I discover the option "disable workspace resolution", which works fine now.
Works fine. Hope it will for you.