We are using Hudson for our Continuous Integration server and it's great. We have 2 issues with it, which are mildly related.
https://hudson.dev.java.net/issues/show_bug.cgi?id=2736 The build order in Hudson means that the downstream dependencies get built a lot more than they need to be. Hopefully this issue will be addressed soon.
Since these things are getting built so frequently, the build history is massive. We really don't need 1000 build items in the history for some of the jobs.
My question is about point 2. I would like something like a job or plugin to delete old artifacts. Keeping say the last 20 builds of everything around would be fine. At the moment it seems unbounded, which isn't great from an operations perspective.
UPDATE: As per Norbert's answer, it is in the job configuration. In the 1.300 UI, there is a "Discard Old Builds" checkbox, which allows this to be configured.
There is such configuration in our hudson build server. In the project config I can chose between an amount of builds or a period of time to keep. I don't think I installed aplugin for this
Related
I have several java projects with small differences between them for each client, those differences are:
differences in jrxml reports
differences in properties files
Changes in static classes
When I go to update the clients, I committed to SVN and generating the WAR for each project (approx 90mb) using Jenkins, upload to FTP and install it on each server.
The problem I'm having is the time it takes me to do it this way, which is between 3 and 4 hours for 6 projects every week.
It is possible that all clients are handled by a single project (single WAR) and separate the differences outside the war?
What would be the best way to do this?
This is recommended or is there a better way to handle this?
It is possible that all clients are handled by a single project (single WAR) and separate the differences outside the war?
Sure but... bear with me for a second.
With 4 hours of building/deployment time i'd say the bandwith between the jenkins server and you production servers could be the issue. If upgrading it is not viable you will need to optimize your projects setup.
Usually 90% of the size of a war file is composed by the collection of libraries your application relies on to do his job (spring, hibernate, struts and so on...).
Supposing you currently store them in your WEB-INF/lib folder, you could consider to extract those and install them in your application server shared classpath, removing them from your war.
It will not shock me if after this operation you will not need further optimizations...
I found that when I need to save time, this command saves me every time:
rsync -razpv --delete /folder/name server_name:/folder/name
you can also add:
--exclude 'file/inside/folder'
Use it cleanly, and wisely, but a process of copying files for some of my builds that took 20 minutes always, now take 10 seconds when there is no change, and less than 1 minute when there is a change.
Also - did you think about a management tool (puppet/chef/ansible) to do some of the work?
I am working under a Java project together with one of my collegues. Project sources are hosted in private GitHub repository. We are both using IntelliJ Idea 13 to work with it - commit changes and update project from Git repository. The whole IntelliJ Idea project is also in GIT repository.
The issue is that sometimes (approximately after a few commits/updates, can't say for sure since that happens randomly) Idea decides to remove ALL of my local changes and just take whatever comes with an update from Git. Under ALL I mean ALL - all new files, all changes, all resources - whatever was changed locally and not yet committed. Usually update goes well and even if something should be merged - Idea offers a merge dialog where issues can be resolved, so this is not about merging changes for sure.
It is also not an OS-related issue since we've been working on different platforms (at least Win XP/7/8, Ubuntu and Mac OS X) - that issue happens everywhere.
I have been looking for the solution of this issue for some time now and didn't find even a single thread about it, so I have decided to ask here - probably someone might give me some tips. Probably I don't know something about GIT since I didn't actually work with it a lot.
Since this case it pretty vague I am not sure which information I should add to the topic to clarify the situation.
Thanks a lot in advance!
P.S.
A wild guess - IntelliJ Idea has a workspace.xml file in its project files which represents user-related settings of work area plus some other stuff - it is also in GIT in our project and ocasionally gets committed with other changes, might that cause this issue?
I have such a problem:
When customer asked for a new feature of our system,we patched all the changed files into a patch and send it to a workmate who work with customer and do the release.
However,not every patch released in order.So,there is a chance patch A is depending patch B but patch B is releasing in front of patch A.
Due to the workmate is not familiar with programming so he can't figure out the reason.I have to spend time to see what's wrong.
When the number of patch waiting for release is growing up,releasing a patch seem like a nightmare for us.
Is there a tool for such dependency analysis?so we can see the dependency of patch and can reduce the time spending for figure out the dependency.
Thx a lot.
You can do it using any dependency management system like Maven, internal artifact repository for production-ready components (like Nexus) and release branches for hot-fixes (if you have to ship updates in a near realtime way).
Using this approach you get:
Testing of any complete release (including integration tests) which you ship to a customer.
Dependency-broken versions are not to build, because you can switch to production-ready repository for pre-ship build.
You can mark production packages with SCM tags and know what exactly is pushed to a customer.
You can simply make a diff between shipped and current package.
In few words:
Divide development and production releases and you protect yourself to build a dependency broken production package.
the correct way to handle these situation is using a Configuration Management procedure.
a simple one, for example, involves a CVS/SVN and a changelog between revision A and revision B.
every file in the changelog will compose the patch.
a more complex procedure will introduce baselines and intermediate releases.
To clarify the question :
I am looking for established best-practices or a pro/con analysis of known practices
by project lifecycle I mean : deploy to pre-integration, integration, QA, preprod and prod environment.
For some context:
Our project deploys to integration and QA every week, currenlty we create a new release for each integration deployment, but this doesn't feel right. It leads to updating all the poms every week breaking dev level dependencies, forcing every dev to do a refresh of their eclipse configurations. We have large workspaces and eclipse doesn't handle the refreshes so well thus a lot of wasted time.
I am not overly familiar with the maven release conventions and have been unable to find the ones regarding the point of the application lifecycle when mvn release should be used.
If the pattern we use now is accepted/correct/established I will have another question :)
The approach I use to avoid the Eclipse dev level dependency update issue is to leave the relevant trunk or branch version number unchanged until such time as the release becomes significant. This way you can have properly tagged/versioned releases to QA etc so that you can track issues back but not require devs to update dependencies. To achieve this I use the following command but override the version numbers to get the desired release number but re-enter the current snapshot version as the new snapshot version:
mvn release:prepare -DautoVersionSubmodules=true
P.S. I have a diagram that demonstrates this but unfortunately insufficient rights in this forum to attach it. I would happily provide it if someone can facilitate attaching.
P.P.S Maybe now...
Note also the support for early branching (2.1) and late branching (2.2).
In our shop, all of our POMs in SVN have <version>9999-SNAPSHOT</version> (for their own version as well as internal dependencies). This never changes.
During the build, we have a simple ant build.xml that takes the version number (established outside of maven) as a -Dversion=... parameter, and simply does:
<replace includes="**/pom.xml" token="9999-SNAPSHOT" value="${version}"/>
<artifact:mvn ... />
That change is local to the build process's working copy -- it's never checked in to version control.
This way all release builds have a "real" version number, but dev effectively never has to deal with version numbers.
The above is, as you say in your question, emphatically not The Right Way to do this, but it has worked well for us for the ~9 mos since we adopted maven. We have tens of maven modules, all of which move in lock-step through the QA/release process.
One implication of this approach is that you'll need separate eclipse workspaces for each branch you're working on, as otherwise the copies of a project from dif't branches will collide.
[Not really an answer, but the best I have...]
Related to MNG-624.
Depending on how many projects you have, even the burden on your source-control system may be an issue.
Does anyone use an independent numbering scheme with Maven snapshots to avoid version-number churning? In theory, you could do what you'd do without Maven - use an internal numbering system of some kind for the weekly builds. The builds would be deployed to different repositories as dictated by workflow; you'll need separate repositories for dev, QA, maybe one in between for integration test. When you're down to release candidates, start using non-snapshot releases. I'm just evaluating Maven, though - I've no experience with doing this.
Some of the Nexus documentation (for the Professional version) talks about how to do build staging, which may be relevant.
In the past I used a numbering scheme of my own devising: http://wiki.secondlife.com/wiki/Codeticket_Service
I'm now in the situation where I need to think about maven again, and I'm tempted to re-use the codeticket scheme to generate version numbers/build numbers and apply them via the release plugin but without checking the pom files back in. The checked in pom files will keep the SNAPSHOT version numbers.
For those who care about reproducible builds, you can include the modified POM file in your build result. Personally, I care more about tracking the build artifacts and ensuring that the exact same bits that have been tested end up getting released, so my concern about reproducing builds is slightly less religious than with most (See here).
There is a discussion going on in the maven users list (in which I'm participating) that seems relevant. Basically we're discussing how to avoid all that POM editing that has to be done whenever you cut a (release or feature) branch. The release plugin can do the editing for you, when you create a release branch, but it does not help with feature branches that need to be reintegrated later. Also, all that POM editing causes unecessary pain when you do merges, either rebase merges from trunk or reintegration merges to trunk.
The idea being discussed there is based on the notion that the proper location to record artifact version numbers is in the SCM tool and not in the POM. Basically, maven should be able to derive the artifact version number from the actual SCM tag or branch that the working area is associated to.
Note that there is not a complete solution yet due to some issues still pending on Maven's issue tracker (e.g. MNG-2971). But they are issues with many votes already and I'm optimist they will be fixed soon.
I work at a software company where our primary development language is Java. Naturally, we use Hudson for continuous builds, which it works brilliantly for. However, Hudson is not so good at some of the other things we ask it to do. We also use Hudson jobs to deploy binaries, refresh databases, run load testing, run regressions, etc. We really run into trouble when there are build dependencies (i.e. load testings requires DB refresh).
Here's the one thing that Hudson doesn't do that we really need:
Build dependency: It supports build dependencies for Ant builds, but not for Hudson jobs. We're using the URL invocation feature to cause a Hudson job to invoke another Hudson job. The problem is that Hudson always returns a 200 and does not block until the job is done. This means that the calling job doesn't know a) if the build failed and b) if it didn't fail, how long it took.
It would be nice to not have to use shell scripting to specify the behavior of a build, but that's not totally necessary.
Any direction would be nice. Perhaps we're not using Hudson the right way (i.e. should all builds be Ant builds?) or perhaps we need another product for our one-click deployment, load testing, migration, DB refresh, etc.
Edit:
To clarify, we have parameters in our builds that can cause different dependencies depending on the parameters. I.e. sometimes we want load testing with a DB refresh, sometimes without a DB refresh. Unfortunately, creating a Hudson job for each combination of parameters (as the Join plugin requires) won't work because sometimes the different combinations could lead to dozens of jobs.
I don't think I understand your "build dependency" requirements. Any Hudson job can be configured to trigger another (downstream) job, or be triggered by another (upstream) job.
The Downstream-Ext plugin and Join plugin allow for more complex definition of build dependencies.
There is a CLI for Hudson which allows you to issue commands to a Hudson instance. Use "help" to get precise details. I believe there is one which allows you to invoke a build and await its finish.
http://wiki.hudson-ci.org/display/HUDSON/Hudson+CLI
Do you need an extra job for your 'dependencies'?
Your dependencies sound for me like an extra build step. The script that refreshes the DB can be stored in your scm and every build that needs this step will check it out. You can invoke that script if your parameter "db refresh" is true. This can be done with more than just one of your modules. What is the advantage? Your script logic is in your scm (It's always good to have a history of the changes). You still have the ability to update the script once for all your test jobs (since hey all check out the same script). In addition you don't need to look at several scripts to find out whether your test ran successful or not. Especially if you have one job that is part of several execution lines, it becomes difficult to find out what job triggered which run. Another advantage is that you have less jobs on your Hudson and therefore it is easier to maintain.
I think what you are looking for is http://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin This plugin lets you execute other jobs based on the status of previous jobs. You can even call a shell script from the downstream project to determine any additional conditions. which can in turn call the API for more info.
For example we have a post-build step to notify us, this calls back the JSON API to build a nice topic in our IRC channel that says "All builds ok" or "X,Y failed" , etc.