I am trying to understand all the implications of switching our java projects from a Snaphot/Release policy to build promotion.
One obvious step is that each build ends up creating an artifact that might be going all the way to the production environment, so there's not Snapshot anymore. But then, how should I manage the link from a project to other artifacts, that may or may not be allowed to go to prod?
I've had a hard time finding valuable information on this particular subject. Of course, build promotion is talked about a lot, but dependency management in the light of a migration to build promotion has less visibility.
I see two choices:
One can only depend on artifacts that have been previously promoted to the production environment
When one depends on another artifacts, the built artifact can only go to the last environment of its dependencies. That is, if I depend from an artifact that was allowed to go to test and not prod, then my build won't be allowed to go to prod
Are there industry standards regarding this topic? Or best practices?
Thanks a lot for your help :)
edits:
We deploy to Artifactory three kind of artifacts:
Libraries
EARs
The modules inside the EARs. Some of those are "public" layers that are needed by any EAR that wants to interact with the currently built EAR
We deploy EARs to JEE servers. Our libraries and public layers are deployed to Artifactory and packaged in the EARs, so they are not directly deployed on the JEE containers.
One project builds several modules, and everything is packaged in an EAR, along with its dependencies. One project can depend on a module of another project and that's where it gets complicated...
We distinguish between "deployable artifacts" and "libraries".
Deployable artifacts (like ears, wars, standalone jars) go through a pipeline, so they are promoted and tested in different steps. They cannot be dependencies for any other artifact.
Libraries, on the other hand, are not promoted. When they are built (as a release version), are immediately available as possible dependency for all other artifacts (the release build includes unit tests and some integration tests). They are tested and promoted indirectly when they are used in deployable artifacts.
Related
I just started developing a project in Spring MVC and i want to know how important Maven is.
The following are the key features of Maven :
Simple project setup that follows best practices - get a new project or module started in seconds
Consistent usage across all projects - means no ramp-up time for new developers coming onto a project
Superior dependency management including automatic updating, dependency closures (also known as transitive dependencies)
Able to easily work with multiple projects at the same time
A large and growing repository of libraries and metadata to use out of the box, and arrangements in place with the largest Open Source projects for real-time availability of their latest releases
6.Extensible, with the ability to easily write plugins in Java or scripting languages
Instant access to new features with little or no extra configuration
Ant tasks for dependency management and deployment outside of Maven
Model based builds: Maven is able to build any number of projects into predefined output types such as a JAR, WAR, or distribution based on metadata about the project, without the need to do any scripting in most cases.
Coherent site of project information: Using the same metadata as for the build process, Maven is able to generate a web site or PDF including any documentation you care to add, and adds to that standard reports about the state of development of the project. Examples of this information can be seen at the bottom of the left-hand navigation of this site under the "Project Information" and "Project Reports" submenus.
Release management and distribution publication: Without much additional configuration, Maven will integrate with your source control system (such as Subversion or Git) and manage the release of a project based on a certain tag. It can also publish this to a distribution location for use by other projects. Maven is able to publish individual outputs such as a JAR, an archive including other dependencies and documentation, or as a source distribution.
Dependency management: Maven encourages the use of a central repository of JARs and other dependencies. Maven comes with a mechanism that your project's clients can use to download any JARs required for building your project from a central JAR repository much like Perl's CPAN. This allows users of Maven to reuse JARs across projects and encourages communication between projects to ensure that backward compatibility issues are dealt with.
Getting started with maven only takes about 10 minutes. Reasons why you should learn maven:
It helps you manage your dependencies very easily so you don't need to add jars to your project classpath manually
You can run unit tests
Has over 20 useful plugins which you can use. Plugins make up lifecycles like test, package which make your work more efficient
You can use it to build your project
The most important thing about it at the beginning is that you don't need to worry about setting up your project by adding dependencies, maven does it for you automatically.
Read this spring guide for building with maven
Any other guides in this section for spring boot has the same mechanism
We have started to implement Continuos Delivery for our Java Builds using Maven and Teamcity tooling for CI and Build automation.
We have few common jars that are built as standalone jar artefacts and are consumed by web modules.
Frequency of the change to these common modules is high; we have started to adopt the approach discussed in various forums What is the Maven way for automatic project versions when doing continuous delivery? and in this blog
http://blog.xebia.com/2012/09/30/continuous-releasing-of-maven-artifacts/ to use Major.Minor.BugFix-${revision} for all the common jars.
Value for revision is set in Parent POM as SNAPSHOT for local development and in case of Teamcity builds it is set to ${BuildNumberCounter}-${SVNRepoRevisionNumber} e.g. 1.0.0-10-233
For a Web Module that needs to consume the jar and always wants to pick the latest version Dependency range is defined as [1.0.0,2.0.0). This seems to be working fine; however to be honest we have not yet used this in anger, so will see if we hit challenges.
The problem that we have straightaway is that for local desktop development the dependency range in the Webmodule always resolves to the latest numbered release rather than snapshot build that was created by the developer for local testing of the common jar with the Web Module. We believe it is valid for the developer to be able to test the change of common jar with web modules locally. Only way it can be achieved is by committing the change and Teamcity producing new numbered release which is not ideal as it would potentially break the build of all Web Modules that use that common jar.
Wonder if anyone has faced similar problem and would have a solution.
EDIT: This is about doing Continuous Delivery with Maven and having it orchestrated with Jenkins. Maven is definitively not designed for that, and this question is part of our effort to get an efficient workflow without using Maven releases. Help is appreciated.
We use Maven -SNAPSHOTs within major versions to ensure customers always get the latest code for that given version, which works well. For technical reasons we have two independent Maven jobs - one for compiling sources to jars, and one for combining the appropriate jars to a given deployment. This also works well.
We then have Jenkins orchestrating when to invoke the various steps, and this is where it gets a bit tricky, because if we do the normal mvn clean install in step one, this means that all the snapshot artifacts get recompiled, which in turn makes Jenkins think that all the snapshots changed (as their fingerprint - aka MD5 checksum - changed) even if the sources used to generate the artifacts did not change, triggering all the downstream builds instead of just those which dependencies did change.
I have so far identified these things as varying between builds:
META-INF/maven/.../pom.properties (as it contains a timestamp)
META-INF/MANIFEST.MF (contains JDK and user)
timestamps in jar file
I have found ways around the two first, but the latter is a bit more difficult. It appears that AbstractZipArchiver (which does all the work in zipFile() and zipDir()) is not written to allow any kind of extension to how the archive is being generated.
For now I can imagine four approaches (but more ideas are very welcome):
Create a derivative of the current maven-jar-plugin implementation allowing for a timestamp=<number> attribute which is then used for all entries inserted into the jar file. If not set, the current behavior is kept.
Revise the Jenkins fingerprinting scheme so it knows about jar files and only looks at the entries contents, not their metadata.
Attach a plugin to the prepare-package stage responsible for touching the files with a specific time stamp. This requires all files to be present at that time (meaning that the jar plugin cannot be allowed to touch the MANIFEST.MF file)
Attach an extra plugin to the "package" phase which rewrites the finished jar file, zeroing out all zip entry timestamps in the process.
Again, the goal is to make maven SNAPSHOT artifacts fully time independent so given the same source you get an artifact with the same MD5 checksum. I also believe, however, that this could be beneficial for release builds.
How should I approach this?
As per my comment, I still think the answer is to do none of the things you suggest, and instead use releases in preference to snapshots for artifacts which you are in fact releasing to customers.
The problems you describe are:
you have a multi-module project which takes a long time to build because you have more than 100 modules,
you have two snapshot artifacts which you think ought to be identical (because the source code and metadata were identical at build time), but they have different checksums.
My experience with Maven tells me that if you try and adhere to the "Maven Way", tools will work well for you out-of-the-box, but if you deviate then you'll have a bad time. Unfortunately, the Maven Way is sometimes elusive :-)
Multi-module projects in Maven are very useful when you have families of modules with code that varies in sympathy, e.g. you have a module containing a bunch of interfaces, and some sibling modules providing implementations. It would be unusual to have more than a dozen modules in a multi-module project. All the modules ought to share the version number of the parent (Maven doesn't enforce this, which in my opinion is confusing).
When you build a snapshot version of a multi-module project, snapshots of all modules are built, even if the code in a particular module hasn't changed. Therefore you can look at a family of modules in your repositiory, and know that at compile time the inter-module code references were satisfied.
For example, in a domain model module you might have an interface:
public interface Student {
void study();
}
and in some sibling modules, which would declare compile-scoped dependencies on the domain model in their POMs, you might have implementations.
If you were then to change the interface in the domain model module:
public interface Student {
void study();
void drink(Beer beer);
}
and rebuild the multi-module project, the build will fail. The dependent modules will fail to build, even though their code and POMs have remained the same. In a multi-module project, you only install or deploy artifacts if all the child modules build successfully, so rebuilding snapshots is usually very desirable - it's telling you something about the inter-module dependencies.
If:
you have an excessive number of modules, and/or
those modules can't reasonably share the same version number, and/or
you don't need any guarantees about code references between modules,
then your modularisation is incorrect. Don't use multi-module projects as a build system (you have Jenkins for that), use it instead to express relationships between modules of your code.
In your comment, you say:
RELEASE artifacts behave the same way when being rebuilt by Jenkins.
The point of point of release artifacts is that you do not rebuild them - they are definitive! If you use something like Artifactory, you will find that you cannot deploy a release artifact more than once - your Jenkins job should fail if you attempt it.
This is a fundamental tenet in Maven. One of the aims of Maven is that it if two developers on separate workstations were to attempt the same release, they would build artifacts which were functionally identical. If you are build an artifact which expresses a dependency (maybe for compilation purposes, or because it's being assembled into .war etc.) on another, then:
if the dependency is a snapshot, Maven might seek a newer version from the repository.
if the dependency is a release, the version in your local repository is assumed to be definitive.
If you could rebuild a release artifact, you would create the possibility that two developers have dissimilar versions in their repository, and you'd have dissimilar builds depending on which workstation you used. Don't do it.
Another critical detail is that a release artifact cannot depend on snapshot artifacts, again, you would lose various guarantees.
Releases are definitive, and it sounds like you want your assembly to depend on definitive artifacts. Jenkins makes tagging and releasing multi-module projects very straightforward.
In summary:
Check your modularisation: one enormous multi-module project is not useful.
If you don't want to continually rebuild snapshots, you need to do releases.
Never release snapshots to your customer.
Follow the dependency graph of your assembly project and release any snapshots.
Release the assembly project, bumping your minor version.
Ensure your customer refers to the complete version number of your assembly in communications.
I have two maven projects, which I need to deploy through automatic deployment process. (like nightly build or similar)
Scenario is as:
mv-proj1
-dependency-1.jar
-dependency-2.jar
-dependency-3.jar
mv-proj2
-dependency-3.jar
-dependency-4.jar
-mv-proj1.jar
sources of mv-proj1 and mv-proj2 can not be disclosed.
mv-proj2 is executable jars and provide services to other application modules.
So what is the standard way of deploying these to production machine or lets say UAT machine?
Do I need to set up intra-organization maven repository?
Do I need to install maven repository to UAT machine?
One possible way I could think is to set up and host intra-organization maven respository as well as setting up maven on UAT machine to fetch data from intra-organization maven respository. and deploy only pom.xml.
I would let my choice depend on what the consumers of your artifacts are.
If the consumers are also Maven projects that can pull in your JARs from said intra-organizational Maven repository, that's definitely a great way to go. I believe that every organization that is serious about using Maven is sooner or later going to have use cases for such own repository anyway. I've worked with Artifactory and Nexus and feel that both are great products (and free beer for the use case as stated here). They're both easy to install, and it should not be an exploration that is daunting, go for it!
If your UAT machines would use Maven to build and install anything that pulls in your artifacts as a dependency, them would be running Maven client-side. There would be a local repository (artifact cache) on these clients, but that's a different beast than the organizational repository mentioned above, which you would likely deploy not on the actual UAT machines.
If mv-proj2 is rather a "final delivery", executable as you say, you may want to pack it all up as a nice, single JAR (Maven can do that for you) and distribute that to your users. You could do that again through an organizational repository. You could ultimately release it to some network drive or web server. Many ways to do so, e.g. use maven-jar-plugin with outputDirectory pointing to wherever you want to release.
I hope I can keep this question specific enough, my team at work is currently debating the best way to manage our dependencies for a huge project (150+ dependencies ~300mb).
We have two main problems
Keeping all the developers dependencies the same so we are compiling against the same files
Ensure the project (once compiled) is comliped against the same dependencies
The two ideas that have been suggested are using a BirJar (all dependencies in one file) and just adding a version number to it and using a shared folder and pointing everyone's machines at the same place.
Or making including all the dependencies in the jar when we compile it (a jar, of jars, of jars) and just have a project that "has no dependencies"
Someone also mentioned setting up an internal version of Ivy and pointing all the code to pull dependencies from there.
What are the best practices regarding massive dependency management?
Why don't you use Maven and its dependency management ?
You can specify each dependency, its particular version and its scope (compile-time, for testing, for deployment etc.). You can provide a master pom.xml (the config file) that specifies these, and developers can override if they need (say, to evaluate new versions).
e.g. I specify a pom.xml that details the particular jars I require and their versions (or range). Dependent jars are determined/downloaded automatically. I can nominate which of these jars are used for compilation vs. deployment etc. If I use a centralised repository such as Nexus I can then build my artefact (e.g. a library) and deploy that into Nexus, and it'll become available for other developers to download in exactly the same manner as 3rd party libs etc.
Incase you dont like/want to follow the Maven project structure...
If you already use Ant, then your best bet is to use Ivy for dependency management.
http://ant.apache.org/ivy/
It provides a rich set of ant tasks for dependency manipulation.
from : Ant dependency management