I have a multi-module maven project and looking to leverage the parallel build feature in Maven 3. But some of the modules lay the groundwork for future modules (when built sequentially, like creating some directories, downloading some non-maven jars etc). Is there a way for me to control the parallelism partially so that the build starts of sequentially and then goes parallel and eventually converge to a final module that does aggregation and cannot be run before the others?
Thanks in advance
Just add a dependency on the pom (pom) we use this to order our docker builds (images depend on base images that need to built earlier)
I haven't tested this, but I think it should be possible. The downside is you will have to control that branch/join logic yourself by running mvn multiple times.
First, check out this article for how to do parallel builds: https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
Now when it comes to building individual modules, you should use this command line option:
-pl,--projects <arg> Comma-delimited list of specified
reactor projects to build instead
of all projects. A project can be
specified by [groupId]:artifactId
or by its relative path.
I think you will need to run the mvn command multiple times. Let me know if you need more info.
Side Note: Consider this line
This build-mode analyzes your project's dependency graph and schedules modules that can be built in parallel according to the dependency graph of your project.
You may not need to do any branch/join logic. See if it works without it.
Related
In my new project I am confronted with a complex infrastructure with several modules which have grown over the years in an unpleasant, uncontrolled way.
To come to the point: The build process is the horror. There are over 40 different, complex Ant files, which are connected multiple times and the SOA framework also generates several dynamic Ant files. It took a few days to really understand all the dependencies and to finally build the whole project without any errors.
My plan was or is to migrate the whole project from Ant to Maven, since new components are planned and I would like to avoid these problems in the future and well, because it is just horrible the way it is now ;-)
Since I am new to the migration of bigger projects, I am a little bit confused about the best workflow. There are dozens of XML files and scripts involved, which are distributed in a non-Maven directory structure. Overall there are over 3000 files involved. One of the main problems is that I don't know if I really should try to migrate everything in the known Maven directory structure and therefore risk endless editing and refactoring of every single file. Or should I keep the folder structure as it is and bloat my pom.xml files and possibly run into problems with all the different involved plugins? Honestly, both ways don't sound quite constructive.
Does it even make sense to migrate a project in this dimension to Maven? Especially when the SOA framework must use its own Ant files - therefore a combination of Ant and Maven would be necessary. What would be the best strategy to simplify this process?
Thanks for all suggestions.
Here's a simple and quick answer to Mavenizing an Ant project:
DON'T DO IT!
This is not some anti-Maven screed. I use Maven, and I like Maven. It forces developers not to do stupid things. Developers are terrible at writing build scripts. They want to do things this way and not the way everyone else does. Maven makes developers setup their projects in a way that everyone can understand.
The problem is that Ant allows developers to do wild and crazy things that you have to completely redo in Maven. It's more than just the directory structure. Ant allows for multiple build artifacts. Maven only allows for one per pom.xml1. What if your Ant project produces a half dozen different jar files -- and those jar files contain many of the same classes? You'll have to create a half dozen Maven projects just for the jars, and then another half dozen for the files that are in common between the jars.
I know because I did exactly this. The head of System Architecture decided that Maven is new and good while Ant must be bad and Evil. It didn't matter that the builds worked and were well structured. No, Ant must go, and Maven is the way.
The developers didn't want to do this, so it fell to me, the CM. I spent six months rewriting everything into Maven. We had WSLD, we had Hibernate, we had various frameworks, and somehow, I had to restructure everything to get it to work in Maven. I had to spawn new projects. I had to move directories around. I had to figure out new ways of doing things, all without stopping the developers from doing massive amounts of development.
This was the inner most circle of Hell.
One of the reasons why your Ant projects are so complex probably has to do with dependency management. If you are like our current shop, some developer decided to hack together develop their own system of dependency management. After seeing this dependency management system, I now know two things developers should never write: Their own build files, and dependency management systems.
Fortunately, there is an already existing dependency management system for Ant called Ivy. The nice thing about Ivy is that it works with the current Maven architecture. You can use your site's centralized Maven repository, and Ivy can deploy jars to that repository as Maven artifacts.
I created an Ivy project that automatically setup everything for the developers. It contained the necessary setup and configuration, and a few macros that could replace a few standard Ant tasks. I used svn:externals to attach this Ivy project to the main project.
Adding the project into the current build system wasn't too difficult:
I had to add in a few lines in the build.xml to integrate our ivy.dir project into the current project.
I had to define an ivy.xml file for that project.
I changed any instance of <jar and </jar> to <jar.macro and </jar.macro>. This macro did everything the standard <jar/> task did, but it also embedded the pom.xml in the jar just like Maven builds do. (Ivy has a task for converting the ivy.xml file into a pom.xml).
I Ripped out all the old dependency management crap that the other developer added. This could reduce a build.xml file by a hundred lines. I also ripped out all the stuff that did checkouts and commits, or ftp'd or scp'd stuff over. All of this stuff was for their Jenkins build system, but Jenkins can handle this without any help from the build files, thank you.
Add a few lines to integrate Ivy. The easiest way was to delete the jars in the lib directory, and then just download them via ivy.xml. All together, it might take a dozen lines of code to be added or changed in the build.xml to do this.
I got to the point where I could integrate Ivy into a project in a few hours -- if the build process itself wasn't too messed up. If I had to rewrite the build.xml from scratch, it might take me a two or three days.
Using Ivy cleaned up our Ant build process and allowed us many of the advantages we would have in Maven without having to take a complete restructuring.
By the way, the most helpful tool for this process is Beyond Compare. This allowed me to quickly verify that the new build process was compatible with the old.
Moving onto Maven Anyway...
The funny thing is that once you have integrated your Ant projects with Ivy, turning them into Maven projects isn't that difficult:
Clean up the logic in your build.xml. You might have to rewrite it from scratch, but without most of the dependency management garbage, it's not all that difficult.
Once the build.xml is cleaned up, start moving directories around until they match Maven's structure.
Change source to match the new directory structure. You may have a WAR that contains *css files in a non-standard location, and the code is hardwired to expect these files in that directory. You may have to change your Java code to match the new directory structure.
Break up Ant projects that build multiple projects into separate Ant projects that each builds a single artifact.
Add a pom.xml and delete the build.xml.
1 Yes, I know this isn't entirely true. There are Maven projects with sub-projects and super poms. But, you will never have a Maven project that builds four different unrelated jars while this is quite common in Ant.
I have done a similar migration in the past, and I had the same doubts you had; however, I went for the "keep the folder structure intact and specify the paths in the POM files" way and I noticed it wasn't as bad as I thought.
What I actually had to do was to appropriately set the <sourceDirectory> and the <outputDirectory>and maybe add some inclusion and exclusion filters, but in the end I'd say that even if Maven's way is really convention-over-configuration-ish and makes your life easier if you follow its directives on where to place files, it doesn't really make it much harder if you don't.
Besides, something that really helped me when migrating was the possibility to divide the Maven project in modules, which I initially used to replicate the Ant structure (i.e. I had one Maven module for each build.xml file) making the first stage of the migration simpler, and then I changed the module aggregation to make it more meaningful and more Maven-like.
Not sure if this does actually make any sense to you, since I didn't have any generated Ant files which I recon may be the biggest issue for you, but I would definitely follow this road again instead of refactoring and moving files everywhere to Mavenize my project structure.
I'm looking at migrating from ant to Gradle because we want to speed up builds by running projects in parallel and to simplify configuration, however it seems to me that these goals are mutually exclusive for Gradle builds. Hopefully I'm missing something.
So I'd like a nice example of a multi-project Gradle setup compiling 3 java projects such that A and B can be compiled in parallel and C depends on both. Also such that some properties can be configured (dynamically) and used in all three projects, perhaps configured in an additional root project.
What do you mean by configuring properties "dynamically"? in general you should be able to get a parallel executed build for a project with three subprojects as you have described. I pushed a sample skeleton project setup at https://github.com/breskeby/gradle-snippets/tree/master/multiparallel
hope that helps,
cheers,
René
here is an example app I built a week or two ago that covers this usecase: https://github.com/jadekler/git-java-websiteskeleton. Check out the settings.gradle in particular, as well as the application package which bundles all the other components (an individual java app, each) into one.
I have multiple java projects in a folder. Also there is a second folder with libraries, that might be used as build dependencies from the projects. The projects may also have dependencies to other Projects. What's the best approach to build all projects ?
In other words I want to build the projects without explicit telling their dependencies.I think the biggest problem is the dependecy between the projects.
There are multiple build systems that are available that you may use. Maven has a complete dependency system built into it. Almost all third party open source jars are directly accessible via the World Wide Maven repository system. Basically, you describe the jar you need (groupId, artifactId, and version) and Maven will automatically fetch it for you. Not only that, but Maven also will build your project without having to create a build file. Instead, you have to describe your project in a project object model (a pom.xml file) and Maven will download everything you need, including all compilers, etc.
Almost all new projects use Maven, but Maven has a few downsides:
Since you don't control a build process, it can sometimes feel like poking a prodding a black box to get the build to work the way you want.
Documentation can be scant -- especially if you're moving beyond basic Java compiles.
You usually have to arrange your project in a specific layout. For example, source files should go under src/main/java while JUnit tests are under src/test/java. You don't have to follow the recommended layout, but then you'd have to modify the pom.xml file this way and that to get your build to work. That defeats the whole purpose of the pom.xml in the first place.
If you already have another build system setup (like Ant), you lose everything. There's no easy way to move from Ant to Maven.
The other is called Ant with Ivy. Ivy uses Ant for building, but can access Maven's world wide repository system for third party dependencies. It's a great compromise if you already are heavily invested in Ant. I also find Ant with Ivy to be better documented than Maven (although that's not too difficult). There's an excellent chapter going over the basics of Ivy in Manning Publication's Ant in Action.
With either process, I would recommend that you build a company wide Maven repository using either Nexus or Artifactory. This way, any proprietary third party jars (like Oracle jars) can also be stored in your company wide Maven repository since they won't be in the standard World Wide Maven repository.
By the way, if this is a company wide effort, and you are moving multiple Ant projects into Ivy, I have an Ivy project I use in Github that makes things easier.
Oh, there's a third possibility called Gradle which I know nothing about. I also believe it can use the World Wide Maven repository. It's based on Groovy which is based on Java syntax, and that's about all I can say. Maybe others can fill you in on the details. The Gradle group contends it solves a lot of problems of both Ant/Ivy and Maven.
Whatever tool you use, if you have various projects interdependent, you need to be clear on the independent ones which will be built first before building the dependent projects. You need to have a clear dependency structure for your projects.
You can do this with Apache Ivy. You can lay out the locations for you common libraries, define published artifacts and inter-dependencies in an ivy.xml document in each project, and let a top-level Ant build with the Ivy tasks figure out what the build order should be based on those dependencies.
I am porting a project from ant to maven. The project consists of a core set of classes which gets jar'd and used by 3-4 other subprojects. Importantly (I think?), the core jar has its own configuration that gets set at build time, and each subproject also has configuration set at build time. In the old ant system, anytime I built one of the subprojects with something like...
ant -Dconfiguration=stage clean dist
...it would build the core jar using the same (stage) configuration, and copy it into the classes dir of the subproject's build. I'd like to duplicate this functionality using maven. From the reading I've done, it seems like Maven can handle this using either project inheritance or aggregation. What are the chief differences between the two methods, and which would you recommend for the use case I've described?
Inheritance is used for sharing common things between your projects, such as 3rd party library dependency or properties, or scm locations. Aggregation is used to build your projects as a group. You want aggregation, but that doesn't mean that you won't also be using inheritance to factor out common stuff.
Update: also keep in mind that maven is not just about building (unlike ant), it is about dependency management, so you could just create dependencies on your core and not use aggregation at all. You would depend on a particular version of your core. This however doesn't satisfy your requirement of building everything at once.
Our software is written in Java and comprise many (7) projects.
These projects are Netbeans ant projects.
I'm considering to converting them to maven2.
Where can I find some hints for doing such thing?
Don't read that book. It will only make you confused. Read this book instead: "Maven - The definitive guide" http://www.sonatype.com/books/maven-book/reference/ .
Also, the maven site has a lot of information, but the structure is terrible so you'll need to use google to navigate in it.
Here is my suggestion:
Do this by hand, not with "automagic" "help" from the IDE. Maven integration doesn't work that good yet, not in any IDE.
Make sure you program project is divided into modules under a common umbrella module, so that each module produces a single binary artifact (jar, war,...) possibly accompanied by the javadoc of the source code behind that artifact, a zip with the source code etc. The basic principle is that each module produces a single artifact, containing all the non-test-code under that module. You can do this while the project is still built by ant.
Each module should conform to the standard maven directory layout. The build destination is under [module]/target/[output-type, e.g. "classes"]. The source code is under [module]/src/main/[src-type e.g. "java"] and [module]/test/[src-type]. The artifact consists of all the code under src/main, and none of the code under src/test, as it built to the target directories. You can do this while the is still built by ant.
Start by transforming the sub-module that has no dependencies on other modules in the project.
Now you can create the parent maven module pom.xml with artifact type "pom", consisting of one of the modules below. Make a child module for the first submodule (the one with only external dependencies), using the umbrella module as "parent". Remember that you need to specify version for the parent. Remember to add the child module as a "module" in the parent too. Always use ${project.version} as version in the child modules when you create multi-module projects like this. All modules under a parent must be released simultaneously in a single operation, and if you use this setting maven will make sure the version fields stay the same across all modules and gets updated everywhere during the release. This may make it difficult to re-use the existing numbering scheme, but that doesn't matter. You are never going to run out of version numbers anyway.
Add the necessary dependencies, and make sure you can build the parent and the child module together using the command "mvn clean install" from the parent module.
Proceed with the rest of the modules the same way. Dependencies to other modules under the same parent project should also use ${project.version} as the "version" they are depending on, meaning "the same version as this". NOTE THAT in order to build, the module you are depending on must be built using "mvn install", so that it gets deployed to you local (computer) repository. Otherwise the depending module will not be able to find the classes. There are NO source-code dependencies between modules in maven, only dependencies to built and packed versions installed in local and remote repositories. This can be very confusing if you come from ant-projects. Build from the root module until you get comfortable with this. It takes two days.
Don't use maven integration in IDEs. It is a bad idea. Use "mvn idea:idea" or "mvn eclipse:eclipse" to set up your workspace as a non-maven ordinary IDE project. The inter-module dependencies mechanisms in maven and the IDE aren't identical and will never be. Also, if you have several mavenized projects with dependencies in between, you want to have several of these in your workspace with dependencies set up between. You can do this with mvn idea:idea / eclipse:eclipse if you create a separate maven project file called "workspace.xml" (or whatever) in the same directory as parent module, set up as a multi-module project containing modules "." and "../otherproject" (only one-way reference here, no parent ref back). If you run "mvn idea:idea / eclipse:eclipse -f workspace.xml" you get a workspace with all these modules linked together. No IDE integration lets you do that. This sound like a lot of extra work, but the workspace.xml-file is really small. It doesn't have to contain all that dependency stuff and all that, only the reference to the modules you want to bind together in your IDE.
I did a succeful migration of NetBeans Ant project to Maven project using the instruccions by Joseph Mocker here: http://forums.netbeans.org/ptopic55953.html
I cite the important part:
close the project
rename the build.xml, nbproject files/folders to something so NB won't recognize them.
close and restart NB (so any memory cache knowledge of the project is gone)
copy in an empty pom from some other project.
open the project back up in NB (NB should now identify it as a maven project)
rearrange the files to follow the maven way (™)
This won't be an easy task since Maven2 expects the files to be organized in a specific way. Anyway Better Builds with Maven is a free book that should get you started. It will help you understand Maven and it also has a chapter on migration.
I discovered that the migration is not necessary. The real requirements that I need was automatic download of dependencies (libraries).
This is also achieved by Ivy which nonetheless uses maven repositories.
I solved converting project from ant to ant+ivy with IvyBeans.
I have built a script to migrate Ant builds to Maven. You can find more information here:
https://github.com/ewhauser/ant2maven
It won't help you with fixing your directory structure and or any additional Ant tasks, but it removes a lot of the tedious steps to get started.