I have multiple java projects in a folder. Also there is a second folder with libraries, that might be used as build dependencies from the projects. The projects may also have dependencies to other Projects. What's the best approach to build all projects ?
In other words I want to build the projects without explicit telling their dependencies.I think the biggest problem is the dependecy between the projects.
There are multiple build systems that are available that you may use. Maven has a complete dependency system built into it. Almost all third party open source jars are directly accessible via the World Wide Maven repository system. Basically, you describe the jar you need (groupId, artifactId, and version) and Maven will automatically fetch it for you. Not only that, but Maven also will build your project without having to create a build file. Instead, you have to describe your project in a project object model (a pom.xml file) and Maven will download everything you need, including all compilers, etc.
Almost all new projects use Maven, but Maven has a few downsides:
Since you don't control a build process, it can sometimes feel like poking a prodding a black box to get the build to work the way you want.
Documentation can be scant -- especially if you're moving beyond basic Java compiles.
You usually have to arrange your project in a specific layout. For example, source files should go under src/main/java while JUnit tests are under src/test/java. You don't have to follow the recommended layout, but then you'd have to modify the pom.xml file this way and that to get your build to work. That defeats the whole purpose of the pom.xml in the first place.
If you already have another build system setup (like Ant), you lose everything. There's no easy way to move from Ant to Maven.
The other is called Ant with Ivy. Ivy uses Ant for building, but can access Maven's world wide repository system for third party dependencies. It's a great compromise if you already are heavily invested in Ant. I also find Ant with Ivy to be better documented than Maven (although that's not too difficult). There's an excellent chapter going over the basics of Ivy in Manning Publication's Ant in Action.
With either process, I would recommend that you build a company wide Maven repository using either Nexus or Artifactory. This way, any proprietary third party jars (like Oracle jars) can also be stored in your company wide Maven repository since they won't be in the standard World Wide Maven repository.
By the way, if this is a company wide effort, and you are moving multiple Ant projects into Ivy, I have an Ivy project I use in Github that makes things easier.
Oh, there's a third possibility called Gradle which I know nothing about. I also believe it can use the World Wide Maven repository. It's based on Groovy which is based on Java syntax, and that's about all I can say. Maybe others can fill you in on the details. The Gradle group contends it solves a lot of problems of both Ant/Ivy and Maven.
Whatever tool you use, if you have various projects interdependent, you need to be clear on the independent ones which will be built first before building the dependent projects. You need to have a clear dependency structure for your projects.
You can do this with Apache Ivy. You can lay out the locations for you common libraries, define published artifacts and inter-dependencies in an ivy.xml document in each project, and let a top-level Ant build with the Ivy tasks figure out what the build order should be based on those dependencies.
Related
Semi-greenthumb here. I'm looking to download some Apache Commons and Google Guava libraries to use in Eclipse. Multiple Q&As (example, example) have said to download the library myself, and then either load it in Eclipse by path as a "User Library" that I can add manually to projects or go through an automated project management plugin like Maven. However, that leaves the question, where should I actually store the library on my system? (Mac OS)
Ideally, I want it in a directory that is common to all Mac/*NIX systems. However, this Q&A seems to suggest that doing so would be a bad idea, and this comment implies that I should keep a separate copy of the library within each project that uses it. This seems like it would be both a waste of space (for projects that use the same library version), as well as make linting Java files in a separate text editor a hassle due to libraries being stored within an Eclipse project's file structure rather than at the system level.
So where should I put 3rd party Java libraries?
I faced the same issue when I was maintaining my project dependencies in a manual way. It is difficult to have control over them, and sometimes updating a library can be a really painful experience if that update breaks a transitive dependency.
All this pain went away when I switched to Maven.
When you configure Maven, you can set the directory where these libraries will reside (common path is {user.dir}/.m2 } and every time a dependency is added to a project (via POM), then Maven will check if that library is already downloaded. If not, it will download it and store it for any future use (of the same version). It also resolves transitive dependencies for you, so you don't have to worry of breaking it when manually replacing a JAR.
This way you don't have to worry where the libraries are, your IDE will reference them automatically using the apropiate Maven plugin
I'm not saying you should use Maven, but if your problem is managing dependencies, then Maven (or any other dependency management system, eg: Gradle) may help you.
The comment you cited is pretty naïve in its approach. There are far too many build management tools to handle dependencies without having to deal with these minutiae.
If you decide on a tool such as Maven, your dependencies will be downloaded into a specified local repository (a directory on your filesystem), and all Mavenized applications can easily be configured to use those (shared) artifacts.
Most Java supported IDEs like Eclipse come with the option to initialize projects with Maven (or Gradle, as another example) and have sleek interfaces to easily edit their configuration files to specify which dependencies your projects will use.
I would strongly recommend either of those as opposed to manual JAR/artifact management, even for basic personal tinkering projects.
In my new project I am confronted with a complex infrastructure with several modules which have grown over the years in an unpleasant, uncontrolled way.
To come to the point: The build process is the horror. There are over 40 different, complex Ant files, which are connected multiple times and the SOA framework also generates several dynamic Ant files. It took a few days to really understand all the dependencies and to finally build the whole project without any errors.
My plan was or is to migrate the whole project from Ant to Maven, since new components are planned and I would like to avoid these problems in the future and well, because it is just horrible the way it is now ;-)
Since I am new to the migration of bigger projects, I am a little bit confused about the best workflow. There are dozens of XML files and scripts involved, which are distributed in a non-Maven directory structure. Overall there are over 3000 files involved. One of the main problems is that I don't know if I really should try to migrate everything in the known Maven directory structure and therefore risk endless editing and refactoring of every single file. Or should I keep the folder structure as it is and bloat my pom.xml files and possibly run into problems with all the different involved plugins? Honestly, both ways don't sound quite constructive.
Does it even make sense to migrate a project in this dimension to Maven? Especially when the SOA framework must use its own Ant files - therefore a combination of Ant and Maven would be necessary. What would be the best strategy to simplify this process?
Thanks for all suggestions.
Here's a simple and quick answer to Mavenizing an Ant project:
DON'T DO IT!
This is not some anti-Maven screed. I use Maven, and I like Maven. It forces developers not to do stupid things. Developers are terrible at writing build scripts. They want to do things this way and not the way everyone else does. Maven makes developers setup their projects in a way that everyone can understand.
The problem is that Ant allows developers to do wild and crazy things that you have to completely redo in Maven. It's more than just the directory structure. Ant allows for multiple build artifacts. Maven only allows for one per pom.xml1. What if your Ant project produces a half dozen different jar files -- and those jar files contain many of the same classes? You'll have to create a half dozen Maven projects just for the jars, and then another half dozen for the files that are in common between the jars.
I know because I did exactly this. The head of System Architecture decided that Maven is new and good while Ant must be bad and Evil. It didn't matter that the builds worked and were well structured. No, Ant must go, and Maven is the way.
The developers didn't want to do this, so it fell to me, the CM. I spent six months rewriting everything into Maven. We had WSLD, we had Hibernate, we had various frameworks, and somehow, I had to restructure everything to get it to work in Maven. I had to spawn new projects. I had to move directories around. I had to figure out new ways of doing things, all without stopping the developers from doing massive amounts of development.
This was the inner most circle of Hell.
One of the reasons why your Ant projects are so complex probably has to do with dependency management. If you are like our current shop, some developer decided to hack together develop their own system of dependency management. After seeing this dependency management system, I now know two things developers should never write: Their own build files, and dependency management systems.
Fortunately, there is an already existing dependency management system for Ant called Ivy. The nice thing about Ivy is that it works with the current Maven architecture. You can use your site's centralized Maven repository, and Ivy can deploy jars to that repository as Maven artifacts.
I created an Ivy project that automatically setup everything for the developers. It contained the necessary setup and configuration, and a few macros that could replace a few standard Ant tasks. I used svn:externals to attach this Ivy project to the main project.
Adding the project into the current build system wasn't too difficult:
I had to add in a few lines in the build.xml to integrate our ivy.dir project into the current project.
I had to define an ivy.xml file for that project.
I changed any instance of <jar and </jar> to <jar.macro and </jar.macro>. This macro did everything the standard <jar/> task did, but it also embedded the pom.xml in the jar just like Maven builds do. (Ivy has a task for converting the ivy.xml file into a pom.xml).
I Ripped out all the old dependency management crap that the other developer added. This could reduce a build.xml file by a hundred lines. I also ripped out all the stuff that did checkouts and commits, or ftp'd or scp'd stuff over. All of this stuff was for their Jenkins build system, but Jenkins can handle this without any help from the build files, thank you.
Add a few lines to integrate Ivy. The easiest way was to delete the jars in the lib directory, and then just download them via ivy.xml. All together, it might take a dozen lines of code to be added or changed in the build.xml to do this.
I got to the point where I could integrate Ivy into a project in a few hours -- if the build process itself wasn't too messed up. If I had to rewrite the build.xml from scratch, it might take me a two or three days.
Using Ivy cleaned up our Ant build process and allowed us many of the advantages we would have in Maven without having to take a complete restructuring.
By the way, the most helpful tool for this process is Beyond Compare. This allowed me to quickly verify that the new build process was compatible with the old.
Moving onto Maven Anyway...
The funny thing is that once you have integrated your Ant projects with Ivy, turning them into Maven projects isn't that difficult:
Clean up the logic in your build.xml. You might have to rewrite it from scratch, but without most of the dependency management garbage, it's not all that difficult.
Once the build.xml is cleaned up, start moving directories around until they match Maven's structure.
Change source to match the new directory structure. You may have a WAR that contains *css files in a non-standard location, and the code is hardwired to expect these files in that directory. You may have to change your Java code to match the new directory structure.
Break up Ant projects that build multiple projects into separate Ant projects that each builds a single artifact.
Add a pom.xml and delete the build.xml.
1 Yes, I know this isn't entirely true. There are Maven projects with sub-projects and super poms. But, you will never have a Maven project that builds four different unrelated jars while this is quite common in Ant.
I have done a similar migration in the past, and I had the same doubts you had; however, I went for the "keep the folder structure intact and specify the paths in the POM files" way and I noticed it wasn't as bad as I thought.
What I actually had to do was to appropriately set the <sourceDirectory> and the <outputDirectory>and maybe add some inclusion and exclusion filters, but in the end I'd say that even if Maven's way is really convention-over-configuration-ish and makes your life easier if you follow its directives on where to place files, it doesn't really make it much harder if you don't.
Besides, something that really helped me when migrating was the possibility to divide the Maven project in modules, which I initially used to replicate the Ant structure (i.e. I had one Maven module for each build.xml file) making the first stage of the migration simpler, and then I changed the module aggregation to make it more meaningful and more Maven-like.
Not sure if this does actually make any sense to you, since I didn't have any generated Ant files which I recon may be the biggest issue for you, but I would definitely follow this road again instead of refactoring and moving files everywhere to Mavenize my project structure.
I've been asked to apply Maven to a project. After browsing a dozen sites it appears that it's quite vast and I'm not familiar as I'd like with similar tools like Ant. Why is it used/preferred and what does it offer over a standard Eclipse project? Also, how could it be added to an existing project?
Why is it used/preferred and what does
it offer over a standard Eclipse
project?
It is a build tool which can build your project without the need for an IDE like Eclipse. It can create a jar or war or other artifacts from the source, performing a bunch of steps like compilation, running unit tests, etc.
Where maven scores over ant is in managing third-party dependencies and in convention over configuration (which mean less lines of build script if you follow convention).
Also, how could it be added to an
existing project?
You start by creating a new maven project, following the step here.
Place it in the root folder of your project
If your source and resource files do not follow maven folder convention, update maven properties suitably referring to this documentation.
Run mvn package
It will fail if it needs any third party dependencies, which you can add as specified in the doc
With some trial and error, you should have your project running with maven, possibly, much quicker than if you were to set up the same with ant.
Others are already provided sufficient resources to read more about maven.
I suggest to start reading here:
http://www.sonatype.com/books/mvnref-book/reference/public-book.html
Maven is a great tool when you know how to use it. Maven (at core) is a dependency manager.
You include in your pom.xml (similar in function to the build.xml from Ant) all the librairies your project depends on (example : apache commons) along with their version and Maven get them directly from a repository (by default, the central maven repository)
Then you do not have to manually install any jar to make your project work. All is downloaded and cached on your local machine. You can even create an enterprise repository where you put all the jars needed by your company
Maven uses the concept of artifacts which are pre-built library projects with their own dependencies
To mavenize a project, you'll have to write a pom.xml describing your project (examples are numerous), get rid of your libs directory (or whatever classpath you described under Eclipse) and add all your dependencies to your pom.xml
You could also check Mavenizer for a first-start
But Maven is a lot more what i've just said. Read the docs, read poms from librairies and you'll get used to it quickly ;-)
If you use the M2Eclipse plugin from Sonatype, it's just a matter of right clicking the project in the package explorer and choosing Enable Dependency Management in the Maven menu. You are also advised to adjust the directories that contain the sources to the Maven standard directory layout but if you absolutely can't, you can configure that later.
Apart from that: Well, look for tutorials and documentation (for example there is the free book Better builds with Maven. Maven is very complex (yes, I don't think it is simple) and very powerful.
Perhaps the reason I stalled learning Java until now is because I HATE how Java handles external libraries. I'm stuck keeping them in one place, adding them individually, fixing problems with versioning and every time I move/rename them, and copying and writing the classpath over and over each time I release a Java application.
There has to be an elegant solution to all of this. I keep all of my libraries (regardless of task, platform, or other) in their own little folder inside a "lib" folder in my development folder, kind of like this:
Dev
-lib
+JS-jQuery
+Flex-Degrafa
-Java-Xerces
+Xerces-1.2.3
+More libraries
I can use either Netbeans or Eclipse for Java dev, but none of them provide a very streamlined (and not to mention idiot-proof) way of managing all of these.
A nudge in the right direction or an online article/tutorial on this would be greatly appreciated.
You can either use Ant + Ivy or Maven to manage your library dependencies.
If it is only dependency management you're after and you're happy with the rest of your build process, I would use Ivy, as it can unobtrusively manage your dependencies, leaving your existing build process intact. There is a plugin for Eclipse called IvyIDE that contributes your dependencies via a classpath container.
Maven 2 has a steeper learning curve but provides a much richer set of functionality for building your projects and Eclipse integration through m2eclipse or IAM.
Personally I use Maven as I have a large number of projects to work with and Maven is particularly suited to efficient development across lots of projects.
Have a look at the introductory documentation to see what works for you.
Ivy Tutorial
Maven Getting Started Guide
Netbeans 6.7.1's Maven support is quite good and comes out of the box with the IDE.
The Eclipse addon was frustrating enough that I gave Netbeans another try.
A third choice besides ChssPly76's options is to use Ant with the Maven Ant Tasks. I don't know if I'd call any of these solutions particularly "elegant," but they do spare you the need to manage your own lib/ directory and classpath variables.
If you're working on Linux you can install Java libraries with APT or RPM.
Otherwise, I normally check precompiled JARs into a lib directory in my project's version control repository and make sure the names of the JAR files include full version information. E.g. lib/foo-1.5.6.jar, not lib/foo.jar.
To avoid having to manually set the classpath before running your app, you can set the classpath in the Manifests of the JARs themselves to define the dependencies of each JAR file. The JVM will follow all the dependencies when loading classes.
Maven is often more trouble than it's worth, but the ability to open a maven project directly into IDEs such as IntelliJ is excellent. For example, IntelliJ will download all dependencies and have them available without having to run a build first, or an mvn command and then a project refresh. It also isn't necessary to re-generate the project every time a dependency is added. I work with a number of Eclipse developers who switched to IntelliJ for this alone.
However, one shortfall of Maven is that many libraries (or versions of libraries) are not available on public repositories. Therefore it is often necessary to set up a local repository such as archiva. In ant, it would just be a matter of adding it to the lib directory in the repository.
Maven can also attack when you need to do something that maven doesn't directly support via a plugin. What would normally be a few lines of ant can often turn into a morning's worth of work.
Finally, buildr is an excellent way of using Maven's dependency management and plugins, while also supporting ad-hoc tasks.
I have seen many ivy files in my application's build projects. What is Ivy, and its relation with ant?
Ivy is an Ant extension for managing external libraries during the development process, giving you a way of adding them to your classpath, bundling them into your application build, etc.
There's some overlap with what Maven does, but Ivy is much more lightweight (and doesn't do as much).
Ivy is a dependency manager -- it manages and controls the JAR files that your project depends on. If you don't have the JARs, it will pull them down for you by default (from the Maven 2 repository), which can make project setup a lot easier.
Ivy is originally a Jayasoft.fr product which at that time was under BSD license, however the core of Ivy has always been the same: It's a dependency management tool.
Ivy's relation to Ant is that it is an Ant extension in the sense that it is through Ant, however it doesn't really depend on it (beyond the obvious parts). Ivy was merged into an Apache project under Ant as subproject through incubation process on October 11, 2007.
Very common misconception people have is that they compare Ivy with Maven as a whole. However that doesn't really work since Ivy only handles dependencies in an agile manner and is very good at it with simplistic configuration and wide automated support with various build systems etc. through Ant while Maven really is a build system in itself. Personal opinions may and do differ in which one does what best and that is worth several Stackoverflow/Google searches on its own.
I believe these are significant and useful points to add:
Ivy can be used standalone from command line without Ant. If your CI environment does not include Ant it will not preclude you from using Ivy. Yes, integration with Ant is nice and, by far, better documented. One could apply Ant integration documentation to standalone usage though.
Ivy is not just for Java. My team successfully uses it to define dependencies for a fairly large .NET project (10 interdependent packages with several versions of each in production with multiple third party upstream dependencies). Works quite well. I wish documentation were a bit clearer and fuller.
There are additional perks that Ivy offers for Build Time: upstream and downstream builds triggering.