I'm starting to fix a java project that has used maven and while I've got the project to build, at runtime it fails with missing dependencies. I've had a look and the errors are missing optional dependencies of included compile time dependencies. I can go through and add these but it seems to me that I can have everything building and running nicely only for some piece of code that I missed to use a missing dependency and the whole thing falls apart.
What I really want to know is whether there is an automated way to find optional dependencies that I have chosen to not include. I have used mvn dependency:tree but this only shows the dependencies I have (not sure of the scope it checks) and I have tried mvn dependency:analyze but this seems to show dependencies it thinks I don't use and those that have been pulled down indirectly. What I cannot see is how to see a list of optionals I don't include.
Currently my method of working around this is to read the poms and try to work it out from there, but I don't see this as particularly robust.
For reference, I am fairly new to maven style dependency management and on the face of it like it, but this optional thing is a bit of a stumbling block for me. I understand that optionals stop me pullin down dependencies I won't be using, but it hasn't clicked for me how I can workout what optionals are available and that I do need.
I am using Eclipse Juno, m2Eclipse (also have maven 3.0.5 cli), java 6/7.
Anyone got any ideas of how I can do this better, or what I am completely overlooking?
No the things are - somewhat - just this way. Maven does not do dependency management, it allows you to do dependency management by offering tools to use and analyze them. So the work still is on the developers side. People often mix that up.
This is mainly caused by the fact that projects often have different deployment targets. As a result sometimes they collect a bunch of jar files which are copied once into tomcat and a different set of files for weblogic. So there might be a readme in your project that states what to copy prior to deployment of the maven artifacts. Or it is implicit knowledge - then you're doomed.
dependency:analyze works on bytecode not on sources. therefore it does not see what maven knows.
Maybe mvn help:effective-pom gives a better basis to analyze the whole thing? Or you could try to modify the dependency plugin to show that information as well. Maven plugins are not so hard to work with.
I'm not aware of a plugin that displays all optional transitive dependencies. But since the pom.xml files of dependencies are downloaded into the local maven repo you could do a text search there.
A while ago there was a discussion on optional dependencies as well: Best strategy for dealing with optional dependencies - it might be helpful too.
Related
Is there a way to fail a build in Jenkins if a certain jar is used in a Java Maven Project?
For example I know org.example:badartifact:1.0.1 has a security vulnerability. I told everyone about that, and they fixed their projects..., but maybe some third-party artifacts bring this with them as a transitive and nobody realizes that.
Or maybe someone down the line forgets this old bug...
So I would like to have a last check in Jenkins preferably, so that we don't end up with projects that have that special artifact included.
How do you handle situations like that, what tools do you use? (Whitelisting libs? Blacklisting libs?, etc)
Any suggestions are appreciated.
Possible Maven solution
You could have a company super POM (parent POM of all Maven projects within the company/department/team) and in that super POM configure the Maven Enforcer Plugin, its bannedDependencies rule to ban any library, version or even scope. I have personally used this option even for trivial mistakes (i.e. junit not in test scope would make the build fail).
This solution is a centralized one and as such easier to maintain, however requires all the projects to have the same parent POM and developers could at any time change the parent pom and as such skip this governance. On the other hand, a centralized parent POM is really useful for dependencies Management, common profiles, reporting and so on.
Note: you cannot configure it in the Maven settings of the Jenkins server via an active by default profile, for instance, in order to have it applied to all running Maven build, because Maven limits customization of builds in profiles provided by the settings (it's a design choice, to limit external impact and as such have an easier troubleshooting). I've tried it in the past and hit the wall.
Profiles in external files
Profiles specified in external files (i.e in settings.xml or profiles.xml) are not portable in the strictest sense. Anything that seems to stand a high chance of changing the result of the build is restricted to the inline profiles in the POM. Things like repository lists could simply be a proprietary repository of approved artifacts, and won't change the outcome of the build. Therefore, you will only be able to modify the and sections, plus an extra section
Possible Jenkins solution
If you want to have governance centralized in Jenkins directly, hence independently than Maven builds, I have applied these solutions in the past (and they perfectly work):
Jenkins Text Finder Plugin: you can make the build fail in case a regex or a matching text was found as part of the build output. In your case, you could have a Jenkins build step executing always mvn dependency:tree and as such have as part of the build output the list of dependencies (even transitive). A Text Finder rule matching your banned dependency will then match it and fail the build.
Fail The Build Jenkins Plugin: similar to the one above, but with a centralize management of configured Failure Causes. Again, failures are based on matching text, but no build configuration is required: it will be applied by default to all builds.
Here is one solution to do the job :)
With the Maven License plugin, you can scan the 3rd party dependencies for your Maven project and produce a THIRD_PARTY.txt report (in the target/generated-sources/license folder).
Maven command line:
mvn license:aggregate-add-third-party
Next, you can use the TextFinder plugin to search the "unsafe" dependencies in the THIRD_PARTY.txt file (ex: org.example:badartifact:1.0.1) and change the status of the build if needed.
Another solution is to use a 3rd party tool to do that.
I'm doing some investigation with this one: http://www.whitesourcesoftware.com/
This tool can provide a list of 3rd party dependencies with vulnerability issues.
I am currently dealing with a huge Eclipse project (not written by me). This project doesn't use any dependency management tools. It references hundreds of JARs.
Some of these JARs contain same packages (and classes), but in different versions. Currently, resolving conflicts works by manually (and randomly!) reordering these JARs in Order&Export (in Project Properties).
This was done for a long time now, and there are now lots of packages/classes with different vendors/versions/product-lines.
Reordering causes some parts of the project to fail while other parts start working, and oppositely.
Strangely, lots of orders do not cause build errors, but only runtime errors.
Can this mess be solved by an tool, which would suggest certain automatic order of dependent JARs?
Google for JarAnalyzer, that helps at least to figure how the dependecies are build up. Use the jars, your eclipse project is producing, as well. However you can not really automate this. Imagine one of your eclipse projects in needing bad-1.0.jar and another one uses bad-1.2.jar. Very often you can not replace the 1.0 one with the 1.2 one because your project wont compile any more. So in the long run you have to REMOVE outdated jars, switch to a "common version" amoung all subprojects and fix the compiler errors. And while you do that, switch to ivy or maven.
Do your jar files even have proper names or do you have 3 different versions of bad.jar which look the same in the filesystem but are in fact of different version? If so, start by renaming all relevant jar files to include the version number (can often eb found in the manifest file) ... heck I once did what you do and wrote me with JArAnalyzer, a bit groovy and some shell scripts a small tool that generated all the ivy files for the project.
you can use maven, ivy to clean the mess :) . And that spring doesn't work properly try this:first clean then build the project.
"Strangely, lots of orders do not cause build errors, but only runtime
errors."
This is not strange. As you wrote, classes are present in different versions, which does not necessarily means compilation error, but means different behaviour and different sub dependencies.
Avoid a "random" or "automatic order" approach. I would advise you the usage of Maven for handling your dependencies (in order to know precisely which library depends on which one). You will probably discover that many of the libraries you're including are not required, and that the dependency management tool will handle for you "automatically" all dependencies between dependencies, you will have however to add/force exclusion for specific libraries versions.
Much more, it will help you to simplify the code and eventually remove one line of code and 40 dependencies...(relying on a side framework misused such Spring or any other one).
I think most of you will know, programmers often reuse code from other software. I think, most of the time it is a good idea. But if you use code from another project your program depends on the other project.
I my current case I got three java projects A, B and C. Now A uses B and B uses C. I'm using eclipse IDE and added B to the buildpath of A and C to the buildpath of B. Now there is an compiler error that A can't resolve something from C. So I have to add C to the buildpath of B.
So what is the best way, to resolve the dependencies while keeping your programm as independent as possible from other projects?
I would like to know is in general and in reference to my current situation. Are there better ways to do this? I.e. there are classpath settings in the launch / debug configuration view, but I think they won't help at compile time.
Thanks in advance.
This sounds like part of the problem set fixed by Maven. Using Maven and Eclipse, namely m2eclipse, you can have projects use other projects and all the dependency resolution is handled for you.
It sounds to me like you're doing what you have to without incorporating a dependency management tool like Ivy or Maven, which provide you the capability of "transitive dependency management". With either of these tools you can just specify that A depends on B and B depends on C and they will automatically know that A is going to need C as well.
The advantages of Maven (this is what I have experience in) also comes into play when it's time to package your projects for deployment since it can easily gather all of those dependencies (all the way down the hierarchy) and place them together into a distribution folder or a fat JAR that contains all of your dependencies. It takes some reading and set-up time to get into a tool like Maven, but it does make the task of managing your dependencies a whole lot easier, especially as they grow.
We use Maven and it's essential for our projects. It's a good time for you to learn - dependencies on more than 3 projects can be frightening. Maven deals with versions so that if, for whatever reason, you have to depend on Foo.1.2.3 then Maven will ensure you don't get the wrong version.
However it's not trivial. If you use Netbeans it's built in better than Eclipse and may help you learn. (Also projects are fairly switcheable between the two systems).
Maven supports a lot of concept in its POM (pom.xml) file including licence info, contributors, arguments, etc. so you get a lot more than just dependency management. And it supports modularisation of projects.
Don't skip the learning curve - you need to know how it works. But you will also find previous SO questions that will help
Others have mentioned several of the good tools, maven probably being the most common. Ivy is another one that is more targeted at just dependency management. I personally use gradle which has some of the best of all of those features underneath a familiar groovy wrapper... that is still evolving and spottily documented. ;)
One thing to be aware of is how these tools handle transitive dependencies. In your example, C is a transitive dependency of A because A depends on B which depends on C. Some of these build tools will handle this type of dependency differently and it can surprise you when you least expect it.
For example, if A actually refers to code from C, ie: it has a compile-time dependency on C, then your A->B->C setup will work in something like Maven. On the other end, gradle will also make you declare that A depends on C... since it does. Runtime dependencies are fully resolved either way.
The surprise comes when you've been transitively including something for months and some of your code has relied on aspects of C and you decide you no longer need a B dependency. Suddenly your code won't build until you figure out you need a A->C dependency specified. In this example, that's pretty trivial to discover but sometimes it isn't.
And if talk like that makes your head swim a little and you don't plan on your project getting much more complicated... then you can probably just stick with what you are doing for a while. As others mentioned, it's the right way to do it without a tool helping you.
Use maven to manage your dependencies and then use the dependency plugin to see the dependencies.
you can run
mvn dependency:analyze
or
mvn dependency:tree -Dverbose=true
this will help you a lot.
no doubt you should use a dependency management tool as people have noted... manually though, archive B and C in B_C.jar. Test that B's dependence on C is resolved within the Jar.
Then add B_C.jar in the classpath...
Dependency management is a huge topic. Maven, Ivy and other tools have been developed to ease the pain with some success. Both of those tools create a dependency hierarchy so you don't run into the situation you described. They also have Eclipse plugins so that Eclipse will recognize that hierarchy.
To truly use these frameworks, you will have to change your current build process. Maven probably requires more of a commitment than Ivy, but neither is trivial, and understanding how to set it up takes some time. That said, it is very helpful to have your dependencies defined and managed clearly.
Free maven books:
http://www.sonatype.com/documentation/books
What would you suggest as a replacement to the Maven Java build toolset? Just plain Ant scripts? SCons?
It depends on what you use anyway. Maven 1? Maven 2? I find Maven 2 decently documented, and quite powerful, when used in combination with Ant tasks and some home baked Java plugins.
There are Ant tasks to use Maven features from Ant: http://maven.apache.org/ant-tasks/index.html . So you don't really need to write your own "library".
And you can call Ant from Maven: http://maven.apache.org/plugins/maven-antrun-plugin/ .
You can also script things using Groovy for example: http://groovy.codehaus.org/GMaven+-+Executing+Groovy+Code .
Or just write Java code for the custom bits you need: http://maven.apache.org/guides/plugin/guide-java-plugin-development.html .
There's no reason not to mix and match, when you can do it ;)
While I like the idea of Maven, I always find myself fighting it for anything but the simplest configurations. Its architecture is plugin-based, and the quality of the plugins varies widely. Documentation is sketchy at best; half the time I can't figure out if the doc I'm reading is for Maven 1 or 2. And how often have you had to resort to using the maven-ant-plugin to do something basic?
I've been using Apache Ivy for about a year now, and I'm pretty happy with it. There is a bit of a learning curve though. But there's an active mailing list, and the author is very responsive.
What I ended up doing was writing a "library" of a few simple Ant targets such as "resolve", "compile", "install", etc. that expect things to be in standard Maven places (such as src/main/java) and then <import> the targets into my various projects. This way you get the best of both worlds- Maven-style configuration, and all the power of Ant.
I'm fond of using Ant with Ivy. Ivy uses Maven repositories to fetch library dependencies without forcing you to change your entire build system to suit Maven.
I've been using buildr for some of my projects. Very terse and readable build scripts (no xml) and it uses maven repos (with trivial effort, maven 1 and maven 2).
I was looking for something post-maven1 to work on for some projects and m2 wasn't all that appealing. Maven had already left all m1 users high and dry with the need to redo all of their build systems (and some of mine cannot be expressed in m2 as far as I can tell).
I really like using ant4eclipse - you set up your project dependencies in eclipse as normal and you can write a single ant script that'll build all the projects in the right order.
For third-party libs, you can either set up a single eclipse project containing (and exporting) all of your jars [I DON'T RECOMMEND THIS], or a separate project for each third-party lib, exporting the jar. [I recommend the latter b/c you can track project->third-party deps and upgrade parts when needed.]
I would go with buildr, if i am expecting complex build targets in my build. The benefit is, you are not dependent on any particular library at all. You have all the APIs from Ruby to do whatever you want; unlike Ivy in which case you are still tied up to ANT. Effort of migration to Ivy is worth if you want to introduce library dependency to an already established ANT based build mechanism which does all the operations you want to do.
There are at least three different parts of Maven: the repository, the implementation independent lifecyle and the conventions for a default project. Using Ivy provides only the repository.
I've not used it but I think that easyant is designed to be a more complete replacement to Maven. It uses Ivy as the repository but then adds conventions and standard modules.
What are the possibilities to enforce restrictions on the package dependencies in a Java build system? For example, the myapp.server.bl.Customer class should not be allowed to refer to the myapp.client.ui.customlayout package.
I'm interested in either Ant-based or IDE-specific solutions.
I'd like to get an error message in the build indicating that a (custom) package dependency rule has been violated and the build aborted. I also would like to maintain the dependencies in a list, preferably in a text file, outside of the Ant scripts or IDE project files.
(I don't know Maven but I've read it here it has better support for module dependency management)
I believe Checkstyle has a check for that.
It's called Import Control
You can configure Eclipse projects to specify Access Rules. Access rules can specify "Forbidden", "Discouraged", and "Accessible" levels all with wildcard rules. You can then configure violations of either Discouraged or Forbidden to be flagged as either warnings or errors during builds.
Kind of an old article on the idea (details may be out of date):
http://www.eclipsezone.com/eclipse/forums/t53736.html
If you're using Eclipse (or OSGi) plugins, then the "public" parts of the plugin/module are explicitly defined and this is part of the model.
ivy seems like a good solution for your problem (if you are using ant). Ivy is the offical dependency management component of Ant and thus integrates nicely with ant. It is capable of resolving dependencies, handle conflicts, create exclusions and so on.
It uses a simple xml structure to describe the dependencies and is easier to use than Maven, because it only tries to address dependency resolution problems.
From the Ivy homepage:
Ivy is a tool for managing (recording, tracking, resolving and reporting) project dependencies. It is characterized by the following:
flexibility and configurability - Ivy is essentially process agnostic and is not tied to any methodology or structure. Instead it provides the necessary flexibility and configurability to be adapted to a broad range of dependency management and build processes.
tight integration with Apache Ant - while available as a standalone tool, Ivy works particularly well with Apache Ant providing a number of powerful Ant tasks ranging from dependency resolution to dependency reporting and publication.
For the IDE specific solutions, IntelliJ IDEA has a dependency analysis tool that allows one to define invalid dependencies as well.
http://www.jetbrains.com/idea/webhelp2/dependency-validation-dialog.html
The dependency violation will be shown both when compiling and live, while editing the dependent class (as error/warning stripes in the right side error bar).
Even more automation can be obtained with JetBrains' TeamCity build server, that can run inspection builds and report the above configured checks.
For another IDE independent solution, AspectJ can be used to declare invalid dependencies (and integrate the step in the build process, in order to obtain warning/error info for the issues).
Eclipse has support for this via Build Path properties / jar properties. I think it may only work across jar / project boundaries.
Maybe Classsycle can be used:
http://classycle.sourceforge.net/ddf.html
You can use multiple modules in IDEA or Maven or multiple projects in Eclipse and Gradle. The concept is the same in all cases.
A trivial interpretation would be a module for myapp.server.bl and another for myapp.client.ui.customlayout with no compile time dependencies between either of them. Now any attempt to compile code or code-complete against the opposite module/project will fail as desired.
To audit how extensive the problem already is, a useful starting point for IntelliJ IDEA is Analyzing Dependencies:
http://www.jetbrains.com/idea/webhelp/analyzing-dependencies.html
From that article you can see how to run and act on dependency analysis for your project.