Develop Reference

Java Dependency Management For Large Projects

I hope I can keep this question specific enough, my team at work is currently debating the best way to manage our dependencies for a huge project (150+ dependencies ~300mb).
We have two main problems
Keeping all the developers dependencies the same so we are compiling against the same files
Ensure the project (once compiled) is comliped against the same dependencies
The two ideas that have been suggested are using a BirJar (all dependencies in one file) and just adding a version number to it and using a shared folder and pointing everyone's machines at the same place.
Or making including all the dependencies in the jar when we compile it (a jar, of jars, of jars) and just have a project that "has no dependencies"
Someone also mentioned setting up an internal version of Ivy and pointing all the code to pull dependencies from there.
What are the best practices regarding massive dependency management?

Why don't you use Maven and its dependency management ?
You can specify each dependency, its particular version and its scope (compile-time, for testing, for deployment etc.). You can provide a master pom.xml (the config file) that specifies these, and developers can override if they need (say, to evaluate new versions).
e.g. I specify a pom.xml that details the particular jars I require and their versions (or range). Dependent jars are determined/downloaded automatically. I can nominate which of these jars are used for compilation vs. deployment etc. If I use a centralised repository such as Nexus I can then build my artefact (e.g. a library) and deploy that into Nexus, and it'll become available for other developers to download in exactly the same manner as 3rd party libs etc.

Incase you dont like/want to follow the Maven project structure...
If you already use Ant, then your best bet is to use Ivy for dependency management.
http://ant.apache.org/ivy/
It provides a rich set of ant tasks for dependency manipulation.
from : Ant dependency management

Managing maven dependancies - New Versions and Non-Repo libraries

Warning: I have just picked up Maven, so things mentioned might be wrong or not best practice.
I have a medium size open source project that I am migrating to Maven from the basic
NetBeans project management. This is not a developer team sharing the same room, this is 1-5 people over the internet sharing a SVN repo. Reading over the how-tos on dependencies, it seems that the only way to get dependencies is to get them from an online repo or install them locally.
This is not what I was looking for. I want to keep all dependencies in the SVN for many reasons including portability (anybody can pass by, check out the repo, build, and use; all that simply without manual adding to local repo's and whatnot), getting newer versions (discussed below), and manual versioning.
The other issue I have with the maven repository is that they are quite behind in versions. Logback for example is 0.9.18 in mvnbrowser but 0.9.24 officially. PircBot is 1.4.6 in mvnbrowser but 1.5.0 officially. Why such old versions?
Issue 3 is that I have dependencies that don't even exist in the repos, like Easier Java Persistence.
So
How can I force all dependencies to come from /lib for example
On a related note, can mvn build from library's SVN repo directly? Just curious
Is there an automatic way to get the newest version directly from a dependencies site/svn repo if they also use Maven? IE libraries like commons-lang or logback
Is there a better way of managing dependencies? (IE Ivy or some weird POM option I'm missing)
FYI, this is a Java project with 3 modules, project global dependencies and module specific dependencies.
Bonus points if it can work with the bundled version of Maven that comes with NetBeans.
Not a duplicate of
Maven: add a dependency to a jar by relative path - Not wanting to install to local repository
maven compile fails because i have a non-maven jar - Don't think a System dependency is the right answer
maven look for new versions of dependencies - Still uses(?) repository, just the latest (old) version

This is not what I was looking for. I want to keep all dependencies in the SVN for many reasons (...)
I will come back on this but the solution I described in Maven: add a dependency to a jar by relative path (using a file-based repository) allows to implement such a solution.
The other issue I have with the maven repository is that they are quite behind in versions. Logback for example is 0.9.18 in mvnbrowser but 0.9.24 officially. PircBot is 1.4.6 in mvnbrowser but 1.5.0 officially. Why such old versions?
It looks like mvnbrowser indices are totally out of date (making it useless as repository search engine) because the maven central repository does have logback-core-0.9.24.jar (the logback project is doing what has to be done to make this happen) but only has an old pircbot-1.4.2.jar. Why? Ask the pircbot team. Anyway, you're right, the central repository might not always have ultimate versions.
Issue 3 is that I have dependencies that don't even exist in the repos, like Easier Java Persistence.
Yeah, this happens too.
How can I force all dependencies to come from /lib for example
As previously hinted, you should re-read carefully the solution suggested in Maven: add a dependency to a jar by relative path. This solution is not about installing libraries to the local repository but is about using a file-based repository (that could thus be stored in SVN). You might have missed the point, this matches your use case. And also check Brett's answer for a variation.
On a related note, can mvn build from library's SVN repo directly? Just curious
Didn't get that one. Can you clarify?
Is there an automatic way to get the newest version directly from a dependencies site/svn repo if they also use Maven? IE libraries like commons-lang or logback
Maven supports version ranges and you could use a syntax allowing to use "any version greater than X". But I do NOT recommend using version ranges at all, for the sake of build reproducibility. You don't want the build to suddenly fail because of some automatic update that happened on your back. Only upgrade if you need bug fixes or new features, but do it explicitly (if it ain't broke, don't fix it).
You might also find mentions of the LATEST and RELEASE version markers. I don't recommend them neither for the same reasons as above and even less since they're removed from Maven 3.x.
Is there a better way of managing dependencies? (IE Ivy or some weird POM option I'm missing)
Can't say for Ivy. But in the Maven land, if you can't host up a "corporate" repository for your project (Nexus, Archiva, Artifactory), then the file-based repository is IMO the best approach.

Setup your own Maven repository.
http://archiva.apache.org/

Is there a standard place to store Spring library jar files?

I've downloaded Spring 3.0.2 with dependencies and found that it contains 405 jar files. I usually keep third party libraries in a "lib" subdirectory, but there are so many Spring jars that it seems sensible to keep them separately so that they don't swamp the other libraries and to simplify version upgrades.
I suspect that I want to keep the full set of libraries in Subversion, but only deploy the subset that is actually used.
Do Spring users have a standard way to deal with this problem?

The vast majority of the "dependencies" are unnecessary, it really is a "kitchen sink" distribution. I would suggest just putting the Spring JARs themselves into lib, and only add the others as and when you need them.
In fact, you can pick and choose which Spring JARS you need - it's split up into several, so that you can pick the appropriate ones. There should be a readme file in the distribution describing which JARs you need, and what they depend on.

If you insist on using Ant, you can use its companion, Ivy, for dependency management. Personally, I have been a fairly happy Maven user for years.

If you build using Maven, you can specify that you require particular Spring libraries. Maven will download these and their declared dependencies into your local repository, and package those jars required into your final solution. You don't need to declare anything other than your top-level dependency on Spring.

Specifiy classpath for maven

Quite new to maven here so let me explain first what I am trying to do:
We have certain JAR files which will not be added to the repo. This is because they are specific to Oracle ADF and are already placed on our application server. There is only 1 version to be used for all apps at anyone time. In order to compile though, we need to have these on the class path. There are a LOT of these JARS, so if we were to upgrade to a newer version of ADF, we would have to go into every application and redefine some pretty redundant dependencies. So again, my goal is to just add these JARs to the classpath, since we will control what version is actually used elsewhere.
So basically, I want to just add every JAR in a given network directory (of which devs do not have permission to modify) to maven's classpath for when it compiles. And without putting any of these JAR files in a repository. And of course, these JARs are not to be packaged into any EAR/WAR.
edit:
Amongst other reasons why I do not want to add these to the corporate repo is that:
These JARs are not used by anything else. There are a lot of them, uncommon and exclusive to Oracle.
There will only be one version of a given JAR used at anyone time. There will never be the case where Application A depends on 1.0 and Application B depends on 1.1. Both App A and B will depend on either 1.1 or 1.2 solely.
We are planning to maintain 100+ applications. That is a lot of pom.xml files, meaning anytime we upgrade Oracle ADF, if any dependency wasn't correctly specified (via human error) we will have to fix each mistake every time we edit those 100+ pom.xml files for an upgrade.

I see three options:
Put the dependencies in a repository (could be a file repository as described in this answer) and declare them with a scope provided.
Use the dirty system scope trick (i.e. declare the dependencies with a system scope and set the path to the jars in your file system.
Little variation of #2: create a jar with a MANIFEST.MF referencing all the jars (using a relative path) and declare a dependency on this almost empty jar with a system scope.
The clean way is option #1 but others would work too in your case. Option #3 seems be the closest to what you're looking for.
Update: To clarify option #3
Let's say you have a directory with a.jar and b.jar. Create a c.jar with a Class-Path entry in its META-INF/MANIFEST.MF listing other jars, something like this:
Class-Path: ./a.jar ./b.jar
Then declare a dependency in your POM on c (and only on c) with a system scope, other jars will become "visible" without having to explicitly list them in your POM (sure, you need to declare them in the manifest but this can be very easily scripted).

Although you explicitly stated you don't want them in the repository, your reasons are not justified. Here's my suggestion:
install these jars in your repostory
add them as maven dependencies, with <scope>provided</scope>. This means that they are provided by your runtime (the application server) and will not be included in your artifacts (war/ear)
Check this similar question
It is advisable that an organization that's using maven extensively has its own repository. You can see Nexus. Then you can install these jars in your repository and all developers will use them, rather than having the jars in each local repository only.
(The "ugliest" option would be not to use maven at all, put put the jars on a relative location and add them to the classpath of the project, submitting the classpath properties file (depending on the IDE))

if you are developing ADF (10g / 11g I guess) components, I suppose you'll be using JDeveloper as IDE. JDeveloper comes with a very rich Library Management Tool that allows you to define which libaries are required for compiling or which ones should be packaged for deployment. I I suppose you will already know how to add libraries to projects and indicate in the deployment profile which ones should be picked while packaging. If you want to keep your libraries out of maven, maybe this could be the best approach. Let´s say the libraries you refer too are the "Webcenter" ones, using this approach will guarantee you you have the adequate libraries as JDeveloper will come with the right version libraries.
Nevertheless, as you are using maven I would not recommend to keep some libraries out of control and maven repositories. I´d recommend choose between maven and Oracle JDeveloper library management. In our current project we are working with JDeveloper ADF 11g (and WebCenter) and we use maven, it simply make us library management easier. At the end of the day, we will have a big amount of third party libraries (say Apache, Spring, etc.) that are useful to be managed by maven and not so many Oracle libraries really required for compiling in the IDE (as you would only need the API ones and not their implementations). Our approach has been to add the Oracle libraries to our maven repository whenever they are required and let maven to control the whole dependency management.
As others say in their answers if you don´t want the dependencies to be included in any of your artifacts use <scope>provided</scope>. Once you configure your development environment you will be grateful maven does the work and you can (almost) forget about dependency management. To build the JDeveloper IDE files we are using the maven jdev plugin, so mvn jdev:jdev would build generate our project files and set up dependencies on libraries and among them to compile properly.
Updated:
Of course, you need to refer to ADF libraries in your pom files. In our project we just refer to the ones used on each application, say ADF Tag Libraries or a specific service, not the whole ADF/WebCenter stack. For this purpose use the "provided" scope. You can still let JDeveloper to manage your libraries, but we have found that it's simpler to either have a 100% JDeveloper libraries approach or a 100% maven approach. If you go with the maven approach it will take you some time to build your local repo at first, but once that's done it's very easy to maintain, and the whole cycle (development, build, test, packaging and deployment) will be simpler, having a more consistent configuration. It's true that in a future you'll have to update to later ADF versions, but as your repository structure will already be defined it should be something fast. For future upgrades I'd recommend to keep the ADF version as a property on the top pom, that will allow you to switch faster to a new version.

Should libraries (jar) go into the repository?

When you start to use a third party library in Java, you add their jars to your project. Do you also add the jars to the repository or do you just keep a local installation. And if the latter, how do you sync between team members to be able to work?
Thanks.

Yes. You should add to the repository whatever is required for a developer on a clean system (aside from having the JDK and ant installed) to check out and build the project.

No, if you use Maven. Put them into Maven repository (if they are not there yet, most open source libraries are in public Maven repositories already).
Yes, if you use Ant.

If you're using ANT and you want maven style dependency management without maven, take a look at Ivy. It can download from maven repositories and can even read maven pom files.

Yes, third-party libraries should be version controlled since you will want everyone to compile against the same version of that library. This way when updates to that third-party library happen you will simply have to change it in one place and everyone will update to the new version. Otherwise you could easily end up with the famous: "It compiles on my machine!"-syndrome.

Assuming you do not use Maven, then you should definitely check in versions of 3rd party jars. You (or more importantly, the maintenance developer coming along after you) should be able to pull a version of your application from the repository and have all the correct versions of all the 3rd party jars required at compile- and runtime.
You also should maintain the versioning history of the 3rd party jars, again to help the maintenance developer out. Checking them in to the repository is the easiest and most effective way to do this.

I would recommend versioning everything you need to build a project. Unless your using a build tool like maven.

Short Answer
No
Longer answer
In the interim, use a folder on your network separate from the repo.
A Long Term Solution
Use maven. It was built for handling your build, configuration and external jar dependencies.
Go here for documentation though. (The official maven documentation is known to be pretty spotty).

Lately I've been keeping JARs out of SVN. I have a separate repository that I use to pull them into projects, manage dependencies, and keep track of different versions.
This is what Maven is born for. I've got to learn how to use it well, but until then I'm keeping my own JAR repository.