Develop Reference

Java dependancy management on GitHub private repos?

My team uses a GitHub.com organization to keep all of our source code in private repos. (Prior, our old workflow was emailing Dropbox links). Most of the time each repo is one separate project with no dependancy of any other (the only dependancies are on third-party open source libraries). Or if there is some dependancy, then the .java files have just been copy pasted into the other project.
I've recently been splitting up some of my code into reusable modules, but I don't know any way to do the dependancy management when I use the libraries I'm creating in another project.
I know with Gradle you can add a git repo like this:
gitRepository('https://github.com/user/project.git') {
producesModule('user:project')
}
but I don't know if there's a way to make it work with private repos, and I don't know if there's a way to specify versions.
My currently solution is to just build the library JAR, and keep track of the binary version with GitHub release tagging, and when I need to use the library in another project, I download the desired version of the JAR (typically the most recent) and add it to a local /lib/ folder in the other project and import the JAR into the module path as a local JAR. Of course I need to go through the whole process again manually if I want to make a change to the library.
I also heard you can set up private Gradle or Maven servers and some companies do that, but I guess that would mean migrating away from GitHub.com?
Is there any way to make this work (either Gradle or Maven, it doesn't matter) to manage dependancies between GitHub private repos?
Can someone tell me, what is the most sensible way (or ways) to solve this?
Thanks.

What you need is a very typical maven/gradle based setup where
each of your projects will be producing an artifact with a coordinate
of the form group:name:version
your projects do not have to be explicitly aware of each other. They depend on the artifacts produced by other projects. This is called binary dependency
for a project to locate a binary dependency, you will need a central registry where you can publish all your artifacts to. GitHub has a product called GitHub Package for precisely this purpose.
If you don't want to use GitHub Package yet, or your setup (number of projects, size of each projects, size your team) is small enough, you can locally checkout all the projects and include them into a gradle composite build so that binary dependencies will be substituted with local project dependencies. The good thing about the composite build is that when you decide to invest in a package registry, your build.gradle requires no change at all.
BTW, where you run your private package registry does not really matter. You can use the GitHub Package, or some other hosted services, or even run e.g. jfrog artifactory on your own server. It is completely unrelated to where you host your source code, so you dont need to migrate away from GitHub in any case.

How to develop a library and a program that uses it using Gradle

I have a library and a program, both under my control and built using Gradle. What's the best way to develop these two at the same time?
I have set up a private maven repository to distribute the library and that's working, but I don't want to release to that repository every little experiment I make during development. It's slow and disruptive to users of the library.
I tried installing the jar to the local maven repository as explained here: Gradle alternate to mvn install but the project that's using the library is not picking up that newly installed version.

I think, you can try to use multi-project builds for that if it's possible. But you will likely need to restructure both your current projects to become modules of the same new project.

What's the best way to develop these two at the same time?
It depends by how the team is organized and what are your policies.
Of course if the team can use a git repo and access to the source code you can just use git without pushing a new version on the maven server for each commit or push.
Otherwise if other users can only use the final library, you have to push the version on the maven server.
I have set up a private repository to distribute the library and that's working, but I don't want to release to that repository every little experiment I make during development. It's slow and disruptive to users of the library.
Every maven repo has 2 different repositories:
release
snapshot
Usually release repo is used only for stable releases and the snapshot repo is used to publish little change, beta release and so on.
In any case it is not required that every changes in the code is pushed in the maven repo (it is a your choice)
It's slow
The time to upload artifacts usually is not so big, in any case you can evaluate to push the release in the maven repo with a CI server.

The best method seems to be to make one project include the other one when present by adding:
if (file("../libraryproject").exists()) {
includeBuild "../libraryproject"
}
to the settings.gradle file of the project that uses the library. That can be committed to the source code repo because when that directory doesn't exist, the dependency will be included in the traditional way.
This is called Composite Build in the Gradle world, IntelliJ seems to handle properly and theres'a recorded webcast showing the whole setup: https://www.youtube.com/watch?v=grPJanXfRPg

What is Maven in Android Studio?

There are lot of Answers on how to use Maven in Android and work with it.
However, As Gradle being already available, which manages the build process of Android projects and support complex scenarios in creating Android applications, such as Multi-distribution and Multi-apk,
What exactly is the role of Maven in Android Studio?
And how is it different from Gradle?

The short answer is: The tools do the same thing but in different ways. The difference is how you can use them to build your project. Apache Maven goes for "follow our convention" way and Gradle gives you flexibility. Some devs don't want a new tool or don't have time to learn how to use it properly.
Now the long answer...
To understand properly why some devs are attached to Apache Maven, we have to look at some years in the past.
Apache Maven is a build automation tool just like Gradle.
Maven got released on 2004. Gradle first version got out on 2007 but it did not become as popular as Maven in the early days.
Some devs don’t like having to learn anything new and most companies don't want to risk the exchange of an already running and mastered tool (Maven) for the new kid on the block.
Gradle became popular when the Android development raised. Android's project has a different project structure from Java EE/Web projects. Trying to use Maven on Android's project just feels unnatural, the tool was not prepared to provide flexibility.
Until now, Apache Maven can be used on simple Java EE projects without being a pain in the ass if you already know the forced conventions. So devs that aren't aware of Gradle/did not get in touch with Android apps don't have motivation to change to another tool.
In my opinion, Gradle is better than Apache Maven in every way that you could imagine. It gives you flexibility and tries not to get in your way, a "feature" that you cannot find in Apache Maven. If you don't follow Apache Maven life cycle, your build will fail, that's it.
For example, your code can be spread across many directories in any kind of layout if you are using Gradle. If you are on Maven and you don't follow the 'convention', you'll lose some hours changing your pom (Maven's build file) to be able to understand and handle your folder structure.
E.g.: Java source code must be on src/main/java folder. If you got an old project and the structure is src/java, sorry, You have no guarantee that all Maven plugins will run as expected.
Gradle does us a favor and puts more features for comparison in a very organized chart. Take a look. https://gradle.org/maven_vs_gradle/

Maven is just a tool that manages and simplifies how you build your project. Among many other things (running tests, managing conflicts, documentation, modularization), its most useful purpose is that of automatically fetching dependencies that your project needs and dependencies of those dependencies (transitive dependencies) if any. Dependencies are usually just JAR files that contain re-usable code.
So what does this mean. Suppose you want to use a library such as OpenCSV to generate a CSV file in your application. Non-Maven way: Google search for the library, check if it needs other libraries and if so download them, put them in your project's classpath then build. When you move your project to another PC, copy all the JAR's or it won't work. Maven way: insert something like this in a file named pom.xml:
<dependency>
<groupId>net.sf.opencsv</groupId>
<artifactId>opencsv</artifactId>
<version>2.3</version>
</dependency>
Done. Maven will do the rest for you when you build (download to local cache if not already existing, fetch dependencies, javadoc etc). Even if you were to copy your project to another PC without the JAR's, delete the JAR's etc, if Maven is present it will re-download them automatically when you build.
For more: Link

Maven: download file and run command to build dependency

Recently, a project I need as a dependency for some of my programs has switched to providing patch files instead of actual jars (legal reasons).
They included a small tool to automatically patch your existing jars with the new update you downloaded.
I could write a small program that automatically downloads and patches the file, all I need is a way to tell maven not to download the file but to run a command and then use the file from my local repo.
Is there a way to do this (maybe using a custom plugin)?

You could write a small ant task run the tool and call this task from your maven build using maven-antrun plugin.
You could also write your own maven plugin but for such a simle task I would not recommend it.

The reason why your IDE does not find the dependency is that it is not published anymore (not the version you seek apparently).
Building the artifact and installing it locally as #Joe suggested is an elegant solution to solve your issue from my perspective.
The advantage is twofold:
Your IDE can seamlessly find the missing artifact
You can keep multiple version of that dependency (one for every new patch) and thus reproduce previous (working) build in case a new patch breaks something.
if you have a local repository, it is even better as this would maven the dependency available to all your team.
Of course you would still need a tool to download new patch, apply them and deploy the produced artifact to the repository automatically. But doing this as a separate task give you more flexibility concerning the tools you could use.

Best way to automatically check out and compile Eclipse projects with Ant in Hudson or another CI tool?

We have several products which have a lot of shared code and which must be maintained several versions back.
To handle this we use a lot of Eclipse projects, some contain library jars, and some contain shared source code (in several projects to avoid getting a giant heap with numerous dependencies while being able to compile everything from scratch to ensure that source and binaries are consistent). We manage those with projectSet.psf's as these can directly pull all projects out from CVS and leave a fully prepared workspace. We do not do ant builds directly or use maven.
We now want to be able to put all these projects and their various versions in a Continous Integration tool - I like Hudson but this is just a matter of taste - which essentially means that we need to get an automatic way to check out the projects to a fresh workspace, and compile the source folders as described in the project-files in each project. Hudson does not provide such an approach to build a project, so I have been considering what the best way to approach this would be.
Ideas have been
Find or write an ant plugin/converter that understands projectSet.psf's and map to cvs-checkout and compile tags.
Create the build.xml files from within Eclipse and use those. I tried this, and found the result to be verbose and with absolute locations which is not good with automatic tools putting files where they want to.
Write a Hudson plugin which understands projectSet.psf's to derive a configuration and build it.
Just bite the bullet and manually create and update the CI configuration whenever stuff breaks - I don't like this :)
I'd really like to hear about other peoples experiences so I can decide how to approach this.
Edit: Another option might be using a CI which knows better about Eclipse projects and/or project sets. We are not religious - this is just a matter of getting stuff running without having to do everything ourselves. Would Cruise Control be a better option perhaps? Others?
Edit: Found that ant4eclipse has a "Team Project Set" facility. http://ant4eclipse.sourceforge.net/
Edit: Used the ant4eclipse and ant-contrib ant extensions to build a complete workspace as a sjgned runnable jar file similar to the Runnable Jar facility in Eclipse 3.5M6. I am still depending on Eclipse to create the initial empty workspace, and extract the ProjectSet, so that is the next hurdle.
Edit: Ended up with a dual configuration, namely that Hudson extracts the same set of modules as listed in the ProjectSet.pdf file from CVS (which needs to have the same tag) causing them to be located next to each other. Then ant4eclipse works well with the projectSet.psf file embedded in the main module. Caveat: Module list in Hudson must be manually updated, and it appears that a manual workspace cleanup is needed afterwards to let Hudson "discover" that there is more projects now than earlier. This has now worked well for us for a couple of months, but it was quite tedious to get everything working inside the ant file.
Edit: The "Use Team Projects" with ant4eclipse and a Ctrl-A, Ctrl-C in Project Panel with a Ctrl-V in the CVS projects in Hudson has turned out to work well enough for us to live with (for mature projects this is very rarely changed). I am awaiting the release of ant4eclipse 1.0 - http://www.ant4eclipse.org/, currently in milestone 2 - to see how much homegrown functionality can be replaced with ant4eclipse things.
Edit: ant4eclipse is as of 20100609 in M4 so the schedule at http://www.ant4eclipse.org/node?page=1 is slipping somewhat.
Edit: My conclusion after using our ant4eclipse approach for a longer period is that the build script get very gnarly and is hard to maintain. Also the Team ProjectSet facility (which ant4eclipse use to locate the projects) which works well for CVS based repositories, but not after we migrated to git (which is a big thing in itself). New projects will most likely be based on maven, as this has good support in Jenkins.

I'm not completely sure I understand the problem, but it sounds like the root issue is that you have many projects, some of which are dependent on others. Some of the projects that are closer to the "leaf" of the dependency tree need to be able to use "stable" (or previously "released") versions of the more "core" projects.
I solve exactly this problem using Hudson, ant, and ivy. I follow a pattern demonstrated by Clark in Pragmatic Project Automation (he doesn't demonstrate the dependency problems and solutions, and he uses CruiseControl rather than hudson.)
I have a hand-written ant build file (we call it "cc-build.xml", because of our CruiseControl roots.) This file is responsible for refreshing the working space for the project from the CM repository and labeling the contents for future reference. It then hands off control to another hand-written ant build file (build.xml) that is provided by each project's developers. This project is responsible for the traditional build steps (compile, packaging, etc.) It is required to spit out the installable artifacts, unit test reports, etc, to the Hudson artifacts directory. It is my experience that automatically generated build files (by Eclipse or other similar IDE's) will never get close to getting this sufficiently robust for use in a CI scenario.
Additionally, it uses ivy to resolve its own dependencies. Ivy supports precisely-specified dependency versions (e.g. "use version 1.1") and it supports "fuzzy versions" (e.g. "use version 1.1+" or "use the latest version in integration status.") Our projects typically start out specifying a very "fuzzy" version for internal projects under ongoing development, and as they get close to a release point, they "freeze" the dependency version so that stuff stops moving underneath them.
The non-leaf projects (projects that are dependents for other projects) also use ivy to publish their artifacts to our internal ivy repository. That repository keeps all past builds of the dependents, so that any project can always depend on any other previous version.
Lastly, each project in Hudson is configured to have a build trigger that causes a rebuild when any of its dependent projects successfully build. This causes them to get built again with the (possibly) new ivy dependent version.
It is worth noting that once you get this up and running, consistent automated "labeling" or "tagging" of an automated build's inputs is going to be critical for you - otherwise troubleshooting post-build problems is going to result in having to untangle a hornet's nest to find the original source.
Getting all of this setup for our environment took quite a bit of effort (primarily in setting up the ivy repository and ant build files,) but it has paid for itself many times over in saved headaches in manually managing the dependencies and decreased troubleshooting effort.

Write a Hudson plugin which
understands projectSet.psf's to derive
a configuration and build it.
That seems like the winning answer to me.
I work with CruiseControl rather than Hudson but in my experience if you can create a plugin that solves your problem it will quickly payoff. And it is generally pretty easy to write a plugin that is custom fit for your solution as opposed to one that needs to work for everyone in a similar situation.

I have tried both Cruise Control (CC) and Hudson for our CI solution. We (as a company) decided on Hudson. But for your question "Does CC support Eclipse project build" the answer is no as far as I know. CC supports many more different build tools and Source Control systems but it is a bit more difficult to configure and use. As for Hudson, it is more simple to configure and use it. We developed our custom plugins for both CC and Hudson for the parts of our build cycle that they do not provide as is. As for plugin development, if you know / use Maven, Hudson is simpler too. But if you are not familiar to Maven, first you need to learn the basic usage of maven to successfully develop a Hudson plugin. But once you understand the basic usage of maven, plugin development, test and even debug is simpler in Hudson.
For your specific problem, I can think of a solution that makes use of Eclipse plugins as well. You can develop your own Eclipse plugin that for instance gets the psf files from a (configurable) folder, and use Eclipse internals to process these psf's. I mean you can use existing Eclipse source codes that takes a psf file, check-outs it's project definitions and compile these projects. This Eclipse plugin of yours may have a preference page (which you can access by Eclipse -> Window -> Preferences) and configure which folder it will use to look for psf files. Your Eclipse plugin should also have a way to start psf processing without user interaction. For this, you can use ipc to trigger your process. I mean your Eclipse plugin can listen for a port, and you can write another java application that will connect to your plugin through this port and trigger its process. As for CI part, you can use either CC or Hudson and use their external process execution support. If you are using Windows, you can write a bat file (for Linux sh file) that first launchs Eclipse that has your plugin installed. Then it launches your java application that will communicate with your Eclipse plugin to trigger your process. From your CI tool you will need to run your bat / sh file to trigger your process.