I'm setting up a new Java project on GitHub, and I'll have some Apache Commons libraries as dependencies.
What are the best practices to establish that my project needs those jar files? Should I upload the dependencies to my GitHub repository (ugly)? Or use a Maven-like tool for that?
Or is there a way to link a file in another git repository? Apache provides git repositories for they libraries. They are read-only, but I'm o.k. with that, since I just want to use the jars. The bad thing is that they contain all the sources, and I just care about the compiled jar. It seems we can't git submodule just a file.
The two approaches are:
declarative and component-based, where you declare (describe) what components (jars, exe, other binaries) you need for your project to (compile, execute, deploy, etc.), and you use a third-party tool (like Maven/Nexus) to bring those components on demand.
So they aren't versioned in your repo. They are declared/described (like in a pom.xml, if you were to use Nexus)
See also "Difference between Git and Nexus?".
inclusive and system-based, where you complete your project with other project sources/binaries, in order to get in your repo everything you need right after the clone step (no need to call a third-party tool or to do anything: every other part of your system in there).
With Git, especially if those "other parts" are in a git repo (like the apache libs one), then you would declare those sub-repos as submodules of your main repo.
That way, all you keep in your main repo is a special entry (gitlink, mode 160000) referencing a specific SHA1 of another repo (but you can make that submodule follow a branch too, a bit like svn external).
And with sparse checkouts in submodules (as in this example), you even can update those modules for them to checkout only the part of the repo you want (like just the jars, not the sources).
Note that you aren't supposed to store any delivery that you would produce (like jars of your own) in your GitHub repo.
You can associate those deliveries to GitHub releases though.
Related
My team uses a GitHub.com organization to keep all of our source code in private repos. (Prior, our old workflow was emailing Dropbox links). Most of the time each repo is one separate project with no dependancy of any other (the only dependancies are on third-party open source libraries). Or if there is some dependancy, then the .java files have just been copy pasted into the other project.
I've recently been splitting up some of my code into reusable modules, but I don't know any way to do the dependancy management when I use the libraries I'm creating in another project.
I know with Gradle you can add a git repo like this:
gitRepository('https://github.com/user/project.git') {
producesModule('user:project')
}
but I don't know if there's a way to make it work with private repos, and I don't know if there's a way to specify versions.
My currently solution is to just build the library JAR, and keep track of the binary version with GitHub release tagging, and when I need to use the library in another project, I download the desired version of the JAR (typically the most recent) and add it to a local /lib/ folder in the other project and import the JAR into the module path as a local JAR. Of course I need to go through the whole process again manually if I want to make a change to the library.
I also heard you can set up private Gradle or Maven servers and some companies do that, but I guess that would mean migrating away from GitHub.com?
Is there any way to make this work (either Gradle or Maven, it doesn't matter) to manage dependancies between GitHub private repos?
Can someone tell me, what is the most sensible way (or ways) to solve this?
Thanks.
What you need is a very typical maven/gradle based setup where
each of your projects will be producing an artifact with a coordinate
of the form group:name:version
your projects do not have to be explicitly aware of each other. They depend on the artifacts produced by other projects. This is called binary dependency
for a project to locate a binary dependency, you will need a central registry where you can publish all your artifacts to. GitHub has a product called GitHub Package for precisely this purpose.
If you don't want to use GitHub Package yet, or your setup (number of projects, size of each projects, size your team) is small enough, you can locally checkout all the projects and include them into a gradle composite build so that binary dependencies will be substituted with local project dependencies. The good thing about the composite build is that when you decide to invest in a package registry, your build.gradle requires no change at all.
BTW, where you run your private package registry does not really matter. You can use the GitHub Package, or some other hosted services, or even run e.g. jfrog artifactory on your own server. It is completely unrelated to where you host your source code, so you dont need to migrate away from GitHub in any case.
We have a git project that has some 3rd party jars which are not available in any Maven repo and are in a "lib" folder in the project. I need to include them for compiling, building and then package them into the WAR in WEB-INF/lib.
I cannot add them as a local maven repo from the command line because this projects needs to be buildable for anyone cloning the repo without requiring them to run extra commands (I have no way around this requirement).
I saw some people suggesting System scope but that then Maven won't package them into your WAR:
<dependency>
<groupId>com.roufid.tutorials</groupId>
<artifactId>example-app</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${basedir}/lib/yourJar.jar</systemPath>
</dependency>
How do I get these jars to be used for compiling/inspection, building and then packaged into the WAR?
You can use :
System scope but make the packaging yourself via assembly plugin
A maven repo along with your project (i.e. maven repo on-the-fly, basically same as local repo but without having a extra moving part to worry about because this repo follows your project).
For Maven repo on-the-fly option, you can do as described here (which is, take any already existing Maven repo, which already contains your needed jars, such as your local one, put it in your project, and then reference this repo from your project using relative paths).
I'll assume you've verified that whatever mechanism you might use to distribute these jars would be in compliance with the relevant licenses. If it would, then it seems there would be little reason for the jars' creators not to provide for official Maven distribution, so your best option might be to lobby for them to do that. But if not, and yet for some reason they'll allow for you distributing the jar (either through cloning of your repo, or via a separate Maven repo you maintain):
There are several ways. I give preference to approaches that don't put the jars in the git repo.
Publish a Maven Repo
So it's possible to host a public-facing repo and serve the artifacts that way. The pom can add your public-facing repo to the build, so that those who clone can build without having to run any special commands.
Running your own repo isn't terribly difficult. The OSS versions of Nexus or Artifactory jFrog would probably be perfectly capable.
But, if we're assuming the authors' refusal to publish their own jars via Maven means they don't want them distributed that way, then there's no reason to spend much time on the details of this option. So moving on...
Distribution in the Git Repo
I guess this is what you're doing, though again if Maven distribution violates the license I'd make sure you're splitting hairs the right way in thinking that this doesn't.
So the question would be how to get Maven to deal with the artifacts distributed in this way, and again there are some options.
Your objection to putting the jars in the local repo is that it would require extra commands of the user; but actually this could be automated in the "validate" phase of the build. Binding install:install-file to the validate phase should work.
Alternately, your objection to using system scope is that the file isn't copied into the final war. You might be able to use the dependency plugin to force the issue, but I'm not sure of that. What I am sure of is you could treat the directory containing the jars as a web resource with suitable configuration in the war plugin. (You'd want it to be treated as unfiltered and to map to the WEB-INF/lib folder.)
In any case, if you distribute jars (or other large binaries) in the git repo, I strongly recommend you look at git lfs. This will require one-time configuration by each of your users, but it will prevent your repo from gradually becoming bloated and unusable.
Use forward slash (/) to backslash () in the systemPath.
<dependency>
<groupId>com.roufid.tutorials</groupId>`enter code here`
<artifactId>example-app</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${basedir}\lib\yourJar.jar</systemPath>
I am using eclipse BndTools with a few dedicated workspaces each stored in a single git repo and I've been quite happy sofar.
I've been sharing projects between workspaces by copying them. But recently decided to pull common code into a shared code git repository. In eclipse this is trivial, just use subfolders in your workspace, one per repository.
However to my surprise bndtools demands that I place one cnf project next to my projects in the filebase. At the same time I can only have one cnf project in my workspace. Which effectively means ALL my projects should be peers.
Which in turn means I cannot use multiple git repositories as they cannot share the same directory. Unless I split each project into it's own repository and with 50+ projects this is clearly not where I want to go.
I know eclipse can do this, but is there a way to get bndtools to play ball?
Which effectively means ALL my projects should be peers.
...
Which in turn means I cannot use multiple git repositories as they cannot share the same directory. Unless I split each project into it's own repository
This is where submodule is coming for rescue.
Submodules allow foreign repositories to be embedded within a dedicated subdirectory of the source tree, always pointed at a particular commit.
How to use submodules
# Create each project in its own repository
# now add the desired submodule to your project
git submodule add <url>
# now init/update one by one or recursively all at once
git submodule init
git submodule update
I am a new IntelliJ user (I've used Eclipse for many years). I'm having a hard time wrapping my head around the-project-is-a-module concept in IntelliJ, since this does not hold true in Eclipse. The main issue I'm having is that I am using my top level package as the project in IntelliJ. I would like this top level package to be in a git repo. I would also like all the dependencies of this package to be in their own respetive git repos. When I check these packages out into my project, a do git status on the top level package, all of the dependencies show up in the untracked files. This behavior seems incorrect to me. How can I fix it?
Thanks!
Edit:
To summarize the clarifications in the comments:
I would like to support hundreds of libraries any of which could change at a time. The dependency graph will also be frequently changing. For this reason, having one git repo or constantly updating .gitignore files is not maintainable.
Currently, I'm using Maven to manage dependencies but I'm open to using whatever is best suited for this job.
Finally, I would like to check out any library into my workspace and modify it and, if possible, have Intellij reflect my local changes when running code as if my local code were already built into the dependency graph. A type of local override if you will.
IntelliJ's directory structure places all of the modules in their parent project's directory. If you are developing libraries which are shared between several other projects, importing the library as a module is probably not the correct solution. Instead, you should treat each library as its own independent project and make "releases" using a build tool such as gradle or maven. Then your projects can treat the libraries the same way they do third-party libraries and use the build tool to import the library.
I have multiple projects, some are single module, and others are multi-module. Some of these modules are libraries intended to be used in multiple projects.
How do I share these modules between projects, and what would I store in version control?
If I store each module in it's own repository, then the .idea folders will not be available, and as such information about the module such as its dependencies, copyright notices, artefact config etc will be missing. Therefore, I would like to store each project in a repository, but then how would I reference one module from another project as a dependency?
Note: I am using git for version control.
How do I share these modules between projects.
I would either have a project for common modules, or add them to each project.
and what would I store in version control?
I wouldn't store IntelliJ specific files if you are working with another developer. If not, you can just store everything. What ever you do, you can swap later.
If I store each module in it's own repository,
Or in its own sub-directory. Git does encourage creating more repositories, either way works.
then the .idea folders will not be available,
I would check them in anyway, but if you check them in, they are as available as any other file.
and as such information about the module such as its dependencies, copyright notices, artefact config etc will be missing.
I would use maven for dependencies and build.
Therefore, I would like to store each project in a repository, but then how would I reference one module from another project as a dependency?
You can import a module from one project to another in IntelliJ, but I would use maven which supports this.