On the development shop I work for, we have an internal MAVEN repository, to keep our libraries (proprietary & open-souce). A common problem that we face is that, sometimes, the open-source libraries in our local MAVEN repository gets obsolete. Is there an automatic way to keep all the open-source libraries I use in my MAVEN repository always updated? What do you suggest to resolve this issue?
Archiva has been mentioned, but nexus seems more popular. Both have been designed to solve problems like the one you're having
Assuming you:
Don't want to download everything
Don't want to run another server
process
Only want to track a limited number
of projects
You might want to create a separate pom.xml file with dependencies like this:
<dependency>
<groupId>org.openfoo</groupId>
<artifactId>jfoo</artifactId>
<version>[1.0.0,2.0.0)</version>
</dependency>
This will tell maven to use jfoo 1.0.0 up to jfoo 2.0.0 so when jfoo releases version 1.2.17, you'll be fetching that in the next build assuming your settings are set to check versions each time.
This pom doesn't have to actually build anything. Just list those things you want to track.
Running:
cd the-path-to-the-project; mvn -q -B -U package
Via cron once a day will update all the dependencies in that pom and only report when there is a problem
BTW, this is a hack. If the number of developers is > 3 and you have the resources to run nexus, don't bother with the hack.
Take a look at Apache Archiva, a repository manager for Maven.
Related
Unfortunately the maven initial build is very slow due to artifacts downloading
f.e. I've tried to download the same jar using curl - it is 3 times faster!!!
Why? And how I can speed up it? Maybe maven has some config keys related to artifacts downloading speed?
Thank you.
I know its an old question, but stumbled here from Google. I already had proxy in place, just needed to speed up with concurrent downloads. You can using mvn option:
-Dmaven.artifact.threads=30
Source:
https://maven.apache.org/guides/mini/guide-configuring-maven.html
Configuring Parallel Artifact Resolution
By default, Maven 2.1.0+ will download up to 5 artifacts (from different groups) at once. To change the size of the thread pool, start Maven using -Dmaven.artifact.threads. For example, to only download single artifacts at a time:
mvn -Dmaven.artifact.threads=1 verify
You may wish to set this option permanently, in which case you can use the MAVEN_OPTS environment variable. For example:
export MAVEN_OPTS=-Dmaven.artifact.threads=3
Use a local repository manager/mirror/proxy. All downloads will then go against this instead against the public repositories on the internet. The most popular ones are:
Archiva: http://archiva.apache.org/
Artifactory: http://www.jfrog.org/
Nexus: http://www.sonatype.org/nexus/
They are fairly easy to install and set up and provide a lot of value. Most of them have free versions as well. Just use an old development box to get started and move to a real server once you want to broaden the scope and make it available to more people.
The key point of the question was missed in the answers above:
I've tried to download the same jar using curl - it is 3 times faster!!!
This means it is a software issue, mitigation by installing a local proxy or altering the snapshot policy in the settings.xml both come with extra work and potential side effects, such as snapshot dependencies not being updated.
The issue described by the question is maven not utilizing the available bandwidth, thus being slow. This issue was identified in https://issues.apache.org/jira/browse/WAGON-537 and is resolved since maven 3.6.1, see https://maven.apache.org/docs/3.6.1/release-notes.html and https://issues.apache.org/jira/browse/MNG-6591, respectively. There is thus no need to do anything else but update to the latest maven version.
The best optimization is to avoid downloading. Have a look to your settings.xml maven configuration and check if the updatePolicy flag is set to "daily" on releases and snapshots. This should be the default but sometimes it may be set to 'always' - e.g. in repository manager configurations.
Caution: In this case (daily) you have to be cautious on snapshot changes that you might not get immediatly.
I know that this is not a direct answer to your question but the best maven download optimization I know.
You can download artifact using curl (if you think that is faster) and install it to your maven repository using following command:
mvn install:install-file -Dfile=<path-to-file> -DgroupId=<group-id> \
-DartifactId=<artifact-id> -Dversion=<version> -Dpackaging=<packaging>
http://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
Once you install it in local repository, next time maven will pick it up from there and will not try to download again.
Additionally, if the central maven repository site is slower to you, please consider using Maven Repository Mirrors.
Guide to Mirror Settings - http://maven.apache.org/guides/mini/guide-mirror-settings.html
Is there anything already out there that would help with either / both of following?
Perform a Maven install when a file within that module changes
Perform a Maven install on the module and its dependencies if they have changed
I'm pretty sure I've heard of a Maven option to build dependencies as well but struggling to find anything from Googling...
Perhaps this isn't going to be Maven specific but instead involve a file watching tool that is OS specific, if so I would be interested in hearing about tools for Windows (XP).
Even though the answer comes late, I was looking for the same thing and I found https://github.com/rzymek/watcher-maven-plugin. Maybe someone else will be looking for the same thing.
I tried #Adrian's solution, but I couldn't get it working.
I found this maven package, https://github.com/fizzed/maven-plugins#watcher-fizzed-watcher-maven-plugin, that will watch for file changes, and I was able to get it working pretty quickly.
To building a module and its dependencies, use:
mvn -pl :module -amd
Automated builds are usually triggered from a version system like subversion or git.
Then you can use continuous integration tools like Jenkins.
There's also mvn reactor:make-scm-changes, which detects what modules have local changes vs. the configured scm system.
Unfortunately the maven initial build is very slow due to artifacts downloading
f.e. I've tried to download the same jar using curl - it is 3 times faster!!!
Why? And how I can speed up it? Maybe maven has some config keys related to artifacts downloading speed?
Thank you.
I know its an old question, but stumbled here from Google. I already had proxy in place, just needed to speed up with concurrent downloads. You can using mvn option:
-Dmaven.artifact.threads=30
Source:
https://maven.apache.org/guides/mini/guide-configuring-maven.html
Configuring Parallel Artifact Resolution
By default, Maven 2.1.0+ will download up to 5 artifacts (from different groups) at once. To change the size of the thread pool, start Maven using -Dmaven.artifact.threads. For example, to only download single artifacts at a time:
mvn -Dmaven.artifact.threads=1 verify
You may wish to set this option permanently, in which case you can use the MAVEN_OPTS environment variable. For example:
export MAVEN_OPTS=-Dmaven.artifact.threads=3
Use a local repository manager/mirror/proxy. All downloads will then go against this instead against the public repositories on the internet. The most popular ones are:
Archiva: http://archiva.apache.org/
Artifactory: http://www.jfrog.org/
Nexus: http://www.sonatype.org/nexus/
They are fairly easy to install and set up and provide a lot of value. Most of them have free versions as well. Just use an old development box to get started and move to a real server once you want to broaden the scope and make it available to more people.
The key point of the question was missed in the answers above:
I've tried to download the same jar using curl - it is 3 times faster!!!
This means it is a software issue, mitigation by installing a local proxy or altering the snapshot policy in the settings.xml both come with extra work and potential side effects, such as snapshot dependencies not being updated.
The issue described by the question is maven not utilizing the available bandwidth, thus being slow. This issue was identified in https://issues.apache.org/jira/browse/WAGON-537 and is resolved since maven 3.6.1, see https://maven.apache.org/docs/3.6.1/release-notes.html and https://issues.apache.org/jira/browse/MNG-6591, respectively. There is thus no need to do anything else but update to the latest maven version.
The best optimization is to avoid downloading. Have a look to your settings.xml maven configuration and check if the updatePolicy flag is set to "daily" on releases and snapshots. This should be the default but sometimes it may be set to 'always' - e.g. in repository manager configurations.
Caution: In this case (daily) you have to be cautious on snapshot changes that you might not get immediatly.
I know that this is not a direct answer to your question but the best maven download optimization I know.
You can download artifact using curl (if you think that is faster) and install it to your maven repository using following command:
mvn install:install-file -Dfile=<path-to-file> -DgroupId=<group-id> \
-DartifactId=<artifact-id> -Dversion=<version> -Dpackaging=<packaging>
http://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
Once you install it in local repository, next time maven will pick it up from there and will not try to download again.
Additionally, if the central maven repository site is slower to you, please consider using Maven Repository Mirrors.
Guide to Mirror Settings - http://maven.apache.org/guides/mini/guide-mirror-settings.html
I have a Maven project that needs to be versioned. I have chosen to use the versions-maven-plugin as my versioning plugin but am unsure if that's the best option.
I have read the documentation that such plugin actually modifies the POM and I don't really like that approach. I have worked on projects where they had separate build.properties file that got modified manually.
What I want to achieve is to have my CI generating the artifact for me ready to be deployed and update the version number automatically.
So, any suggestions? How have you done before?
Thank you
I'd get the version number from the one source that matters: that's the source code management system (Subverson, Mercurial, or Git), not Maven.
I'd say that Maven might be out of synch unless your Maven plug-in is getting it from SCM.
Use the Release Plugin. You want to perform automatic deployment and batch release. The Versions Plugin is designed for something else.
We have found MAVEN-RELEASE-PLUGIN super useful and can not imagine releasing and managing version with it.
Warning: I have just picked up Maven, so things mentioned might be wrong or not best practice.
I have a medium size open source project that I am migrating to Maven from the basic
NetBeans project management. This is not a developer team sharing the same room, this is 1-5 people over the internet sharing a SVN repo. Reading over the how-tos on dependencies, it seems that the only way to get dependencies is to get them from an online repo or install them locally.
This is not what I was looking for. I want to keep all dependencies in the SVN for many reasons including portability (anybody can pass by, check out the repo, build, and use; all that simply without manual adding to local repo's and whatnot), getting newer versions (discussed below), and manual versioning.
The other issue I have with the maven repository is that they are quite behind in versions. Logback for example is 0.9.18 in mvnbrowser but 0.9.24 officially. PircBot is 1.4.6 in mvnbrowser but 1.5.0 officially. Why such old versions?
Issue 3 is that I have dependencies that don't even exist in the repos, like Easier Java Persistence.
So
How can I force all dependencies to come from /lib for example
On a related note, can mvn build from library's SVN repo directly? Just curious
Is there an automatic way to get the newest version directly from a dependencies site/svn repo if they also use Maven? IE libraries like commons-lang or logback
Is there a better way of managing dependencies? (IE Ivy or some weird POM option I'm missing)
FYI, this is a Java project with 3 modules, project global dependencies and module specific dependencies.
Bonus points if it can work with the bundled version of Maven that comes with NetBeans.
Not a duplicate of
Maven: add a dependency to a jar by relative path - Not wanting to install to local repository
maven compile fails because i have a non-maven jar - Don't think a System dependency is the right answer
maven look for new versions of dependencies - Still uses(?) repository, just the latest (old) version
This is not what I was looking for. I want to keep all dependencies in the SVN for many reasons (...)
I will come back on this but the solution I described in Maven: add a dependency to a jar by relative path (using a file-based repository) allows to implement such a solution.
The other issue I have with the maven repository is that they are quite behind in versions. Logback for example is 0.9.18 in mvnbrowser but 0.9.24 officially. PircBot is 1.4.6 in mvnbrowser but 1.5.0 officially. Why such old versions?
It looks like mvnbrowser indices are totally out of date (making it useless as repository search engine) because the maven central repository does have logback-core-0.9.24.jar (the logback project is doing what has to be done to make this happen) but only has an old pircbot-1.4.2.jar. Why? Ask the pircbot team. Anyway, you're right, the central repository might not always have ultimate versions.
Issue 3 is that I have dependencies that don't even exist in the repos, like Easier Java Persistence.
Yeah, this happens too.
How can I force all dependencies to come from /lib for example
As previously hinted, you should re-read carefully the solution suggested in Maven: add a dependency to a jar by relative path. This solution is not about installing libraries to the local repository but is about using a file-based repository (that could thus be stored in SVN). You might have missed the point, this matches your use case. And also check Brett's answer for a variation.
On a related note, can mvn build from library's SVN repo directly? Just curious
Didn't get that one. Can you clarify?
Is there an automatic way to get the newest version directly from a dependencies site/svn repo if they also use Maven? IE libraries like commons-lang or logback
Maven supports version ranges and you could use a syntax allowing to use "any version greater than X". But I do NOT recommend using version ranges at all, for the sake of build reproducibility. You don't want the build to suddenly fail because of some automatic update that happened on your back. Only upgrade if you need bug fixes or new features, but do it explicitly (if it ain't broke, don't fix it).
You might also find mentions of the LATEST and RELEASE version markers. I don't recommend them neither for the same reasons as above and even less since they're removed from Maven 3.x.
Is there a better way of managing dependencies? (IE Ivy or some weird POM option I'm missing)
Can't say for Ivy. But in the Maven land, if you can't host up a "corporate" repository for your project (Nexus, Archiva, Artifactory), then the file-based repository is IMO the best approach.
Setup your own Maven repository.
http://archiva.apache.org/