I'd like maven to report unresolved dependencies in multi-module maven Java project which has below structure:
multi-module-java-app/
├── app1
│ ├── pom.xml
│ └── src
├── app2
│ ├── pom.xml
│ └── src
└── pom.xml
poms are at the bottom.
Background:
Maven is used as a dependency management and build tool. Artifactory is repository manager. Artifacts may be built using maven locally on a developer’s environment or on Jenkins build server. Artifactory periodically moves artifacts to special archive repository which is part of all repository virtual.
Maven caches locally built artifacts in~/.m2 directory on the computer where Maven runs whether it’s developer environment or build server.
Problem
Several issues may arise:
Local builds on developers’ VMs may succeed, but fail in Jenkins.
Local builds, Jenkins builds may succeed, but fail on another developer’s VM.
Cause
Present/missing artifacts on a developer .m2 cache, missing artifacts in build server’s .m2 cache and/or archive Artifactory repository
Proposed solution
Run [path_to_maven]mvn dependency:list. [path_to_maven] has custom maven installation (with empty .m2 cache) in the root folder of the project. Custom maven is also configured using settings.xml to use special non-archived repository (all repository without the archive).
The output is like below:
It reports unresolved dependencies as well as dependent artifacts which miss them.
However this solution has 2 main drawbacks:
.m2 missing slows down the detection significantly as all the dependencies have to be downloaded
unresolved artifacts or modules which miss them are always snapshots.
Running mvn -B -q versions:set -DnewVersion=[some_version] solves the second. This command runs during release pipeline anyway.
However it's not clear how to solve the first.
How to find unresolved maven dependencies while using .m2, so that unresolved dependencies may be detected quickly during Jenkins build after each push to feature branch?
The only idea that pops is that .m2 on build server will be synced with Artifactory.
.m2 on developers' machines can be synced as well using some sort of custom plugin that uses rsync. Is there a known plugin that does this?
The ultimate goal is to remove archive repository and let the builds fail. But, first developers need to align the dependencies to the latest versions.
root pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.app</groupId>
<artifactId>multi-module-java-app</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>pom</packaging>
<name>multi-module-java-app</name>
<!-- FIXME change it to the project's website -->
<url>http://www.example.com</url>
<pluginRepositories>
<pluginRepository>
<id>plugins</id>
<name>plugins</name>
<url>http://localhost:8081/artifactory/all</url>
</pluginRepository>
</pluginRepositories>
<repositories>
<repository>
<id>all</id>
<name>all</name>
<url>http://localhost:8081/artifactory/all</url>
</repository>
</repositories>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.example</groupId>
<artifactId>spring-boot</artifactId>
</dependency>
</dependencies>
</dependencyManagement>
<modules>
<module>app1</module>
<module>app2</module>
</modules>
</project>
app 1 pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.mycompany.app</groupId>
<artifactId>multi-module-java-app</artifactId>
<version>1.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.mycompany.app.app1</groupId>
<artifactId>app1</artifactId>
<version>1.0-SNAPSHOT</version>
<name>app1</name>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<finalName>${artifactId}</finalName>
</configuration>
</plugin>
</plugins>
</build>
<pluginRepositories>
<pluginRepository>
<id>plugins</id>
<name>plugins</name>
<url>http://localhost:8081/artifactory/all</url>
</pluginRepository>
</pluginRepositories>
<repositories>
<repository>
<id>all</id>
<name>all</name>
<url>http://localhost:8081/artifactory/all</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>com.example</groupId>
<artifactId>spring-boot</artifactId>
<version>0.0.1-20200510.095344-1</version>
</dependency>
</dependencies>
</project>
app2 pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.mycompany.app</groupId>
<artifactId>multi-module-java-app</artifactId>
<version>1.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.mycompany.app.app2</groupId>
<artifactId>app2</artifactId>
<version>1.0-SNAPSHOT</version>
<name>app2</name>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<finalName>${artifactId}</finalName>
</configuration>
</plugin>
</plugins>
</build>
<pluginRepositories>
<pluginRepository>
<id>plugins</id>
<name>plugins</name>
<url>http://localhost:8081/artifactory/all</url>
</pluginRepository>
</pluginRepositories>
<repositories>
<repository>
<id>all</id>
<name>all</name>
<url>http://localhost:8081/artifactory/all</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>com.example</groupId>
<artifactId>spring-boot</artifactId>
<version>0.0.1-20200510.095344-1</version>
</dependency>
</dependencies>
</project>
I have run into this exact issue where the developer's local build environment was different from Jenkins Slave environment. In the ideal world, the developer needs to baseline their local environment with that of the slave, or depend entirely on Jenkins job builds once initial stages of development are complete.
I appreciate the fact that you are trying to provide an automated sync feature of the .m2 repositories, it is feasible but adds scope for error and additional routine maintenance tasks, not to mention user education issues. For instance, how will you ensure that the .m2 is the latest version? In terms of maven dependencies, the developer knows best, or they may be introducing new dependencies which do not exist on the slave yet. Thereby, I suggest fixing the root problem of developer not aligning their dependencies which is more design related than automation.
Not sure if you want to take this path, but may help someone:
Eliminate the need for .m2 repositories in local machines of developers. The m2 cache creates a problem, if the developer's machine gets wiped out or corrupt and there will be a need for updates, audits and reconciliation.
Eliminate the need for .m2 in Jenkins slaves. The problem here is, multiple slaves tend to have different .m2 cache content and syncing back and forth from Artifactory and then developers also syncing it to their local sounds complicated. There is no saying that all these .m2 will be in sync at any point of time and a builds may still get executed with an n-1 version of it.
Now that there is no .m2 , we still need a place for developers to pull dependencies from. Push all your dependencies to a repo in Artifactory and actively maintain it. Use the setting.xml feature to pull the standard dependencies. If a developer is building in his local machine, via Eclipse or other, the dependencies will be available to be pulled from the IDE itself by using the same xml reference so there is no local cache maintained on the developer's machine.
When the build environment has a minor difference between the local and the Jenkins Slave This causes the .jar to be the same size or slightly different in kilobytes of size.
To identify this difference between jar files use any tools listed here in this post, it also helps developers to identify which dependency is out of sync, by themselves:
Comparing two .jar files
If this design is implemented, the Artifactory repository containing the dependencies becomes the single source of truth for dependencies, you need to work with developers to create a cadence as to how the dependencies will be updated and consumed to/from this single source of truth. I hope this is useful.
Related
I made a simple Maven project for my class. According to the teachers tutorial we cannot upload it to our school repo due to some server issues, so we have to store it locally using altDeploymentRepository. I have the following pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dpp</groupId>
<artifactId>simple_lib</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
</properties>
<build>
<plugins>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
<configuration>
<altDeploymentRepository>internal.repo::default::file://${project.basedir}/../${project.name}-mvn-repo</altDeploymentRepository>
</configuration>
</plugin>
</plugins>
</build>
</project>
So in the directory with my Maven project I have two directories:
sample_lib
sample_lib-mvn-repo
In the second one, deep down in : sample_lib-mvn-repo\com\dpp\sample_lib\1.0-SNAPSHOT I have a .jar file which I want to import (but not using just .jar file like passing the path to it - I need to do this "Maven way", import it as Maven lib). Can I do it if the file is not stored on any remote repository, but on my hard drive?
Running simply mvn install will install the file in your local repository. The local repository, by default, is in your home directory, under .m2\repository.
Using your pom above, after running mvn install, you would have jar (and some other files) in .m2\repository\com\dpp\sample_lib\1.0-SNAPSHOT.
To import this subsequently in another project, you would create a dependency in that project's pom like:
<dependency>
<groupId>com.dpp</groupId>
<artifactId>simple_lib</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
This all takes place only on your machine, and will not use any remote repository.
Now you have a local simulation of a repository.
You can import it using the repository tag as described in https://maven.apache.org/pom.html#Repositories. To specify a file make it a file url like file:
Yes, you can add maven repository and point it to a local directory:
<repository>
<id>local</id>
<name>local</name>
<url>file:${user.dir}/sample_lib-mvn-repo</url>
</repository>
https://maven.apache.org/guides/introduction/introduction-to-repositories.html
Given that your jar file is here sample_lib-mvn-repo\com\dpp\sample_lib\1.0-SNAPSHOT.jar, you then can add it as a dependency:
<dependency>
<groupId>com.dpp</groupId>
<artifactId>sample_lib</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
It's not exactly what you're asking. But a quick and dirty solution is to make a POM project (no source code).
Inside the POM project you have the main and external projects.
You can simply make a dependency on the other project.
Here is my situation:
I'm trying to migrate from Ant to Maven
My project has 3 artifacts: shared api (jar), web app (war), desktop swing app (jar). Latter 2 depends on shared api.
At this moment I'm trying to make web app part work. So I've created 4 poms: eftracker (root pom), eftracker-parent, eftracker-shared, eftracker-web.
If I run mvn package on eftracker all works just perfect -- I have eftracker-shared.jar and eftracker-web.war created as expected
I added tomcat7-maven-pluginto run web app with maven goal tomcat7:run to test changes made during development
I also added eftracker-shared as a project to eftracker-web build path.
My goal:
Now I want to work comfortably in Eclipse, meaning I want to change files, hit Run and in couple seconds be able to test my changes.
During development I will change both: shared and web projects.
My problem:
If I never run mvn install than an attempt to invoke tomcat7:run will lead to error: Failed to execute goal on project eftracker-web: Could not resolve dependencies for project com.skarpushin:eftracker-web:war:1.503.0: Could not find artifact com.skarpushin:eftracker-shared:jar:1.503.0 in central (https://repo.maven.apache.org/maven2)
It appears I have to mvn clean install shared project (or even on root module) each time I change it before I can execute tomcat7:run on web app and see recent changes.
Question is:
Is it possible to make this process automatic?
...OR maybe there is other way how to minimize "maven overhead" during development?
eftracker.pom
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.skarpushin</groupId>
<artifactId>eftracker</artifactId>
<version>1.503.0</version>
<packaging>pom</packaging>
<name>eftracker</name>
<modules>
<module>eftracker-parent</module>
<module>eftracker-shared</module>
<module>eftracker-web</module>
</modules>
<properties>
<maven.deploy.skip>true</maven.deploy.skip>
</properties>
</project>
eftracker-parent/pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.skarpushin</groupId>
<version>1.503.0</version>
<artifactId>eftracker-parent</artifactId>
<packaging>pom</packaging>
<name>eftracker-parent</name>
<!-- ...some common properties, dependencies, build plugins... -->
</project>
eftracker-web/pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<groupId>com.skarpushin</groupId>
<artifactId>eftracker-parent</artifactId>
<version>1.503.0</version>
<relativePath>../eftracker-parent</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>eftracker-web</artifactId>
<packaging>war</packaging>
<build>
<plugins>
<plugin>
<groupId>org.apache.tomcat.maven</groupId>
<artifactId>tomcat7-maven-plugin</artifactId>
<version>2.2</version>
<configuration>
<path>/</path>
<port>8080</port>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<warSourceDirectory>src/main/webapp</warSourceDirectory>
<warName>ROOT##${project.version}</warName>
<failOnMissingWebXml>false</failOnMissingWebXml>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>com.skarpushin</groupId>
<artifactId>eftracker-shared</artifactId>
<version>${project.version}</version>
</dependency>
<!-- ...other deps -->
</dependencies>
</project>
Try to use M2Eclipse
https://www.eclipse.org/m2e/
M2Eclipse provides tight integration for Apache Maven into the IDE
with the following features:
Launching Maven builds from within Eclipse
Dependency management for Eclipse build path based on Maven's pom.xml
Resolving Maven dependencies from the Eclipse workspace without installing to local Maven repository
Automatic downloading of the required dependencies from the remote Maven repositories
Wizards for creating new Maven projects, pom.xml and to enable Maven support on plain Java project
Quick search for dependencies in Maven remote repositories
So it appears there are 2 things needs to be done:
run mvn compile on parent project in that way all classes will appear in ../parent/target/classes folder. Note that they'll be automatically updated by Eclipse if you change source code
edit Eclipse run configuration and put this checkbox "Resolve Workspace artifacts"
Now I was able to run project as Maven build... with goal tomcat7:run and it worked without the need of parent project to be installed
I've tried to connect my application with database as described on this site: https://www.visual-paradigm.com/tutorials/hibernateinnetbeans.jsp
I use MySQL and Maven project. But the rest is done the way it was explained on the website and when I run CreateAutoPartsStore I have such output:
The POM for unknown.binary:orm:jar:SNAPSHOT is missing, no dependency information available
[...]
--- exec-maven-plugin:1.2.1:exec (default-cli) # AutoPartsStore ---
org.hibernate.HibernateException: /ormmapping/AutoPartsStore.cfg.xml not found
at org.hibernate.internal.util.ConfigHelper.getResourceAsStream(ConfigHelper.java:173)
at org.hibernate.cfg.Configuration.getConfigurationInputStream(Configuration.java:2026)
at org.hibernate.cfg.Configuration.configure(Configuration.java:2007)
at org.orm.PersistentManager.<init>(PersistentManager.java:72)
at autopartstore.AutoPartsStorePersistentManager.<init>(AutoPartsStorePersistentManager.java:31)
at autopartstore.AutoPartsStorePersistentManager.instance(AutoPartsStorePersistentManager.java:41)
at ormsamples.CreateAutoPartsStoreData.main(CreateAutoPartsStoreData.java:65)
This is my pom.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>AutoPartsStore</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<repositories>
<repository>
<id>unknown-jars-temp-repo</id>
<name>A temporary repository created by NetBeans for libraries and jars it could not identify. Please replace the dependencies in this repository with correct ones and delete this repository.</name>
<url>file:${project.basedir}/lib</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>unknown.binary</groupId>
<artifactId>orm</artifactId>
<version>SNAPSHOT</version>
</dependency>
</dependencies>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
</project>
I found some solutions to the problem concerning copying XML files to another location, but none of them worked for me.
Looks like you need to add hibernate jar to the Maven dependecies list.
This:
<dependency>
<groupId>unknown.binary</groupId>
<artifactId>orm</artifactId>
<version>SNAPSHOT</version>
</dependency>
informs maven that your repository contains the artifact unknown.binary:orm:jar:SNAPSHOT but it seems it was never installed, and obviously is not present in Maven Central to be downloaded. What is it expected to point to?
Note that .jar lying around in folder marked as Maven repository does not make it count as installed. Also, Maven does not understand the notion of "Jar directory" and will not automatically add to classpath every jar from lib directory, if it is not declared as dependency.
I'm new to maven, and trying to understand how to release my project. I have the following project setup in svn:
trunk
|-deployer
| |-pom.xml
|-webapp
| |-pom.xml
|-utils
|-pom.xml
While developing webapp, I always want to develop against the latest snapshot version of utils, so I declare the dependency on utils in webapp/pom.xml via:
com.company
utils
1.0-SNAPSHOT
Webapp itself is also currently versioned at version 1.0-SNAPSHOT. It's pom.xml has the declaration:
<artifactId>webapp</artifactId>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>
<name>Sample Webapp</name>
So everything is working great, but now I want to release my software. In deployer, I have the following in pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company</groupId>
<artifactId>deployer</artifactId>
<packaging>pom</packaging>
<version>1.0-SNAPSHOT</version>
<name>Project Release</name>
<modules>
<module>externals/webapp</module>
</modules>
<properties>
<url.svn>http://<my-server>/<project>/trunk</url.svn>
</properties>
<scm>
<connection>scm:svn:${url.svn}</connection>
</scm>
<build>
<plugins>
<plugin>
<artifactId>maven-release-plugin</artifactId>
<version>2.3</version>
<configuration>
<tagBase>
http://<my-server>/<project>/tags
</tagBase>
</configuration>
</plugin>
</plugins>
</build>
<distributionManagement>
<repository>
<id>Releases</id>
<name>Releases</name>
<url>http://<nexus-server>/nexus/content/repositories/releases</url>
</repository>
<snapshotRepository>
<id>Snapshots</id>
<name>Snapshots</name>
<url>http://<nexus-server>/nexus/content/repositories/snapshots</url>
</snapshotRepository>
</distributionManagement>
</project>
Within the deployer folder itself I have a folder called "externals" which has an svn-externals set to check out my webapp project (I added this because maven complained about not being able to find it), so my modules path should be correct.
Questions
1.) I want to release a version of my web app, but when I run a mvn release:prepare and mvn release:perform in my deployer project, my only tangible output is my trunk tagged in svn and a pom uploaded to my nexus repository. This makes sense in that my deployer artifact packaging is type "pom", but it also doesn't get the job done of getting me a war of my webapp (I should note here that if a do a release in the webapp project by itself though, that I will get the war). I need to release multiple modules, and so I thought I could use maven aggregation from the deployer project to accomplish this, but it doesn't seem to be working.
2.) Is there a better way to achieve what I am trying to do?
Thank you for any insights you can provide.
You should run the maven-release-plugin against each module that you intend to release. I've never seen someone have a special "deployer" module that releases the other modules, this is not how things are commonly done.
Normally to release the webapp module you would run the commands against the webapp module, and to release the utils module you would run the commands against the utils module.
If you have a parent module that ties webapp and util together then I believe you can just run the release commands against that.
As matt b says, simply adding a maven-release-plugin entry at the top pom will probably do what you want. If you want to collect several artifacts into a proper release bundle, you want to create a module for this, but then you want to look at the maven assembly plugin. It can collect various jars (wether from modules or external dependencies) and resources into a directory, zip file or similar.
Is it possible to distribute my application with its POM only ?
I have deployed my application in a remote repository and I think it would be nice if I can distribute only its POM, instead of asking the users to download the complete source first and use the POM to build the application afterwards.
The idea is that users would be able to install the application using the POM and a single Maven command.
I tried adding to the POM a downloadURL in a distributionManagement section without success. Here my experiment:
<?xml version="1.0" encoding="UTF-8"?>
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>aGroupId</groupId>
<artifactId>anArtifactId</artifactId>
<version>0.0.1</version>
<distributionManagement>
<downloadUrl>anURL</downloadUrl>
</distributionManagement>
<repositories>
<repository>
<id>someId</id>
<url>anURL</url>
</repository>
</repositories>
</project>
Thanks in advance for any feedback
If you're distributing source, then you should look at the bootstrap POM method: http://maven.apache.org/scm/maven-scm-plugin/examples/bootstrapping-with-pom.html
where the 'scm' element is key. The user has only to run scm:bootstrap to then receive the project source tree from which to build the project.
-tim
You can use the dependency:get mojo to download an artifact to a specified location:
mvn dependency:get -Dartifact=<groupid>:<artifactid>:<version> /
-Ddest=path/to/destination.jar