Using this method, I could successfully resolve (but NOT retrieve the actual jars) all dependencies of a library when its organization, name and revision are given, all programatically using Apache Ivy.
During the resolution process, it parses given library's POM and finds a parent POM if specified, and Ivy's POM parser always tries to resolve sources and javadoc, which generally doesn't exist for parent POMs.
The problem is that when a non-existing jar's URL is queried to our local artifactory, it takes 2~3 seconds to refresh its cache, adding up tens of minutes to total dependency resolution time when I have many transitive dependencies having parent POMs.
For example, the following is the log message I get when I try to resolve Google Guava 17.0:
:: loading settings :: url = jar:file:.ivy2/cache/org.apache.ivy/ivy/jars/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
:: resolving dependencies :: com.google.guava#guava-envelope;17.0 [not transitive]
confs: [default]
validate = true
refresh = false
resolving dependencies for configuration 'default'
== resolving dependencies for com.google.guava#guava-envelope;17.0 [default]
== resolving dependencies com.google.guava#guava-envelope;17.0->com.google.guava#guava;17.0 [default->master]
tried http://artifactory/repo/com/google/guava/guava/17.0/guava-17.0.pom
resolver: found md file for com.google.guava#guava;17.0
=> http://artifactory/repo/com/google/guava/guava/17.0/guava-17.0.pom (17.0)
downloading http://artifactory/repo/com/google/guava/guava/17.0/guava-17.0.pom ...
resolver: downloading http://artifactory/repo/com/google/guava/guava/17.0/guava-17.0.pom
[SUCCESSFUL ] com.google.guava#guava;17.0!guava.pom(pom.original) (58ms)
tried http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0.pom
resolver: found md file for com.google.guava#guava-parent;17.0
=> http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0.pom (17.0)
downloading http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0.pom ...
resolver: downloading http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0.pom
[SUCCESSFUL ] com.google.guava#guava-parent;17.0!guava-parent.pom(pom.original) (20ms)
resolver: revision in cache: org.sonatype.oss#oss-parent;7
tried http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0.jar
tried http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0.jar
tried http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0-sources.jar
CLIENT ERROR: Could not find resource url=http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0-sources.jar
tried http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0-src.jar
CLIENT ERROR: Could not find resource url=http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0-src.jar
tried http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0-javadoc.jar
CLIENT ERROR: Could not find resource url=http://artifactory/repo/com/google/guava/guava-parent/17.0/guava-parent-17.0-javadoc.jar
tried http://artifactory/repo/com/google/guava/guava/17.0/guava-17.0.jar
tried http://artifactory/repo/com/google/guava/guava/17.0/guava-17.0-sources.jar
tried http://artifactory/repo/com/google/guava/guava/17.0/guava-17.0-javadoc.jar
found com.google.guava#guava;17.0 in resolver
resolved ivy file produced in cache
:: resolution report :: resolve 2593ms :: artifacts dl 0ms
:: modules in use:
com.google.guava#guava;17.0 from resolver in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 1 | 1 | 0 || 0 | 0 |
---------------------------------------------------------------------
report for com.google.guava#guava-envelope;17.0 default produced in .ivy2/cantabile/com.google.guava-guava-envelope-default.xml
resolve done (2593ms resolve - 0ms download)
You can see that Ivy wasted a few seconds while trying nonexistent -sources.jar, -src.jar, -javadoc.jar URLs for parent POMs.
Is there an option to disable this behavior when resolving a parent POM? Ideally I don't want to make HTTP requests to any .jars during resolution. Currently I need a programmatic solution but if anyone knows a command line/XML option it will be a great help.
PS: I think this is also the case when using sbt to resolve library dependencies; it takes very long while resolving oss-parent, commons-parent etc.
You need to specify a configuration mapping that excludes both optional dependencies and implicit dependencies like source and javadoc files.
In an ivy file you'd declare the dependency as follows:
<dependency org="myorg" name="mymodule" rev="1.0" conf="default"/>
Another option is to declare this dependency mapping as the default for your project.
<dependencies defaultconfmapping="default">
<dependency ..
<dependency ..
See the following SO questions on how to exclude sources and how ivy translates Maven repositories:
Ignore sources jar in Ivy retrieve
How are maven scopes mapped to ivy configurations by ivy
In conclusion Maven modules by default only have a single arefact. More can be stored (like sources and Javadocs) but there is no module metadata listing them. This is why Ivy performs a HTTP request to see if the artefacts exist or not. Changing the configuration mapping settings should help guide ivy to do the right thing :-)
PS
I think this can be done by called the "addDependencyConfiguration" method to the DefaultDependencyDescriptor object.
Related
I started with a quarkus multi maven-module project and encountered the following problem when starting quarkus in the dev mode (mvn quarkus:dev) to support hot reload for depended modules' classes:
A module "my-app" configures its' build output path via the
<project><build><outputDirectory>build</outputDirectory> in the pom.xml. The module is build successfully, and the output exists in path my-app/build instead of the default target/classes.
Then the build of another module "momsServerQuarkus", which depends on the "my-app", and which uses quarkus-maven-plugin, fails with error:
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:2.11.0.Final:dev (default-cli) on project momsServerQuarkus: Failed to run: Hot reloadable dependency de.kwsoft.moms:my-app:6.14.0-SNAPSHOT has not been compiled yet (the classes directory S:\trunk\moms\my-app\target\build does not exist)
As visible in the error message, the <outputDirectory>build</outputDirectory>is read correctly from the my-apps' pom.xml, but is appended to the default output path "target" instead of the projects root folder. I played also variables, e.g. <outputDirectory>${basedir}/build</outputDirectory>, but the variable is not resolved by the quarkus-maven-plugin.
Does anyone know how this could be solved, or whether this could be a bug?
Thank you in advance.
I'm working on a project where ant + ivy are used for build process.
Since Intellij IDEA is my primary IDE for development I've downloaded IvyIDEA plugin in order to automate the process of resolving dependencies by ivy.
After a while I noticed that not all the dependencies were downloaded, some of them were, but some custom artifacts were missing. I checked the generated list of dependencies by plugin and some of them were actually missing thought the resolving process finished successfully:
report for projName compile produced in C:\Users\test\.ivy2\cache\projName-compile.xml
resolve done (6375ms resolve - 156ms download)
post 1.3 ivy file: using exact as default matcher
post 1.3 ivy file: using exact as default matcher
post 1.3 ivy file: using exact as default matcher
post 1.3 ivy file: using exact as default matcher
post 1.3 ivy file: using exact as default matcher
post 1.3 ivy file: using exact as default matcher
post 1.3 ivy file: using exact as default matcher
No problems detected during resolve for module 'ProjName' [All configurations].
I started to dive into ivy logs generated by the plugin and saw this for a bunch of different dependencies:
Sort dependencies of : projToImport;1.1.85300.20210326.5 / Number of dependencies = 8
Non matching revision detected when sorting. projToImportdepends on anotherProj;1.1.81201.20210326.1, doesn't match anotherProj;1.1.81201.20210406.1
Module descriptor is processed : junit#junit;4.11
....
IMPORTANT NOTE: when I do the same in Eclipse IDE with installed Ivy plugin, it all works and can I see all the dependencies described in ivy.xml This makes me think that my ivy-settings.xml and ivy.xml files are correct.
So, my assumption that IvyIDEA plugin doesn't work correct way or something. How is it possible to fix this?
My IveIDEA configuration:
The issue was in the version of ivy: the current version of IvyIDEA plugin (1.0.16) I used contained ivy 2.5.0 and I had to use ivy 2.4.0 in order to run my ivy-settings.xml\ivy.xml.
So the solution for me was to downgrade ivy to version 2.4.0.
I have a gradle project with several subprojects. I have defined several dependencies with version numbers in the root project but not all subprojects use all of these dependencies.
root
- build.gradle
- compile 'math:math:1.0.0'
- settings.gradle
- include 'messages'
- include 'message-handler'
\ messages
- build.gradle
- //no math
\ message-handler
- build.gradle
- compile 'math:math'
Will my artifact of the messages project contain a dependency on the math library?
In other words, if I make a separate project that depends on the messages artifact from a nexus repository, would my dependency tree show the math library for this new project?
Yes - your messages project artifact will contain a dependency on the math library.
According to Gradle Documentation:
For most multi-project builds, there is some configuration which is
common to all projects. In our sample, we will define this common
configuration in the root project, using a technique called
configuration injection. Here, the root project is like a container
and the subprojects method iterates over the elements of this
container - the projects in this instance - and injects the specified
configuration
In other words, every configuration which is included in the root project will hold for all the sub-projects.
I have a mojo annotated with #requiresDependencyResolution test.
It works for multi-module projects with a single layer of nesting, but a user has reported an issue with a structure such as below.
-- my_project
|
-- pom.xml
-- submodule1
|
-- pom.xml
-- submodule2
|
-- pom.xml
-- submodule21
|
-- pom.xml
-- submodule22
|
-- pom.xml
If submodule21 depends on submodule1 maven reports
Failed to execute goal on project submodule21: Could not resolve
dependencies for project org.my:submodule21:jar:1.0-SNAPSHOT: Could
not find artifact org.my:submodule1:jar:1.0-SNAPSHOT
Removing the requiresDependencyResolution=test annotation prevents this problem but then I do not have access to the information I require for the mojo to run.
From brief scan of the surefire code on github, it looks to also use requiresDependencyResolution=test but is able to run against this project without issue.
https://github.com/apache/maven-surefire/blob/master/maven-surefire-plugin/src/main/java/org/apache/maven/plugin/surefire/SurefirePlugin.java
https://github.com/apache/maven-surefire/blob/master/maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/AbstractSurefireMojo.java
The only obvious difference from my own code is that it uses java annotations rather than old style javadoc ones.
How is the surefire mojo achieving this?
My code is
http://code.google.com/p/pitestrunner/source/browse/pitest-maven/src/main/java/org/pitest/maven/PitMojo.java
Example project displaying issue
http://code.google.com/p/pitestrunner/issues/detail?id=71
For the benefit of anyone else having this issue - I eventually solved this problem. There was no issue with the plugin.
The difference between surefire and my own plugin was simply the way in which they were being run. Surefire was bound to the test phase, my own plugin was being run by calling a goal directly. When I bind my plugin to the verify phase, everything resolves without issue.
I am using maven-gae-plugin from http://code.google.com/p/maven-gae-plugin/ to create a multi-module project using inheritance (parent-project reference)
The structure that I have is as below:
|
`-- pom.xml
|-- base-api-project (packaging=jar)
| `-- pom.xml
|-- main-www-project (packaging=war)
| `-- pom.xml
The root pom.xml is configured as parent + aggregator (with reference to the modules).
When I build using the root pom.xml, everything is compiled and packaged great.
However, when I run mvn gae:run in the main-www-project folder, I get an error that the base-api-project is not found in the registered repositories. I scans all registered repositories...
But my question is... if it's already there in ~/.m2/repository, then why should I hunt across the net?
I there something wrong that I'm doing?
When using snapshot versions maven will always look for the latest snapshot, whereas releases will be resolved locally first.
However, each artifact in the local repo will also need the corresponding POM, and that POM may declare dependencies that are not in the local repo, or are snapshot versions (etc. recursively).