I have a project where I use multiple dependencies. I'm used to execute my code in a dev environment using
mvn compile exec:java -Dexec.classpathScope=compile -Dexec.mainClass="my.Main"
That worked fine until I recently started using a dependency (let's call it A) that defines its own dependencies with the runtime scope. One of its runtime dependencies (let's call it B) is actually a dependency to another dependency of mine that I set with a provided scope (let's call it C).
So I have
MyProject depends on A[compile] who depends on B[runtime]
MyProject depends on C[provided] who depends on B[no-scope-specified]
I end up having B considered as a runtime dependency which results in a ClassNotFoundException when I launch the above command. The only solution I found is adding a dependencyManagement entry that forces its scope to compile.
That bothers me as I do not directly use that dependency (I have no reference to it in my code) so I should not have to bother with it.
Is there some way to avoid that kind of conflict and maybe load all dependencies in the classpath when using mvn exec:java?
Thanks
Related
I have a graph with 40K artifacts and I can list all possible dependencies of a package (I do so by parsing a list of effective poms)
For example, I have the following for this package:
There's 2 dependencies without taking in mind different versions.
I would like to show that this results are valid by showing that maven also lists these dependencies for this package. But when I use mvn dependency:tree after I add the com.google.guava:guava:14.0.1, I get no dependencies listed.
This is the pom file of the package:
It clearly has those 2 dependencies, but their scopes are provided. Even if I use -Dinclude=provided or -Dscope=provided as a parameter, I still cannot list them.
So, how do I list all dependencies of a package no matter the scope used?
Use Analyze Dependencies... action in the Maven tool window:
It will show the list of dependencies in the project with their scopes and usages in project:
Scope provided means it's provided at runtime, which implies that it's not a package dependency:
A dependency with this scope is added to the classpath used for compilation and test, but not the runtime classpath. It is not transitive.
I have a pom.xml where i've got hadoop-core dependency as provided
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
When I add cfg4j as compile time dependency
<dependency>
<groupId>org.cfg4j</groupId>
<artifactId>cfg4j-core</artifactId>
<version>4.4.0</version>
</dependency>
<dependency>
<groupId>org.cfg4j</groupId>
<artifactId>cfg4j-consul</artifactId>
<version>4.4.0</version>
</dependency>
I've got an exception "java.lang.NoSuchMethodError: javax.ws.rs.core.Response.someMethod". I've investigated the problem and find out that the problem is from hadoop and cfg4j-consul. Hadoop core depends to jersey-core and cfg4j depends on cxf. Both declared javax.ws.rs as dependecy so the problem is that jersey has version 1.1 and cxf has 2.0.2. Hadoop dependency is provided, cause it's needed by Flink (framework) and it's in the lib folder. I can't just upgrade it or remove it, nor add it as compile time and exclude the lib. Even I was able to do it, I have no guarantees that hadoop will work as expected. I guess shading doesn't fix the problem cause it's not with cfg4j but one of the dependency of his dependency.
Is there way to resolve the conflict? Does gradle has it's onw ways to fix such issue?
Two approaches:
Shading: A bit more difficult as you say because this a transitive dependency, but I would have a look a Maven shade plugin and it would still be possible to declare the dependency directly if necessary.
Don't use the dependency and try to find some other library or solution for your problem.
Try the following steps, here is the source: https://reflectoring.io/nosuchmethod/
Your issue has nothing to do with the choice between Mavern and Gradle, switching therefor will not help.
Fixing a NoSuchMethodError
There are a lot of different flavors of NoSuchMethodErrors, but they all boil down to the fact that the compile time classpath differs from the runtime classpath.
The following steps will help to pinpoint the problem:
Step 1: Find Out Where the Class Comes From
First, we need to find out where the class containing the method in question comes from. We find this information in the error message of the NoSuchMethodError:
Exception in thread "main" java.lang.NoSuchMethodError:
io.reflectoring.nosuchmethod.Service.sayHello(Ljava/lang/String;)Ljava/lang/String;
Now, we can search the web or within the IDE to find out which JAR file contains this class. In the case above, we can see that it’s the Service class from our own codebase and not a class from another library.
If we have trouble finding the JAR file of the class, we can add the Java option -verbose:class when running our application. This will cause Java to print out all classes and the JARs they have been loaded from:
[Loaded io.reflectoring.nosuchmethod.Service from file:
/C:/daten/workspaces/code-examples2/patterns/build/libs/java-1.0.jar]
Step 2: Find Out Who Calls the Class
Next, we want find out where the method is being called. This information is available in the first element of the stack trace:
Exception in thread "main" java.lang.NoSuchMethodError:
io.reflectoring.nosuchmethod.Service.sayHello(Ljava/lang/String;)Ljava/lang/String;
at io.reflectoring.nosuchmethod.ProvokeNoSuchMethodError.main(ProvokeNoSuchMethodError.java:7)
Here, the class ProvokeNoSuchMethodError tries to call a method that does not exist at runtime. We should now find out which library this file belongs to.
Step 3: Check the Versions
Now that we know where the NoSuchMethodError is provoked and what method is missing, we can act.
We should now list all of our project dependencies.
In Gradle, we can call:
./gradlew dependencies > dependencies.txt
If we’re using Maven, a similiar result can be achieved with:
mvn dependency:list > dependencies.txt`
In this file, we can search for the libraries that contain the class with the missing method and the class that tries to call this method.
Usually we’ll find an output like this somewhere:
\--- org.springframework.retry:spring-retry:1.2.2.RELEASE
| \--- org.springframework:spring-core:4.3.13.RELEASE -> 5.0.8.RELEASE
The above means that the spring-retry library depends on spring-core in version 4.3.13, but some other library also depends on spring-core in version 5.0.8 and overrules the dependency version.
We can now search our dependencies.txt file for 5.0.8.RELEASE to find out which library introduces the dependency to this version.
Finally, we need to decide which of the two versions we actually need to satisfy both dependencies. Usually, this is the newer version since most frameworks are backwards compatible to some point. However, it can be the other way around or we might even not be able to resolve the conflict at all.
I have a gradle project P which has module A and B. Module A has this jackson dependencies:
...
dependencies {
...
compile 'com.fasterxml.jackson.core:jackson-core:2.12.0-rc1'
compile 'com.fasterxml.jackson.core:jackson-annotations:2.12.0-rc1'
compile 'com.fasterxml.jackson.core:jackson-databind:2.12.0-rc1'
compile 'com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.12.0-rc1'
...
}
...
and module B uses module A, and has no need for including this dependencies because jackson usage is encapsulated in module A. But when code executed from module B reaches a statement that invokes code from Module A using it, I get exception:
java.lang.NoClassDefFoundError: com/fasterxml/jackson/dataformat/xml/XmlMapper
If I add the same dependencies to Module's B gradle.build file, the code works.
The question is, why would I include them if Module A does not use the library?
Shouldn't dependencies in Module A be compiled, packaged, so that when Module A is used elsewhere, its code works (using its included dependencies such as jackson library as in this example)
Disclaimer: I don't know Gradle, but this sounds like a problem common to Maven and Gradle.
The fact that you can build the module, but not run the module, means that somewhere you are not bringing the transitive dependencies into the Spring Boot fat jar. Jackson doesn't do anything weird with metadata files, classloaders, etc. It plays well with others in a fat jar.
Given that you haven't shared much of your build files, the easiest way to figure out if something has excluded the Jackson XML module is to just run jar -xvf target/app.jar and inspect the output to see if it's in there.
If it's not, look for a Gradle equivalent of the Maven dependency plugin's dependency-tree target that will show you the whole transitive dependency tree. If it's being excluded you'll definitely see it missing from a dependency dump.
I have a parent artifact in my pom that is responsible for auto configuring a few properties based on #ConditionOnClass annotation. It also happens that I have a dependency of the same class which when included is causing some errors, even though I have not autowired any instance of the same in the code. This means that just including the dependency causes the #ConditionOnClass annotation to evaluate to true.
Does this mean that including the dependency automatically loads the classes of that jar in the classpath ? I do not have a clear understanding of how class loading works with Maven dependencies. Any pointer to relevant documentation or explanation of the same would be really useful.
For the longest time, I thought that in Java you either had one of two types of dependencies:
Required compile-time dependencies (dependencies always required at compile time)
Possibly optional runtime dependencies (dependency that can be
resolved at runtime)
Recently, I found out that compile dependencies can be optional too. For example, commons-beanutils is listed as an optional compile dependency of JXPath.
How can this work? Can a dependency really be used at the time of compilation yet remain fully optional?
EDIT: I might have been unclear. I'm looking for a case where a dependency is used at compile-time and is at the same time fully optional, or an explanation why such a dependency is impossible.
A class can compile to an interface but the implementation of that interface is not needed during compilation. The implementation is needed during runtime.
Example commons-logging, JPA, JDBC etc which are frameworks, an application can compile based on these. At runtime an implementation is needed to execute the code. Sample implementations - Common Bean utils, Oracle thin driver, Eclipse link etc.
An extensive quote from Maven documentation describes this quite clearly:
Optional dependencies are used when it's not possible (for whatever reason) to split a project into sub-modules. The idea is that some of the dependencies are only used for certain features in the project and will not be needed if that feature isn't used. Ideally, such a feature would be split into a sub-module that depends on the core functionality project. This new subproject would have only non-optional dependencies, since you'd need them all if you decided to use the subproject's functionality.
However, since the project cannot be split up (again, for whatever reason), these dependencies are declared optional. If a user wants to use functionality related to an optional dependency, they have to redeclare that optional dependency in their own project. This is not the clearest way to handle this situation, but both optional dependencies and dependency exclusions are stop-gap solutions.
Why use optional dependencies?
Optional dependencies save space and memory. They prevent problematic jars that violate a license agreement or cause classpath issues from being bundled into a WAR, EAR, fat jar, or the like.
How do optional dependencies work?
Project-A -> Project-B
The diagram above says that Project-A depends on Project-B. When A declares B as an optional dependency in its POM, this relationship remains unchanged. It's just like a normal build where Project-B will be added in Project-A's classpath.
Project-X -> Project-A
When another project (Project-X) declares Project-A as a dependency in its POM, the optional nature of the dependency takes effect. Project-B is not included in the classpath of Project-X. You need to declare it directly in the POM of Project X for B to be included in X's classpath.
A practical example: imagine that you are a developer of a library/framework SuperLib that is built as one superlib.jar. Your library provides multiple features. Its main feature (that most of the users use) is dependency injection based on a third-party di library. However, one of your classes - EmailApi - offers features to send e-mails, using a third-party email library. Since superlib is one artifact, it needs both di and email to be compiled.
Now put yourself in the position of a user who uses superlib. They are interested in the dependency injection features. This is the core role of your library, so the dependency between superlib and di would not be optional.
However, most users are not interested in sending emails and may be bothered by having a useless email library and its dependencies added to their application (which will cause size increase of their application and may cause a dependency version clash between the dependencies of email and dependencies of the user's application). Therefore, you would mark the dependency on email as optional. As long as the user does not use your EmailApi class, everything will run fine. However, if they do use EmailApi, they will need the email dependency, otherwise the application will fail at runtime with ClassNotFoundException for whichever class from email would be referenced in EmailApi. The user of your library will need to add the email dependency explicitly in their POM.
See also When to use <optional>true</optional> and when to use <scope>provided</scope>.
What you described is actually a feature of Maven, the build tool, but not Java itself.
Without build tools, using just 'javac' you need to specify all classes or interfaces that directly used in your code. Sure there are options for dynamic class loading and even runtime compilation, but thats not on topic.
One of use-cases with separation on interface and implementation is described in previous answer, another popular case is based on classpath scanning:
if some specific class is present in classpath and/or has specific annotation - an optional module will be loaded.
That's how Spring Boot modules are loaded.