Develop Reference

why is my maven sub dependency version for spark connector package different from others

I am trying to use a pom file from a existing project and I am getting an error "Cannot resolve org.yaml:snakeyaml:1.15"
What I find out about this error is that the com.datastax.spark:spark-cassandra-connector_2.11:2.5.0 uses a couple dependencies and a couple levels down it is using snakeyaml:1.15 which is quarantined by company proxy. Is there a way to specify for a given maven dependency that I want to use snakeyaml:1.16?
One thing I do not understand is that I look into the reference project that is also using com.datastax.spark:spark-cassandra-connector_2.11:2.5.0, it is using the updated com.datastax.oss:java-driver-core-shaded:4.9.0, which no longer requires snakeyaml:1.15
where as mine uses the old com.datastax.oss:java-driver-core-shaded:4.5.0
Why is it working in that pom? we have the same maven listing version for com.datastax.spark:spark-cassandra-connector_2.11:2.5.0
I see it has some exclusions but none addresses the snake yaml version or any of its parent dependencies.
Is there another section of the pom file that addresses this I am missing? please advise.
My pom
<scala.compat.version>2.11</scala.compat.version>
<spark.cassandra.version>2.5.0</spark.cassandra.version>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_${scala.compat.version}</artifactId>
<version>${spark.cassandra.version}</version>
</dependency>
where it goes wrong
however another project is using the correct shaded version com.datastax.oss:java-driver-core-shaded:4.9.0, which eliminates the snake dependency
working pom
<scala.compat.version>2.11</scala.compat.version>
<spark.cassandra.version>2.5.0</spark.cassandra.version>
<dependency>
<artifactId>spark-cassandra-connector_${scala.compat.version}</artifactId>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
<exclusion>
<artifactId>netty-all</artifactId>
<groupId>io.netty</groupId>
</exclusion>
<exclusion>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</exclusion>
</exclusions>
<groupId>com.datastax.spark</groupId>
<version>${spark.cassandra.version}</version>
</dependency>

You add an entry your <dependencyManagement> section of your POM, where you specify the version of snakeyaml that you want.
This will override all transitive version definitions of snakeyaml.

I suggest to switch to the SCC 2.5.2 (or at least 2.5.1) - there were fixes there regarding dependencies, it has driver upgraded to 4.10.0, etc. Another possibility is to use spark-cassandra-connector-assembly instead, with all dependencies included & shaded.

Spring-Boot: How can we remove some dependencies from Effective pom?

I am using spring-boot 2.0.3.RELEASE. When I am clicking on "show Effective POM" option by using IntelliJ IDEA, it loads Effective POM. And there I can see a few dependencies that my client don't want to have at there side.
Is there any way to tell Maven not to include these dependencies? How can we exclude dependencies from effective poms?

Maven provides a way to exclude dependencies with the exclude tag
Here is an example taken from the documentation website https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html
<dependencies>
<dependency>
<groupId>sample.ProjectA</groupId>
<artifactId>Project-A</artifactId>
<version>1.0</version>
<scope>compile</scope>
<exclusions>
<exclusion> <!-- declare the exclusion here -->
<groupId>sample.ProjectB</groupId>
<artifactId>Project-B</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
The idea is to locate parent dependencie from where you are getting deps you don't want and add an exclusion tag.
If they are needed in runtime you can specify the scope to provided
<dependency>
<groupId>sample.ProjectA</groupId>
<artifactId>Project-A</artifactId>
<version>1.0</version>
<scope>provided</scope>
</dependency>
That will tell maven to use the deps to compile but not no include them in the target package, and they will be provided in the production environment by the JVM executing the code.
Hope this helps

maven: how to disable certain dependencies?

Well, I'm not talking about the well-known commons-logging problem, I know I can disable it by setting the 99.0-does-not-exist version.
My problems is, some packages are contained in different dependencies, say, aspectjlib is contained both in org.aspectj:aspectjlib and aspectj:aspectjlib. In some cases, transitive dependencies may introduce the two jars at the same time, while of different versions, e.g., org.aspectj:aspectjlib:1.7.3, aspectj:aspectjlib:1.6.1. And mis-loading aspectj:aspectjlib:1.6.1 accidentally is not my intention. So is there a way like commons-logging that I can disable aspectj:aspectjlib completely?
I tried the same trick using 99.0-does-not-exist, only to find an error from maven:
[ERROR] Failed to execute goal on project XXX: Could not resolve
dependencies for project XXX:jar:1.0.0-SNAPSHOT: The following
artifacts could not be resolved:
aspectj:aspectjlib:jar:99.0-does-not-exist,
aspectj:aspectjrt:jar:99.0-does-not-exist,
aspectj:aspectjweaver:jar:99.0-does-not-exist: Could not find artifact
aspectj:aspectjlib:jar:99.0-does-not-exist in tbmirror
(http://mvnrepo.taobao.ali.com/mvn/repository) -> [Help 1]

Well, although some repositories do provide 99.0-does-not-exist for logging system dependencies like log4j, slf4j-log4j, commons-logging, etc., this is not a universal solution.
I find a solution to do this: use 'provided' scope.
To clarify, in my example above, I have two conflicting dependencies: org.aspectj:aspectjlib:1.7.3, aspectj:aspectjlib:1.6.1, I want to disable aspectj:aspectjlib:1.6.1, I only need to put this in top-level pom:
<dependency>
<groupId>aspectj</groupId>
<artifactId>aspectjlib</artifactId>
<version>1.6.1</version>
<scope>provided</scope>
</dependency>
in this way, aspectj:aspectjlib:1.6.1 will never appear in the final built lib.

You can use Maven's dependency exclusions to eliminate the version you don't want. Using your example:
<dependencies>
<dependency>
<groupId>com.example</groupId>
<artifactId>includes-new-aspectj</groupId>
</dependency>
<dependency>
<groupId>com.example</groupId>
<artifactId>includes-old-aspectj</groupId>
<exclusions>
<exclusion>
<groupId>org.aspectj<groupId>
<artifactId>aspectjlib</artifactId>
<exclusion>
</exclusions>
</dependency>
</dependencies>
Alternatively, you can simply pin the version you desire using dependency management:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjlib</artifactId>
<version>1.7.3</version>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>com.example</groupId>
<artifactId>includes-new-aspectj</groupId>
</dependency>
<dependency>
<groupId>com.example</groupId>
<artifactId>includes-old-aspectj</groupId>
</dependency>
</dependencies>
If you are not sure which dependencies include which versions, you can use this to discover that info:
mvn dependency:tree -Dincludes='org.aspectj:aspectjlib'

There is no 99.0 version for aspectj:aspectjlib, your project is configured to use wrong version, check for 99.0 in your pom.xml

JSR 181: error in pom.xml

I got a strange error in my pom.xml. Maven (I'm using Maven 2) is signaling Missing artifact javax.jws:jsr181:jar:1.0, even if I get the corresponding dependency in my pom.xml:
<dependency>
<groupId>javax.jws</groupId>
<artifactId>jsr181</artifactId>
<version>1.0</version>
</dependency>
What could possibly be the cause of this error?

Ok, I found the solution to the problem. I think the way to find it could be interesting, too.
When I look on mvnrepository.com, the pom file on the repository pointed on an URL on bea.com, which is not available anymore. So I had to change to the maintenance release, like Brian Agnew suggested. And of course, update some other dependencies in my pom.xml, which needed the obsolete version in their own dependencies. Maven comes with a cost...
<!-- https://mvnrepository.com/artifact/javax.jws/jsr181-api -->
<dependency>
<groupId>javax.jws</groupId>
<artifactId>jsr181-api</artifactId>
<version>1.0-MR1</version>
</dependency>
<dependency>
<groupId>javax.xml.ws</groupId>
<artifactId>jaxws-api</artifactId>
<version>2.1-1</version>
<exclusions>
<exclusion>
<groupId>javax.jws</groupId>
<artifactId>jsr181</artifactId>
</exclusion>
</exclusions>
</dependency>

Looking at my repository, I think you want:
<dependency>
<groupId>sun-jaxws</groupId>
<artifactId>jsr181-api</artifactId>
<version>1.0</version>
</dependency>

How to mark main dependency as "compile" and sub-dependencies as "provide"?

One of my Android projects requires oauth.signpost artifact, so I have something like this in my pom.xml:
...
<dependency>
<groupId>oauth.signpost</groupId>
<artifactId>signpost-core</artifactId>
<version>1.2.1.1</version>
<scope>compile</scope>
</dependency>
...
The problem is that this depends on more artifacts, for instance: org.apache.httpcomponents:httpclient:4.0.1. The problem is that Android already provides that package and thus it makes my building process crash.
So, what I usually do is going to the dependencies configuration on IntelliJ and manually mark those redundant artifacts as provide instead of compile:
This process is annoying, not only because I have a lot of Maven dependencies, but also because sometimes IntelliJ forgets what dependencies were market as provide and it will mark'em all as compile.
The only solution I see is to specify, in the pom.xml, which dependencies are provide and which are compile. But there are a lot of them, so it would take some time (also, I'd have to manually check which version is needed for each artifact, etc.).
So, is it a way to tell Maven to mark sub-dependencies as provide, while keeping main dependencies as compile?

There is the option to add dependency exclusions.
Consider the following:
Let's assume you have two projects - bar and foo.
Bar dependends on log4j.
You would like foo to depend on a different version of log4j.
This would be your pom.xml in the foo module:
<project>
<groupId>kung.fu</groupId>
<artifactId>foo</artifactId>
...
<dependencies>
<dependency>
<groupId>kung.fu</groupId>
<artifactId>bar</artifactId>
<version>1.0</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
</project>