Develop Reference

How to list all dependencies of a package from maven (including scopes)

I have a graph with 40K artifacts and I can list all possible dependencies of a package (I do so by parsing a list of effective poms)
For example, I have the following for this package:
There's 2 dependencies without taking in mind different versions.
I would like to show that this results are valid by showing that maven also lists these dependencies for this package. But when I use mvn dependency:tree after I add the com.google.guava:guava:14.0.1, I get no dependencies listed.
This is the pom file of the package:
It clearly has those 2 dependencies, but their scopes are provided. Even if I use -Dinclude=provided or -Dscope=provided as a parameter, I still cannot list them.
So, how do I list all dependencies of a package no matter the scope used?

Use Analyze Dependencies... action in the Maven tool window:
It will show the list of dependencies in the project with their scopes and usages in project:

Scope provided means it's provided at runtime, which implies that it's not a package dependency:
A dependency with this scope is added to the classpath used for compilation and test, but not the runtime classpath. It is not transitive.

How do optional compile-time dependencies work?

For the longest time, I thought that in Java you either had one of two types of dependencies:
Required compile-time dependencies (dependencies always required at compile time)
Possibly optional runtime dependencies (dependency that can be
resolved at runtime)
Recently, I found out that compile dependencies can be optional too. For example, commons-beanutils is listed as an optional compile dependency of JXPath.
How can this work? Can a dependency really be used at the time of compilation yet remain fully optional?
EDIT: I might have been unclear. I'm looking for a case where a dependency is used at compile-time and is at the same time fully optional, or an explanation why such a dependency is impossible.

A class can compile to an interface but the implementation of that interface is not needed during compilation. The implementation is needed during runtime.
Example commons-logging, JPA, JDBC etc which are frameworks, an application can compile based on these. At runtime an implementation is needed to execute the code. Sample implementations - Common Bean utils, Oracle thin driver, Eclipse link etc.

An extensive quote from Maven documentation describes this quite clearly:
Optional dependencies are used when it's not possible (for whatever reason) to split a project into sub-modules. The idea is that some of the dependencies are only used for certain features in the project and will not be needed if that feature isn't used. Ideally, such a feature would be split into a sub-module that depends on the core functionality project. This new subproject would have only non-optional dependencies, since you'd need them all if you decided to use the subproject's functionality.
However, since the project cannot be split up (again, for whatever reason), these dependencies are declared optional. If a user wants to use functionality related to an optional dependency, they have to redeclare that optional dependency in their own project. This is not the clearest way to handle this situation, but both optional dependencies and dependency exclusions are stop-gap solutions.
Why use optional dependencies?
Optional dependencies save space and memory. They prevent problematic jars that violate a license agreement or cause classpath issues from being bundled into a WAR, EAR, fat jar, or the like.
How do optional dependencies work?
Project-A -> Project-B
The diagram above says that Project-A depends on Project-B. When A declares B as an optional dependency in its POM, this relationship remains unchanged. It's just like a normal build where Project-B will be added in Project-A's classpath.
Project-X -> Project-A
When another project (Project-X) declares Project-A as a dependency in its POM, the optional nature of the dependency takes effect. Project-B is not included in the classpath of Project-X. You need to declare it directly in the POM of Project X for B to be included in X's classpath.
A practical example: imagine that you are a developer of a library/framework SuperLib that is built as one superlib.jar. Your library provides multiple features. Its main feature (that most of the users use) is dependency injection based on a third-party di library. However, one of your classes - EmailApi - offers features to send e-mails, using a third-party email library. Since superlib is one artifact, it needs both di and email to be compiled.
Now put yourself in the position of a user who uses superlib. They are interested in the dependency injection features. This is the core role of your library, so the dependency between superlib and di would not be optional.
However, most users are not interested in sending emails and may be bothered by having a useless email library and its dependencies added to their application (which will cause size increase of their application and may cause a dependency version clash between the dependencies of email and dependencies of the user's application). Therefore, you would mark the dependency on email as optional. As long as the user does not use your EmailApi class, everything will run fine. However, if they do use EmailApi, they will need the email dependency, otherwise the application will fail at runtime with ClassNotFoundException for whichever class from email would be referenced in EmailApi. The user of your library will need to add the email dependency explicitly in their POM.
See also When to use <optional>true</optional> and when to use <scope>provided</scope>.

What you described is actually a feature of Maven, the build tool, but not Java itself.
Without build tools, using just 'javac' you need to specify all classes or interfaces that directly used in your code. Sure there are options for dynamic class loading and even runtime compilation, but thats not on topic.
One of use-cases with separation on interface and implementation is described in previous answer, another popular case is based on classpath scanning:
if some specific class is present in classpath and/or has specific annotation - an optional module will be loaded.
That's how Spring Boot modules are loaded.

Why sometimes dependency not contain version property in pom.xml?

I am a newbie of Maven, currently reading Hadoop source code, and found something interesting in some pom.xml files:
Some of the dependency node do not contain version node at all.
Question: why is it like this?
for instance, this pom.xml.

Because specific version of dependency in parent pom.xml file
https://github.com/apache/hadoop/blob/trunk/pom.xml
Reference: https://maven.apache.org/guides/introduction/introduction-to-the-pom.html

As I commented at first, a pom file can have a parent (via inheritance) and such a parent may provide some governance and harmonization across all of its children. A classic case is to provide versioning for certain dependencies via a dependencyManagement section.
is used by POMs to help manage dependency information across all of its children. If the my-parent project uses dependencyManagement to define a dependency on junit:junit:4.0, then POMs inheriting from this one can set their dependency giving the groupId=junit and artifactId=junit only, then Maven will fill in the version set by the parent. The benefits of this method are obvious. Dependency details can be set in one central location, which will propagate to all inheriting POMs.
The mentioned pom has indeed a parent pom:
<parent>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-project-dist</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../../hadoop-project-dist</relativePath>
</parent>
Which in chain has another parent pom file which defines several dependencies as part of its dependencies management section.
If you really want to check the effective (merged) pom your build is using, you could run:
mvn help:effective-pom -Doutput=effective-pom.xml
And the maven-help-plugin will produce an additional pom as specified by the command above, merging the current pom file and all of its anchestors.

In Maven you can inherit from parents folder in order to merge or inherit some properties. This can be the version of the modules. Usually you have a "super" POM in the root folder of your project and you put there all the commons dependencies in order to controll them in an easier way. I.e. If you must change one module version, you only need to change in the "super" POM and not in each POM inside each subfolder that need it. If you need more information about POM inheritance the documentation has a couple of useful examples.
https://maven.apache.org/guides/introduction/introduction-to-the-pom.html#Project_Inheritance

Maven: Resolving Duplicate Dependencies

I'm developing an application that will be used internally at our company. In order for it to interop with our other internal systems I have to use some maven dependencies that we use internally, but this is causing some issues with using some external 3rd party dependencies that I also need.
So essentially my pom looks like this:
<dependencies>
<dependency>
internal-framework-artifact
</dependency>
<dependency>
necessary-third-party-artifact
</dependency>
</dependencies>
I've come to find that both of these dependencies have the apache's commons-collections as one of their own dependencies (among a large number of others, but we'll just keep it at one for this question's simplicity).
If I place exclusion rules on both of them for the commons-collections pom I can compile the project, but my resulting jar won't have access to either version of commons-collections and will just result in a java.lang.NoClassDefFoundError exception. Removing the exclusion rule on either of them just results in a mvn compiler error:
[WARNING] Rule 2: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:
I've been looking through various so q/a's and I can't really seem to find something that's 100% relevant to my situation. I'm really at a loss as to how to resolve this. Am I missing something really obvious?

I've never actually used the maven-shade-plugin for shading, but I think this is the exact use case it was designed for.
Create a new project that uses the maven-shade-plugin (see: http://maven.apache.org/plugins/maven-shade-plugin/) to produce an uber-jar version of internal-framework-artifact which contains that classes in internal-framework-artifact and all its dependencies. Configure the plugin so that it relocates all the classes that are also dependencies of necessary-third-party-artifact to some non-conflicting package names. This new project should produce a .jar with a different name, something like internal-framework-artifact-with-dependencies.
Now modify your original pom so that it is dependent on internal-framework-artifact-with-dependencies instead, and it should work.

Why does a dependency with scope "provided" hide transitive dependencies in Maven?

I have three modules in my Maven project (this is slightly simplified):
model contains JPA annotated entity classes
persistence instantiates an EntityManager and calls methods on it
application creates instances of the classes in model, sets some values and passes them to persistence
model and persistence obviously depend on javax.persistence, but application shouldn't, I think.
The javax.persistence dependency is moved to a top-level POM's dependencyManagement section because it occurs in a number of submodules where I only reference that entry.
What's surprising to me is that I have to reference the dependency in application when I set its scope to provided, whereas I don't have to when its scope is compile.
With a scope of provided, if I don't list it in the dependencies for application, the build fails with an error message from javac:
com.sun.tools.javac.code.Symbol$CompletionFailure: class file for javax.persistence.InheritanceType not found
What's going on?

model and persistence obviously depend on javax.persistence, but application shouldn't, I think.
That's true. But transitive dependencies resolution has nothing to do with your problem (and actually, javax.persistence is provided to model and persistence on which application depends with a compile scope so it's omitted as documented in 3.4.4. Transitive Dependencies).
In my opinion, you are victim of this bug: http://bugs.sun.com/view_bug.do?bug_id=6550655
I have the same issues with an EJB3
entity that uses the Inheritance annotation:
#Inheritance(strategy=InheritanceType.SINGLE_TABLE)
A client class using this entity won't
compile when the ejb3 annatations are
not on the classpath, but crash with
the following message:
com.sun.tools.javac.code.Symbol$CompletionFailure:
class file for
javax.persistence.InheritanceType not
found
[...]
Note that is a special case of bug 6365854 (that is reported to be fixed); the problem here seems to be that the annotation is using an enum as its value.
The current workaround is to add the missing enum to the CLASSPATH.
In your case, the "less worse" way to do that would be to add javax.persistence as provided dependency to the application module. But that's a workaround to the JVM bug, application shouldn't need that dependency to compile.

umm, because provided dependencies are not transitive? that's builtin behavior for maven.

The dependencyManagement section declares what dependencies will look like if you use them, not that you will use them. So you still need to declare a minimal dependency declaration to have the configuration applied in your child project. See the dependency management section of the Maven book for details.
The minimum required is typically the groupId and the artifactId.
If you want to inherit the configuration without declaring it at all, you should define it in the parent's dependencies section rather than dependencyManagement