Dataflow and Bigtable dependency conflict - java

The error I'm getting:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/beam/sdk/transforms/DoFn
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.transforms.DoFn
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
My pom dependencies:
<dependencies>
<dependency>
<groupId>com.google.cloud.bigtable</groupId>
<artifactId>bigtable-hbase-beam</artifactId>
<version>1.11.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.13.0</version>
<scope>runtime</scope>
</dependency>
</dependencies>
I assumed that the issue was coming from the two dependencies using different versions of the beam sdk, so I changed the beam-runners-google-cloud-dataflow-java version to 2.11.0, which uses the same beam sdk version. But I got the same error.
Any help would be greatly appreciated!

How are you running the pipeline? I'm guessing that the issue is that the beam jars are not on your classpath. You either need to use the maven exec plugin to invoke your main() with all of the maven dependencies on the classpath or you need to create a fat jar using something like the maven shade plugin or the maven assembly plugin. Please see here for example of how to use the maven exec plugin to run a pipeline:
https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/blob/a017a237fa8dae764451a658150190743c1b4127/java/dataflow-connector-examples/pom.xml#L146-L161

Related

When I am trying to run mvn dependency:tree , I am getting : java.lang.NoClassDefFoundError: org/sonatype/aether/version/VersionConstraint

I have no idea why this is happening. I tried deleting my m2 repository as well.
I tried with various versions of maven plugins(2.8) as well.
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.1.1</version>
</dependency>
Here is the complete stacktrace :
Error injecting: org.apache.maven.shared.dependency.graph.internal.Maven3DependencyGraphBuilder
java.lang.NoClassDefFoundError: org/sonatype/aether/version/VersionConstraint
at java.lang.Class.getDeclaredMethods0 (Native Method)
at java.lang.Class.privateGetDeclaredMethods (Class.java:2701)
at java.lang.Class.getDeclaredMethods (Class.java:1975)
at com.google.inject.spi.InjectionPoint.getDeclaredMethods (InjectionPoint.java:766)
at com.google.inject.spi.InjectionPoint.getInjectionPoints (InjectionPoint.java:683)
at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields (InjectionPoint.java:378)
at com.google.inject.internal.ConstructorBindingImpl.getInternalDependencies
(ConstructorBindingImpl.java:181)
Caused by: java.lang.ClassNotFoundException: org.sonatype.aether.version.VersionConstraint
at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass (SelfFirstStrategy.java:50)
at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass (ClassRealm.java:271)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:247)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:239)
at java.lang.Class.getDeclaredMethods0 (Native Method)
at java.lang.Class.privateGetDeclaredMethods (Class.java:2701)
at java.lang.Class.getDeclaredMethods (Class.java:1975)
at com.google.inject.spi.InjectionPoint.getDeclaredMethods (InjectionPoint.java:766)
at com.google.inject.spi.InjectionPoint.getInjectionPoints (InjectionPoint.java:683)
at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields (InjectionPoint.java:378)
You can solve by using an older version of the plugin as indicated by the official page recalled by the error log:
mvn org.apache.maven.plugins:maven-dependency-plugin:2.8:tree
Update Maven version, at least to 3.0.x+ or 3.1.x+ see here
This error is caused by Maven 3.1-alpha-1 migration from Sonatype
Aether to Eclipse Aether (MNG-5354), which is an incompatible change
for some plugins.
Other report here claim to use Maven 3.3.0, so in any case, try with latest stable maven first (you can get it from here)
Side question: are you developing maven plugin? Usually you shouldn't declare dependencies on plugins in your pom.xml

java.lang.NoClassDefFoundError while using SourceAFIS in my Java Application using Maven [duplicate]

I am trying simple Maven app with Log4J ver 2-beta 9. In my pom.xml file I have these two dependencies (as is mentioned in Log4J Maven webpage):
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.0-beta9</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.0-beta9</version>
</dependency>
Eclipse sees Log4J library:
But when I package the app and run it this Exception is thrown:
java -cp target/notification-1.0.0.jar com.example.Sandbox
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/LogManager
at com.example.Sandbox.<clinit>(Sandbox.java:13)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.LogManager
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 1 more
I searched for this Exception and it seems to be related to CLASSPATH variable.
How should be CLASSPATH set for Maven project?
Maven is a build tool. It doesn't help you much running the final application.
You can use mvn exec:java -Dexec.mainClass="com.example.Sandbox" to run your app (see the question Maven Run Project) but this gets tedious when you have to pass arguments to it.
You can get the classpath that Maven used to compile your application with mvn dependency:build-classpath
That will print the classpath to the console. Note that it will be missing target/notification-1.0.0.jar
Another useful tool in this area is the assembly plugin; it will create one very big JAR with all the dependencies rolled into a single file when you specify the descriptor jar-with-dependencies.

IntelliJ & Jersey: java.lang.NoClassDefFoundError

I just tried to use the jersey-server api in a java application.
It's a maven project and I already added the mysql-connector api which works fine.
When the application which is compressed into a jar (Intellij-Artifacts) runs, it throws the exception:
Exception in thread "main" java.lang.NoClassDefFoundError:
com/sun/jersey/api/container/httpserver/HttpServerFactory
at de.myleak.server.Server.init(Server.java:19)
at de.myleak.core.Core.init(Core.java:87)
at de.myleak.core.Core.<init>(Core.java:51)
at de.myleak.core.Core.main(Core.java:40)
Caused by: java.lang.ClassNotFoundException:
com.sun.jersey.api.container.httpserver.HttpServerFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 4 more
I added the dependency to the pom.xml
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-server</artifactId>
<version>1.19.4</version>
</dependency>
I also added the maven dependency in the project structure
screenshot

Nutch: NoClassDefFoundError: org/springframework/retry/RetryCallback

I am running the normal Nutch indexing, it used to work 2 days ago perfectly but today out of nowhere it just stopped working with the previous error. Following the stack error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/retry/RetryCallback
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:161)
at org.apache.nutch.indexer.IndexWriters.<init>(IndexWriters.java:55)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:121)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:228)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:238)
Caused by: java.lang.ClassNotFoundException: org.springframework.retry.RetryCallback
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
I ended up seeing that the error occurred at this exact piece of code :
Object object = extensionClazz.newInstance();
Another question on SO but with different project was able to fix the issue by adding the spring-framework-messaging dependency
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-messaging</artifactId>
<version>4.2.4.RELEASE</version>
</dependency>
well i am using nutch version 1.12 with ivy for dependency management so in my config is the following:
<dependency org="org.springframework.retry" name="spring-retry" rev="1.2.0.RELEASE">
<exclude module="spring-beans"/>
<exclude module="spring-context"/>
<exclude module="spring-core"/>
<exclude module="spring-aop"/>
</dependency>
and even with adding the spring-messaging dependency it's still not working.
Any ideas?
These kind of errors might come when you have more than one version of same jar (same class) in your classpath, You can see the dependency tree by executing
<report conf="compile" /> for ivy projects
And
mvn dependency:tree for maven projects
Narrow down to the version that you want and you can exclude the other jar by explicitly excluding that jar from the module which is importing it .
No idea what went wrong but this was able to fix the problem
ant clean
ant runtime

Unable to resolve a dependency on URLUtil, conflicting requirements

I'm getting the following error when trying to compile my project:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/httpclient/util/URIUtil
at org.apache.commons.vfs2.provider.URLFileName.getPathQueryEncoded(URLFileName.java:91)
at org.apache.commons.vfs2.provider.URLFileName.getURIEncoded(URLFileName.java:161)
at org.apache.commons.vfs2.provider.url.UrlFileObject.createURL(UrlFileObject.java:74)
at org.apache.commons.vfs2.provider.url.UrlFileObject.doAttach(UrlFileObject.java:63)
at org.apache.commons.vfs2.provider.AbstractFileObject.attach(AbstractFileObject.java:1505)
at org.apache.commons.vfs2.provider.AbstractFileObject.getType(AbstractFileObject.java:489)
.... (More project-scope traces)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.util.URIUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
I have the following maven dependencies in my project, among others:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.4</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-vfs2</artifactId>
<version>2.0</version>
</dependency>
I absolutely must depend on a HttpClient version 4.3.4 or greater because of other dependencies and various bugs we ran into with previous versions. After a bit of research, I discovered the class in question (URLUtil) was removed from HttpClient after HttpClient 3.1.
Is there any way to resolve this? Or another library I could use instead of vfs2? I'm attempting to write files over sftp using both user/pass and key authentication schemes.
As it turns out, the artifacts are different between the 3.x and 4.x releases of HttpClient. All I had to do was include:
<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
</dependency>
Silly me, thinking that they'd override each-other.

Categories

Resources