Java giving NoClassDefFoundError when importing maven package

Java giving NoClassDefFoundError when importing maven package - java

I have a simple java code, App.java. I am trying to import the apache commons Fraction class from the apache commons math3 library. So, as a sample code, I put this in my App.java:
package myApp;
import org.apache.commons.math3.fraction.Fraction;
public class App
{
public static void main( String[] args )
{
Fraction f = new Fraction(2, 3);
System.out.println( "Hello World!" );
System.out.println( f );
}
}
Using the following POM:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>myApp</groupId>
<artifactId>Proj1</artifactId>
<version>1.0-SNAPSHOT</version>
<name>Proj1</name>
<url>http://www.example.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-math3 -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
<version>3.6.1</version>
</dependency>
</dependencies>
<build>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version>
</plugin>
<!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
<!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
</plugin>
<plugin>
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<configuration>
<!-- put your configurations here -->
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
I have tried to put the plugin suggested by utdemir in his answer to a similar question, but in my case, when I run:
mvn package
java -cp target/Proj1-1.0-SNAPSHOT.jar myApp.App
While it apparently compiles properly, I get the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/math3/fraction/Fraction
at myApp.App.main(App.java:8)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.math3.fraction.Fraction
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:636)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519)
... 1 more
What should I do?

Its much easier to run your app using mvn, rathen than by calling java directly. This is because in the later case you need to manually set the classpath to the correct value using the -cp flag. This can lead to errors, like in your case.
Here's how you can run your app using mvn
mvn exec:java -Dexec.mainClass="myApp.App"
More details can be found here

Related

Errors appear using Jsoup: Exception in thread "main" javax.net.ssl.SSLException: Tag mismatch

I'm trying to use a simple jsoup program. I'm using eclipse and created a maven project with the jsoup-1.8.1.jar in my Maven depedencies.
package com.testing.com.testing.maven.eclipse;
import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
public class Main{
public static void main( String[] args ) throws IOException{
Document doc = Jsoup.connect("https://en.wikipedia.org/wiki/Main_Page").get();
String title = doc.title();
System.out.println("title is: " + title);
}
}
But when I run it these errors appears
Exception in thread "main" javax.net.ssl.SSLException: Tag mismatch!
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:369)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:312)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:451)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:422)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:574)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:142)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:434)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:181)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:170)
at com.testing.com.testing.maven.eclipse.Main.main(Main.java:8)
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:792)
at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1122)
at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1059)
at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:946)
at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:501)
at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:799)
at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:735)
at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:673)
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2500)
at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1930)
at java.base/sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:260)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:181)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
... 12 more
This is what my pom.xml looks like:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.testing</groupId>
<artifactId>com.testing.maven.eclipse</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>com.testing.maven.eclipse</name>
<!-- FIXME change it to the project's website -->
<url>http://www.example.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>16</maven.compiler.source>
<maven.compiler.target>16</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.8.1</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.11.0</version>
</dependency>
</dependencies>
<build>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version>
</plugin>
<!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
<!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
</plugin>
<plugin>
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
I'm using JDK 16.0.2
I tried checking similar problems in stack overflow but they have different errors. Can anyone suggest a solution to this?

You can try to install manually
mvn install:install-file -Dfile= -DgroupId=
-DartifactId= -Dversion= -Dpackaging=

no main manifest attribute error for my maven project VSCode

After I build my java project using maven and try to run the executable (there are no errors or warnings during building), I get the following error:
C:\Users\User\Documents\Java\clothes-assistant\target>java -jar clothes-assistant-0.0.1-SNAPSHOT.jar
no main manifest attribute, in clothes-assistant-0.0.1-SNAPSHOT.jar
The following is my pom.xml file, I think I'm messing up on the tag but I've taken snippets from other StackOverflow forums and they did something similar so I don't know where I could be messing up.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>me.louisspaghetti</groupId>
<artifactId>clothes-assistant</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>clothes-assistant</name>
<!-- FIXME change it to the project's website -->
<url>http://www.example.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<archive>
<manifest>
<mainClass>me.louisspaghetti.App</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
<!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
</plugin>
<plugin>
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
NOTE: I'm using VSCode so I don't have access to any IDE-Specific build features

You are defining your main class in the clean plugin plugin configuration, which makes no sense; you should configure it in the maven-jar-plugin-configuration instead:
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<archive>
<manifest>
<mainClass>me.louisspaghetti.App</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
That will instruct maven to put a declaration of the main class in the generated jarfile, in the MANIFEST.MF file:
$ unzip -l target/clothes-assistant-0.0.1-SNAPSHOT.jar
Archive: target/clothes-assistant-0.0.1-SNAPSHOT.jar
Length Date Time Name
--------- ---------- ----- ----
131 01-09-2020 09:31 META-INF/MANIFEST.MF
0 01-09-2020 09:31 META-INF/
which will contain something like:
Manifest-Version: 1.0
Created-By: Apache Maven
Build-Jdk: 1.8.0
Main-Class: me.louisspaghetti.App
that value is what java -jar will use to find the main class in the jarfile.

ANTLR4 CharStream class missing

I am using ANTLR4 to parse command lines for my Java Shell project.
When I run a JUnit test in VSCode, everything is fine. However, when I am building the Docker image and I try to run the shell in interactive mode, I get this error:
Error: Unable to initialize main class uk.ac.ucl.jsh.Jsh
Caused by: java.lang.NoClassDefFoundError: org/antlr/v4/runtime/CharStream
Is there any issue with my pom.xml file, or is the problem coming from somewhere else? This is my pom.xml file:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>uk.ac.ucl.jsh</groupId>
<artifactId>jsh</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>jsh</name>
<url>http://maven.apache.org</url>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime</artifactId>
<version>4.7.2</version>
</dependency>
</dependencies>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<argLine>-XX:MaxPermSize=512m</argLine>
</properties>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.4</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>uk.ac.ucl.jsh.Jsh</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
<version>4.7.2</version>
<executions>
<execution>
<goals>
<goal>antlr4</goal>
</goals>
</execution>
</executions>
<configuration>
<visitor>true</visitor>
<listener>true</listener>
<outputDirectory>
${basedir}/src/main/java/
</outputDirectory>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
<reporting>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-pmd-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<reportSets>
<reportSet>
<reports>
<!-- select non-aggregate reports -->
<report>report</report>
</reports>
</reportSet>
</reportSets>
</plugin>
</plugins>
</reporting>
</project>

This doesn't really look like an ANTLR problem. You'll need to look into how you're doing your build and how you're putting your Docker image together.
There are a couple of solutions to this.
Look into how to have Maven produce an Uber jar (i.e. a jar file that bundles up all of it's dependencies)
Ensure that your Docker image has the ANTLR runtime included and in your class path.
The first of those is probably the more "normal" solutions these days.

Spark fails with NoClassDefFoundError for org.apache.kafka.common.serialization.StringDeserializer

I am developing a generic Spark application that listens to a Kafka stream using Spark and Java.
I am using kafka_2.11-0.10.2.2, spark-2.3.2-bin-hadoop2.7 - I also tried several other kafka/spark combinations before posting this question.
The code fails at loading StringDeserializer class:
SparkConf sparkConf = new SparkConf().setAppName("JavaDirectKafkaWordCount");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(2));
Set<String> topicsSet = new HashSet<>();
topicsSet.add(topics);
Map<String, Object> kafkaParams = new HashMap<>();
kafkaParams.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, brokers);
kafkaParams.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
kafkaParams.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
kafkaParams.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
The error I get is:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/StringDeserializer
From Why does Spark application fail with "Exception in thread "main" java.lang.NoClassDefFoundError: ...StringDeserializer"? it seems that this could be a scala version mismatch issue, but my pom.xml doesn't have that issue:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>yyy.iot.ckc</groupId>
<artifactId>sparkpoc</artifactId>
<version>1.0-SNAPSHOT</version>
<name>sparkpoc</name>
<!-- FIXME change it to the project's website -->
<url>http://www.example.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<java.version>1.8</java.version>
<spark.scala.version>2.11</spark.scala.version>
<spark.version>2.3.2</spark.version>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
<build>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version>
</plugin>
<!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
<!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
</plugin>
<plugin>
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
The submission script I use is:
./bin/spark-submit \
--class "yyy.iot.ckc.KafkaDataModeler" \
--master local[2] \
../sparkpoc/target/sparkpoc-1.0-SNAPSHOT.jar
Can anyone please point me in the right direction as to where I am going wrong?

Spark runs the program as by running an instance of a JVM. So if the libraries (JARs) are not in the classpath of that JVM we run into this runtime exception. The solution is to package all the dependent JARs along with main JAR. The following build script will work for that.
Also, as mentioned in https://stackoverflow.com/a/54583941/1224075 the scope of the spark-core and spark-streaming libraries need to be declared as provided. This is because some of the libraries are implicitly provided by the Spark JVM.
The build section of the POM which worked for me -
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2.1</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

You need to use the Maven Shade Plugin to package the Kafka clients along with your Spark application, then you can submit the shaded Jar, and the Kafka serializers should be found on the classpath.
Also, make sure you set the provided Spark packages
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>

Maven running tests annotated with junit #Ignore when building Java project

I've annotated a bad test with #Ignore, both on the method-level and on the class level. When running the test through the command line (I've tried "mvn clean install", "mvn test", "mvn clean install -DskipTests; mvn test"), however, the #Ignore annotation is ignored, the test is run, and- as it is a bad test- it fails.
Here is the test:
public class UserTest extends PersistentTestBase {
#Test
#Ignore
public void testPersistence() {
...
}
}
And here is my pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
...
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
...
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
...
<!-- Testing -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jmock</groupId>
<artifactId>jmock-junit4</artifactId>
<version>2.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jmock</groupId>
<artifactId>jmock-legacy</artifactId>
<version>2.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymock</artifactId>
<version>3.4</version>
<scope>test</scope>
</dependency>
...
<!-- For dep management, see Mykong.com's "How to create a jar file with Maven" -->
<build>
<plugins>
<!-- Necessary to force language level of Java 8 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<junitArtifactName>junit:junit-dep</junitArtifactName>
<useFile>false</useFile>
<trimStackTrace>false</trimStackTrace>
<reuseForks>false</reuseForks>
<forkCount>1</forkCount>
</configuration>
</plugin>
<!-- Packages into a jar, looking for deps in target/dependency-jars -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.2</version>
<configuration>
<excludes>
<exclude>**/log4j.properties</exclude>
</excludes>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>...Application</mainClass>
<classpathPrefix>dependency-jars</classpathPrefix>
</manifest>
</archive>
</configuration>
</plugin>
<!-- Moves all compiled deps to target/dependency-jars -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.5.1</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<includeScope>runtime</includeScope>
<outputDirectory>${project.build.directory}/dependency-jars</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<configuration>
<mainClass>...Application</mainClass>
</configuration>
</plugin>
</plugins>
</build>
Worth noting that the test is skipping when the class is run in Intellij IDEA 14, and that there are some other issues with tests failing on mvn test that are passing in Intellij (but are out of the scope of this question). Thank you for the help!

For those who are working with Junit 5.
#Ignore did not work formaven surefire neither Intellij tests even if I had already included junit-vintage-engine
I used #Disabled and it worked for maven at least!

Wemu posted the solution:
I think you can simply remove:
junit:junit-dep from the
surefire config – wemu Nov 20 at 7:10
Removed the element and all tests passed!

I had a same problem with the JUNIT library. After so many tries, it worked with the Pom.xml below:
<repositories>
<repository>
<id>junit</id>
<url>http://junit.org</url>
</repository>
</repositories>
<dependencies>
<dependency><groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version></dependency>
</dependencies>

For me the solution was to simply update the JUnit version to 4.13:
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>

Develop Reference

Java is a programming language and computing platform first released by Sun Microsystems in 1995.

Java giving NoClassDefFoundError when importing maven package - java

Related

Errors appear using Jsoup: Exception in thread "main" javax.net.ssl.SSLException: Tag mismatch

no main manifest attribute error for my maven project VSCode

ANTLR4 CharStream class missing

Spark fails with NoClassDefFoundError for org.apache.kafka.common.serialization.StringDeserializer

Maven running tests annotated with junit #Ignore when building Java project

Categories

Resources