Develop Reference

java.io.InvalidClassException: local class incompatible - Apache Spark Submit issue for more than one worker

We have been trying to run a java application in Apache Spark using master-worker architecture. Below is the pom.xml for java application:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>Final</groupId>
<artifactId>DLSH</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.0.0</version>
<scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.0.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.12</artifactId>
<version>3.0.0</version>
<scope>provided</scope>
</dependency>
</dependencies>
</project>
Below is the spark configuration for the java application:
SparkSession spark = SparkSession
.builder()
.appName("JavaBucketedRandomProjectionLSHExample")
.master(args[0])
.getOrCreate();
Java Version Used-11
This works fine when I try running using spark-submit by setting the master-URL with one worker node.
Actual Issue:
When I try to run the program in two worker nodes(two different machines using same network) using spark-submit, then I faced the local class incompatible issue.
Below is the command I used to run with two worker nodes using spark-submit:
./spark-master.sh- To start the master
./spark-slave.sh "Master-URL" - To start the workers
Below is the spark WEB UI for the above config:
3.1.0-SNAPSHOT Spark Master at spark://mushfiq:7077
URL: spark://mushfiq:7077
Alive Workers: 2
Cores in use: 16 Total, 0 Used
Memory in use: 29.2 GiB Total, 0.0 B Used
Resources in use:
Applications: 0 Running, 34 Completed
Drivers: 0 Running, 0 Completed
Status: ALIVE
Workers (2)
Worker Id Address State Cores Memory
worker-20200617123647-128.123.63.43-45983 128.123.63.43:45983 ALIVE 8 (0 Used) 14.6 GiB (0.0 B Used)
worker-20200617124042-128.123.63.78-34355 128.123.63.78:34355 ALIVE 8 (0 Used) 14.6 GiB (0.0 B Used)
Below is the spark-submit command used to execute the java application jar using the above two worker nodes:
./spark-submit --class DistributedLSH.Application /home/mushfiq/Desktop/Eclipse/DLSHlatest.jar spark://mushfiq:7077 /home/mushfiq/Desktop/LSH/Datasets/Color/Color.ds /home/mushfiq/Desktop/LSH/Datasets/Color/randomdata.ds
Explanation of the above command:
./spark-submit --class <ClassName>
<Path to Jar File("/home.../DSHlatest.jar")>
<args[0]-Master URL(spark://mushfiq:7077)>
<args[1](/home/..../Color.ds)>
<args[2](/home/.....randomdata.ds)>
Apache spark version for spark-submit- We used the following GitHub version for spark-submit - Apache Spark version
Question
So, when we run the above command, we get the following exception:
Lost task 1.3 in stage 2.0 (TID 9, 128.123.63.78, executor 1): java.io.InvalidClassException: org.apache.spark.sql.catalyst.encoders.ExpressionEncoder;
local class incompatible: stream classdesc serialVersionUID = -2784885576793145316, local class serialVersionUID = -2690984018213736661
I tried a lot of approaches to fix the issue. But couldn't. I think the issue might be the mismatching spark versions of pom.xml spark and spark-submit. Any help here is highly appreciated. Thanks in advance. Also if you have any questions regarding the issue, please let me know.

This usually happens if you have a conflicting version of spark-catalyst_2.12 JAR that contains the class org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
So can you please check the classpath of spark-submit in client mode to see from which jar is it taking that class?
Make sure that you are not mixing the classpath to include both the JARs. You can check the "--jars" parameter values.

Spark2.2.1 incompatible Jackson version 2.8.8

My configuration is:
Scala 2.11 (plugin Scala IDE)
Eclipse Neon.3 Release (4.6.3)
Windows 7 64bit
I want run this simple scala code (Esempio.scala):
package it.scala
// importo packages di Spark
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
object Wordcount {
def main(args: Array[String]) {
val inputs: Array[String] = new Array[String](2)
inputs(0) = "C:\\Users\\FobiDell\\Desktop\\input"
inputs(1) = "C:\\Users\\FobiDell\\Desktop\\output"
// oggetto SparkConf per settare i parametri sulla propria applicazione
// da fornire poi al cluster manager scelto (Yarn, Mesos o Standalone).
val conf = new SparkConf()
conf.setAppName("Smartphone Addiction")
conf.setMaster("local")
// oggetto SparkContext per connessione al cluster manager scelto
val sc = new SparkContext(conf)
//Read file and create RDD
val rawData = sc.textFile(inputs(0))
//convert the lines into words using flatMap operation
val words = rawData.flatMap(line => line.split(" "))
//count the individual words using map and reduceByKey operation
val wordCount = words.map(word => (word, 1)).reduceByKey(_ + _)
//Save the result
wordCount.saveAsTextFile(inputs(1))
//stop the spark context
sc.stop
}
}
So, if I use the Spark-shell everything is ok otherwise, from Eclipse IDE, if I select the file (Esempio.scala) and run it via Run->Run as->Scala application, I obtain this Exception:
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.SparkContext.withScope(SparkContext.scala:701)
at org.apache.spark.SparkContext.textFile(SparkContext.scala:830)
at it.scala.Wordcount$.main(Esempio.scala:47)
at it.scala.Wordcount.main(Esempio.scala)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.8.8
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:745)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
... 4 more
My pom.xml file is:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>it.hgfhgf.xhgfghf</groupId>
<artifactId>progetto</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>progetto</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<!-- Neo4j JDBC DRIVER -->
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-jdbc-driver</artifactId>
<version>3.1.0</version>
</dependency>
<!-- Scala -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.11</version>
</dependency>
<!-- Spark -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.1</version>
</dependency>
</dependencies>
</project>
I noticed that the .jar files that are into spark-2.2.1-bin-hadoop2.7/jars directory are:
jackson-core-2.6.5.jar
jackson-databind-2.6.5.jar
jackson-module-paranamer-2.6.5.jar
jackson-module-scala_2.11-2.6.5.jar
jackson-annotations-2.6.5.jar
Can anyone explain to me in simple terms what this exception is and how can it be resolved?

Spark 2.x contains the jackson 2.6.5 and neo4j-jdbc-driver uses jackson 2.8.8 version, here the dependency conflict between two different version of jackson library.
That's why you are getting this Incompatible Jackson version: 2.8.8 error.
Try to override the dependency version for these[below] modules inside your pom.xml and see if works,
jackson-core
jackson-databind
jackson-module-scala_2.x
or try adding below dependency into your pom.xml
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.11</artifactId>
<version>2.8.8</version>
</dependency>

Not sure if this helps anyone whos had the problem with an sbt project thats using scala 2.12. Putting in jackson-module-scala_2.11 doesn't quite work. There a single version of jackson-module-scala 2.6.7 that has a scala 2.12 build
Following line in build.sbt worked
dependencyOverrides ++= {
Seq(
"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.6.7.1",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.6.7",
"com.fasterxml.jackson.core" % "jackson-core" % "2.6.7"
)
}
This fixed the problem for spark 2.4.5

Scala version 2.1.1 works with Jackson 2.6.5. Use the following:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.6.5</version>
</dependency>

I did run into the same version conflict of Jackson. In addition to override jackson-core, jackson-databind, jackson-module-scala_2.x, I also defined jackson-annotations in my pom.xml, which solved the conflict.

Explanation:
This exception occurs when there is a dependency conflict between two different versions of the Jackson library.
To resolve conflicts maven proceeds as follows:
It uses a a nearest-wins strategy.
If dependencies are on the same level, in that case maven would resolve the conflict by simply using the one, which has a higher position in pom.
This can lead to picking to wrong Jackson version.
Solution:
To detect the conflict, you can use in intelliJ the plugin Maven Helper. It will allow you to exclude conflictuel dependencies using the <exclusions> element in the element by which the problematic jar is included.
Note:
This error can also happen when trying to launch a spark Job on cluster mode. In this case, you have to specify explicitally the jar using the spark.driver.extraClassPath and spark.executor.extraClassPath configurations with the spark-submit command.
--conf spark.driver.extraClassPath
--conf spark.executor.extraClassPath

I was getting error Incompatible Jackson version: 2.9.9-3
I added
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.14.0" in build.sbt
.build project(Build => Rebuild project)
.Invalidate and restart(file=>Invalidate cache)
it's worked for me

Below is the combination that worked for me .
aws-java-sdk-1.7.4.jar
hadoop-aws-2.7.3.jar
joda-time-2.9.6.jar
hadoop-client-2.7.3-sources.jar
hadoop-client-2.7.3.jar
hadoop-client-2.6.0-javadoc.jar
hadoop-client-2.6.0.jar
jets3t-0.9.4.jar
jackson-core-2.10.0.jar
jackson-databind-2.8.6.jar
jackson-module-scala_2.11-2.8.5.jar
jackson-annotations-2.8.7.jar

Unable to connect to spark on remote system

I am trying to connect to spark master on a remote system through java app
I am using
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.1</version>
</dependency>
and code
{
SparkSession sparkSession = SparkSession.builder().
master("spark://ip:7077")
.appName("spark session example")
.getOrCreate();
JavaSparkContext sc = new JavaSparkContext(sparkSession.sparkContext());
}
Getting
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:713)
at org.apache.spark.sql.SparkSession$Builder.master(SparkSession.scala:766)
at com.mobelisk.spark.JavaSparkPi.main(JavaSparkPi.java:9)
Also If I change to
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
**<version>2.0.1</version>**
</dependency>
on the same program getting
Caused by: java.lang.RuntimeException: java.io.InvalidClassException: org.apache.spark.rpc.netty.RequestMessage; local class incompatible: stream classdesc serialVersionUID = -2221986757032131007, local class serialVersionUID = -5447855329526097695
In Spark-shell on remote
Spark context available as 'sc' (master = local[*], app id = local-1477561433881).
Spark session available as 'spark'.
Welcome to
____ __
/ / _ _____/ /
_\ / _ / _ `/ / '/
// .__/_,// //_\ version 2.0.1
//
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
Type in expressions to have them evaluated.
Type :help for more information.
As I am very new to all this, I am not able to figure out the issue in program

I figured it out, posting this in case if someone is going to follow the similar approach.
I had added
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>2.0.0-M3</version>
which comes with scala-library 2.10.6
but there already exists a scala-library 2.11.8 in spark-core
so I had to exclude the earlier one like this
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>2.0.0-M3</version>
<exclusions>
<exclusion>
<artifactId>scala-library</artifactId>
<groupId>org.scala-lang</groupId>
</exclusion>
<exclusion>
<artifactId>scala-reflect</artifactId>
<groupId>org.scala-lang</groupId>
</exclusion>
</exclusions>
</dependency>
Now everything is working fine

This Spark version mismatch:
you use 2.10 in project.
cluster uses 2.11
Update dependency to 2.11.

NoSuchMethodError : Spark + Kafka : Java

Use Case
Simple message fetching and printing from Kafka topic using Spark with Java as programming language
Background
Experience in dealing with Kafka Storm Integration, developed and maintained kafka cluster and storm topologies more than a year.
No experience with Apache Spark and Scala
Simple word count application built and tested successfully using stand alone spark cluster.
Problem
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:64)
at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:110)
at org.apache.spark.streaming.kafka.KafkaUtils.createStream(KafkaUtils.scala)
at com.random.spark.EventsToFileAggregator.main(EventsToFileAggregator.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
At EventsToFileAggregator.java:54
JavaPairReceiverInputDStream<String, String> messages =
KafkaUtils.createStream(jsc, args[0], args[1], topicMap,
StorageLevel.MEMORY_AND_DISK_SER());
pom.xml
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.6.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>1.6.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.11</artifactId>
<version>1.6.1</version>
</dependency>
</dependencies>
Build
Successful without any warnings
Command
./bin/spark-submit --class com.random.spark.EventsToFileAggregator --master spark://host:7077 /usr/local/spark/stats/target/stats-1.0-SNAPSHOT-jar-with-dependencies.jar localhost:2181 test topic 2

NoSuchMethodError is almost always an indication that two libraries are not at a compatible version. In this case Spark-Streaming Kafka is attempting to use a Scala language feature that doesn't exist. Check that the version of Spark-Streaming Kafka is compatible with the version of Scala you're using. Make sure you're not actually running with Scala and not Java.

java.lang.NoClassDefFoundError error when running my project

I have been struggling to get this to work and I think i can use some help. I am working on a Java project where the pom.xml has a bunch of dependencies some of which are themselves indirectly dependent on this jar:
com.sun.jersey:jersey-core:1.17.1 like this:
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
<version>1.0.1</version>
</dependency>
And I need this particular jar in my pom because I want to use the new features in jax-rs api:
javax.ws.rs:javax.ws.rs-api:2.0.The problem is when I build my project I get this error:
Found duplicate classes in [com.sun.jersey:jersey-core:1.17.1,javax.ws.rs:javax.ws.rs-api:2.0] :
[WARNING] javax.ws.rs.ApplicationPath
[WARNING] javax.ws.rs.Consumes
[WARNING] javax.ws.rs.CookieParam
[WARNING] javax.ws.rs.DELETE
[WARNING] javax.ws.rs.DefaultValue
[WARNING] javax.ws.rs.Encoded
[WARNING] javax.ws.rs.FormParam
[WARNING] javax.ws.rs.GET
[WARNING] javax.ws.rs.HEAD
[WARNING] javax.ws.rs.HeaderParam
...
..
I tried to fix this by excluding com.sun.jersey:jersey-core:1.17.1 from dependencies which were including it by checking the dependency tree.
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
<version>1.0.1</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
</exclusion>
</exclusions>
</dependency>
Now the project builds fine but when I try to run it I get this error:
java.lang.NoClassDefFoundError: com/sun/jersey/core/util/FeaturesAndProperties
at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.7.0_51]
at java.lang.Class.privateGetDeclaredMethods(Class.java:2531) ~[na:1.7.0_51]
at java.lang.Class.getDeclaredMethods(Class.java:1855) ~[na:1.7.0_51]
at com.google.inject.internal.ProviderMethodsModule.getProviderMethods
(ProviderMethodsModule.java:81) ~[guice-3.0.jar:na]
at com.google.inject.internal.ProviderMethodsModule.configure
(ProviderMethodsModule.java:73) ~[guice-3.0.jar:na]
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223)
~[guice-3.0.jar:na]
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:232)
~[guice-3.0.jar:na]
at com.google.inject.spi.Elements.getElements(Elements.java:101) ~[guice-3.0.jar:na]
at com.google.inject.spi.Elements.getElements(Elements.java:92) ~[guice-3.0.jar:na]
It seems like jersey core was needed but how do I get around this problem?

You can't mix JAX-RS / Jersey versions
Jersey version 1 is the reference implementation for JAX-RS 1. Jersey version 1 uses the com.sun.jersey group / package prefix.
Jersey version 2 is the reference implementation for JAX-RS 2. Jersey version 2 uses the org.glassfish.jersey group / package prefix
If you have both Jersey versions, or both JAX-RS versions on your classpath, you'll get lots of NoClassDefFoundError, NoSuchMethodError or similar.
If possible use JAX-RS / Jersey version 2

For me this actually means I had somehow mixed things and was using
$ mvn dependency:list | grep jersey
[INFO] com.sun.jersey:jersey-client:jar:1.11:compile
[INFO] com.sun.jersey:jersey-core:jar:1.0.2:compile
(conflict within jersey 1.x)

Jersey is the reference implementation of JAX-RS, so you don't need the latest version of JAX-RS API (javax.ws.rs:javax.ws.rs-api:2.0) but the latest version of Jersey which is 2.8 (see https://jersey.java.net/).