My configuration is:
Scala 2.11 (plugin Scala IDE)
Eclipse Neon.3 Release (4.6.3)
Windows 7 64bit
I want run this simple scala code (Esempio.scala):
package it.scala
// importo packages di Spark
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
object Wordcount {
def main(args: Array[String]) {
val inputs: Array[String] = new Array[String](2)
inputs(0) = "C:\\Users\\FobiDell\\Desktop\\input"
inputs(1) = "C:\\Users\\FobiDell\\Desktop\\output"
// oggetto SparkConf per settare i parametri sulla propria applicazione
// da fornire poi al cluster manager scelto (Yarn, Mesos o Standalone).
val conf = new SparkConf()
conf.setAppName("Smartphone Addiction")
conf.setMaster("local")
// oggetto SparkContext per connessione al cluster manager scelto
val sc = new SparkContext(conf)
//Read file and create RDD
val rawData = sc.textFile(inputs(0))
//convert the lines into words using flatMap operation
val words = rawData.flatMap(line => line.split(" "))
//count the individual words using map and reduceByKey operation
val wordCount = words.map(word => (word, 1)).reduceByKey(_ + _)
//Save the result
wordCount.saveAsTextFile(inputs(1))
//stop the spark context
sc.stop
}
}
So, if I use the Spark-shell everything is ok otherwise, from Eclipse IDE, if I select the file (Esempio.scala) and run it via Run->Run as->Scala application, I obtain this Exception:
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.SparkContext.withScope(SparkContext.scala:701)
at org.apache.spark.SparkContext.textFile(SparkContext.scala:830)
at it.scala.Wordcount$.main(Esempio.scala:47)
at it.scala.Wordcount.main(Esempio.scala)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.8.8
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:745)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
... 4 more
My pom.xml file is:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>it.hgfhgf.xhgfghf</groupId>
<artifactId>progetto</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>progetto</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<!-- Neo4j JDBC DRIVER -->
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-jdbc-driver</artifactId>
<version>3.1.0</version>
</dependency>
<!-- Scala -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.11</version>
</dependency>
<!-- Spark -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.1</version>
</dependency>
</dependencies>
</project>
I noticed that the .jar files that are into spark-2.2.1-bin-hadoop2.7/jars directory are:
jackson-core-2.6.5.jar
jackson-databind-2.6.5.jar
jackson-module-paranamer-2.6.5.jar
jackson-module-scala_2.11-2.6.5.jar
jackson-annotations-2.6.5.jar
Can anyone explain to me in simple terms what this exception is and how can it be resolved?
Spark 2.x contains the jackson 2.6.5 and neo4j-jdbc-driver uses jackson 2.8.8 version, here the dependency conflict between two different version of jackson library.
That's why you are getting this Incompatible Jackson version: 2.8.8 error.
Try to override the dependency version for these[below] modules inside your pom.xml and see if works,
jackson-core
jackson-databind
jackson-module-scala_2.x
or try adding below dependency into your pom.xml
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.11</artifactId>
<version>2.8.8</version>
</dependency>
Not sure if this helps anyone whos had the problem with an sbt project thats using scala 2.12. Putting in jackson-module-scala_2.11 doesn't quite work. There a single version of jackson-module-scala 2.6.7 that has a scala 2.12 build
Following line in build.sbt worked
dependencyOverrides ++= {
Seq(
"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.6.7.1",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.6.7",
"com.fasterxml.jackson.core" % "jackson-core" % "2.6.7"
)
}
This fixed the problem for spark 2.4.5
Scala version 2.1.1 works with Jackson 2.6.5. Use the following:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.6.5</version>
</dependency>
I did run into the same version conflict of Jackson. In addition to override jackson-core, jackson-databind, jackson-module-scala_2.x, I also defined jackson-annotations in my pom.xml, which solved the conflict.
Explanation:
This exception occurs when there is a dependency conflict between two different versions of the Jackson library.
To resolve conflicts maven proceeds as follows:
It uses a a nearest-wins strategy.
If dependencies are on the same level, in that case maven would resolve the conflict by simply using the one, which has a higher position in pom.
This can lead to picking to wrong Jackson version.
Solution:
To detect the conflict, you can use in intelliJ the plugin Maven Helper. It will allow you to exclude conflictuel dependencies using the <exclusions> element in the element by which the problematic jar is included.
Note:
This error can also happen when trying to launch a spark Job on cluster mode. In this case, you have to specify explicitally the jar using the spark.driver.extraClassPath and spark.executor.extraClassPath configurations with the spark-submit command.
--conf spark.driver.extraClassPath
--conf spark.executor.extraClassPath
I was getting error Incompatible Jackson version: 2.9.9-3
I added
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.14.0" in build.sbt
.build project(Build => Rebuild project)
.Invalidate and restart(file=>Invalidate cache)
it's worked for me
Below is the combination that worked for me .
aws-java-sdk-1.7.4.jar
hadoop-aws-2.7.3.jar
joda-time-2.9.6.jar
hadoop-client-2.7.3-sources.jar
hadoop-client-2.7.3.jar
hadoop-client-2.6.0-javadoc.jar
hadoop-client-2.6.0.jar
jets3t-0.9.4.jar
jackson-core-2.10.0.jar
jackson-databind-2.8.6.jar
jackson-module-scala_2.11-2.8.5.jar
jackson-annotations-2.8.7.jar
Related
I am having an issue specifically with the 'http' imports shown below
import static io.gatling.javaapi.http.HttpDsl.http;
import static io.gatling.javaapi.http.HttpDsl.status;
They are not recognised in the following block:
.exec(http("${testType}")
.post(RequestBuilder.launch1p0)
.formParam(LTIParam.context_id.name(), "${district_pid}")
)
My simplified version of my pom.xml looks like this:
<properties>
<java.version>17</java.version>
<gatling.version>3.8.4</gatling.version>
<gatling-maven-plugin.version>4.2.7</gatling-maven-plugin.version>
<scala.version>2.12.8</scala.version>
<scala-logging_2.11.version>3.7.2</scala-logging_2.11.version>
<scala-maven-plugin.version>4.7.2</scala-maven-plugin.version>
<performance-base.version>1.0</performance-base.version>
</properties>
<dependency>
<groupId>io.gatling.highcharts</groupId>
<artifactId>gatling-charts-highcharts</artifactId>
<version>${gatling.version}</version>
</dependency>
<dependency>
<groupId>io.gatling</groupId>
<artifactId>gatling-app</artifactId>
<version>${gatling.version}</version>
</dependency>
<dependency>
<groupId>io.gatling</groupId>
<artifactId>gatling-recorder</artifactId>
<version>${gatling.version}</version>
</dependency>
<plugin>
<groupId>io.gatling</groupId>
<artifactId>gatling-maven-plugin</artifactId>
<version>${gatling-maven-plugin.version}</version>
</plugin>
I have tried reloading the maven projects, deleting the gatling-core.jar files and retrying
mvn clean install -DskipTests
If anyone knows why it has an issue with this specific import, and not the following:
import io.gatling.javaapi.core.ChainBuilder;
import static io.gatling.javaapi.core.CoreDsl.*;
please explain to me.
Thank you.
What you provided is very suspicious.
You mention that you have in your pom.xml:
<scala.version>2.12.8</scala.version>
<scala-logging_2.11.version>3.7.2</scala-logging_2.11.version>
but those properties are not used in what you provided.
If you are really pulling those libraries, those versions are completely wrong and break your build.
Gatling 3.8 requires Scala 2.13 (since 3.5)
you're trying to force Scala 2.12.8 => not compatible
scala-logging_2.11 means "compiles for Scala 2.11"
3 different and incompatible Scala versions!
You should be pulling Scala 2.13.10 (but that shouldn't be necessary) and scala-logging_2.13 3.9.5.
I have an java project with javafx and jaxws. Here a small snippelt from my gradle:
dependencies {
implementation 'com.sun.xml.ws:jaxws-ri:4.0.0'
}
javafx {
version = "19"
modules = [ 'javafx.controls', 'javafx.fxml' ]
}
task runClient(type: JavaExec) {
description = 'Run Client'
classpath = sourceSets.main.runtimeClasspath
jvmArgs = ['--module-path', classpath.asPath, '--add-modules', 'javafx.controls,javafx.fxml']
main = 'ClientMain'
}
I set this jvmArgs because of How to include plugin dependencies in JavaExec task classpath?. I don't actually use modules otherwise in my project.
I now get the error:
Error occurred during initialization of boot layer
java.lang.module.FindException: Module format not recognized: C:\Users\MyName\.gradle\caches\modules-2\files-2.1\com.sun.xml.ws\release-documentation\4.0.0\5eb09d77be092684546352a35f315423d67e044b\release-documentation-4.0.0-docbook.zip
Without javafx or jaxws everything is working. Only together there seems to be problems. Any ideas?
You are using the wrong dependency for jaxws for your purposes. The difference in distributions is explained here:
What's the difference between jaxws-ri and jaxws-rt?
jaxws-ri contains zipped documentation which should not be part of a modular distribution.
These are dependencies you should be using (maven format, you will need to translate to gradle format):
<dependencies>
<dependency>
<groupId>jakarta.xml.ws</groupId>
<artifactId>jakarta.xml.ws-api</artifactId>
<version>4.0.0</version>
</dependency>
</dependencies>
<dependencies>
<dependency>
<groupId>com.sun.xml.ws</groupId>
<artifactId>jaxws-rt</artifactId>
<version>4.0.0</version>
<scope>runtime</scope>
</dependency>
</dependencies>
Additional background info
jaxws reference implementations are now maintained via the Eclipse foundation as jakarta XML services as part of Eclipse Metro.
The code should also use the jakarta package namespace as shown in the jakarta documentation.
This would also apply to module names if you provide a module-info.
When attempting to test a neo4j server using neo4j-harness, I get a noclassdeffound exception when building the embedded Neo4j server
private final Neo4j embeddedNeo4jServer = Neo4jBuilders.newInProcessBuilder()
.withDisabledServer()
.withFixture("")
.build(); // exception here
My pom.xml looks like this:
<dependencies>
<!-- Core Dependencies -->
<dependency><groupId>org.neo4j.driver</groupId><artifactId>neo4j-java-driver</artifactId><version>4.4.5</version></dependency>
<dependency><groupId>org.apache.kafka</groupId><artifactId>kafka-streams</artifactId><version>3.0.0</version></dependency>
<dependency><groupId>org.apache.curator</groupId><artifactId>curator-x-discovery</artifactId><version>4.2.0</version></dependency>
<dependency><groupId>org.json</groupId><artifactId>json</artifactId><version>20211205</version></dependency>
<dependency><groupId>org.slf4j</groupId><artifactId>slf4j-simple</artifactId><version>1.7.30</version></dependency>
<!-- Test Dependencies -->
<dependency><groupId>org.junit.jupiter</groupId><artifactId>junit-jupiter</artifactId><version>5.8.2</version><scope>test</scope></dependency>
<dependency><groupId>org.skyscreamer</groupId><artifactId>jsonassert</artifactId><version>1.5.0</version><scope>test</scope></dependency>
<dependency><groupId>org.neo4j.test</groupId><artifactId>neo4j-harness</artifactId><version>4.4.5</version><scope>test</scope></dependency>
<!-- Spring dependencies for an embedded Kafka instance -->
<dependency><groupId>org.springframework.kafka</groupId><artifactId>spring-kafka</artifactId><version>2.8.3</version><scope>test</scope></dependency>
<dependency><groupId>org.springframework.kafka</groupId><artifactId>spring-kafka-test</artifactId><scope>test</scope><version>2.8.3</version></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope><version>2.6.4</version>
<exclusions><exclusion><groupId>org.junit.vintage</groupId><artifactId>junit-vintage-engine</artifactId></exclusion></exclusions></dependency>
<!-- Mockito dependencies - NOTE: the inline dependency replaces the core one, but might be removed in future versions-->
<!--<dependency><groupId>org.mockito</groupId><artifactId>mockito-core</artifactId><version>4.4.0</version><scope>test</scope></dependency>-->
<dependency><groupId>org.mockito</groupId><artifactId>mockito-junit-jupiter</artifactId><version>4.4.0</version><scope>test</scope></dependency>
<dependency><groupId>org.mockito</groupId><artifactId>mockito-inline</artifactId><version>4.4.0</version><scope>test</scope></dependency>
</dependencies>
There is a clash in the version of scala-library used in spring-kafka-test and neo4j-harness.
To resolve this, you can exclude the scala dependency from spring-kafka-test, meaning the neo4j-harness version will be used for both.
To do this, add an exclusion in the pom.xml:
<dependency><groupId>org.springframework.kafka</groupId><artifactId>spring-kafka-test</artifactId><scope>test</scope><version>2.8.3</version>
<exclusions><exclusion><groupId>org.scala-lang</groupId><artifactId>scala-library</artifactId></exclusion></exclusions></dependency>
I'm using a program that relies on the following two imports:
import org.lwjgl.opencl.CLDevice;
import org.lwjgl.opencl.CLPlatform;
Eclipse is reporting that the "import cannot be resolved" even though I've added LWJGL OpenCL as a dependency to my project.
Here's a snapshot of my POM file:
<dependency>
<groupId>org.lwjgl</groupId>
<artifactId>lwjgl-opencl</artifactId>
<version>3.1.6</version>
</dependency>
I've read somewhere that these classes only exist in an earlier version so I tried changing the version to all of the versions found here ( from 3.1.0 to 3.1.6) but none of them resolved the issue.
Is there an earlier/different version that is not on the Maven repository page? If not where could I find the said class?
Thanks
It seems that you are using the abandoned lwjgl v2 library. It can be found in another Maven repository:
<dependency>
<groupId>org.lwjgl.lwjgl</groupId>
<artifactId>lwjgl</artifactId>
<version>2.9.3</version>
</dependency>
I have installed Xerces through Maven:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jdom</groupId>
<artifactId>jdom</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.11.0</version>
</dependency>
</dependencies>
I then tried the code given in this example from the Xerces FAQ to validate a XML file against a schema in version 1.1. This is my code:
private static void validateFile(File xmlFile, File xsdFile) throws SAXException, IOException
{
// 1. Lookup a factory for the W3C XML Schema language
SchemaFactory factory = SchemaFactory.newInstance("http://www.w3.org/XML/XMLSchema/v1.1");
// 2. Compile the schema.
File schemaLocation = xsdFile;
Schema schema = factory.newSchema(schemaLocation);
// 3. Get a validator from the schema.
Validator validator = schema.newValidator();
// 4. Parse the document you want to check.
Source source = new StreamSource(xmlFile);
// 5. Check the document
try
{
validator.validate(source);
System.out.println(xmlFile.getName() + " is valid.");
}
catch (SAXException ex)
{
System.out.println(xmlFile.getName() + " is not valid because ");
System.out.println(ex.getMessage());
}
}
The code only yields this exception:
java.lang.IllegalArgumentException: No SchemaFactory that implements the schema language specified by: http://www.w3.org/XML/XMLSchema/v1.1 could be loaded
at javax.xml.validation.SchemaFactory.newInstance(SchemaFactory.java:204)
at example.xml.XSDValidator.validateFile(XSDValidator.java:65)
Seems like I failed to configure/install Xerces correctly. Please help me get this working, the XML files force me to use the schema in 1.1, I got a normal validator for 1.0 running but I have huge problems with this. I appreciate every hint!
It looks that you need Xerces2 Java 2.11.0 (XML Schema 1.1) (Beta) version, which isn't in maven repository. You can download it from Xerces website, and install it to your local maven repository:
mvn install:install-file -Dfile=xercesImpl.jar -DgroupId=xerces -DartifactId=xercesImpl -Dversion=2.11.0.beta -Dpackaging=jar
Then you will be able to include it in your Maven project dependencies:
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.11.0.beta</version>
</dependency>
I will add another answer, because for me this dependency did not work (same error as described by OP):
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.11.0</version>
</dependency>
I quess 2.11.0 should be newer than 2.11.0.beta, but it seems like xsd1.1 is not supported in that version !
Instead only the following dependency lead to a working XSD1.1 validation for me:
<dependency>
<groupId>org.opengis.cite.xerces</groupId>
<artifactId>xercesImpl-xsd11</artifactId>
<version>2.12-beta-r1667115</version>
</dependency>
( Found in this SO thread: How to validate XML against XSD 1.1 in Java? )
I think they have added version 2.11 to maven now. The following dependency in Maven works out-of-the-box:
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.11.0</version>
</dependency>
Xerces-J provides a "fully compliant XML Schema 1.1 implementation" since version 2.12.0, see release history here: https://xerces.apache.org/news.html.
Xerces2 Java 2.12.2 (XML Schema 1.1) - tar.gz [PGP] [SHA]
Just got released earlier this week.
https://xerces.apache.org/mirrors.cgi#notes