I'm trying to submit a streaming job using Beam 2.27/Flink 1.12, using the following Maven command-line:
mvn exec:java -Dexec.mainClass=org.example.MyPipelineClass -Pflink-runner -Dexec.args="--runner=FlinkRunner --flinkMaster=flink-host:8081 --filesToStage=target/pipelines-bundled-1.0.0.jar"
Things were working smoothly for a bit, but I recently made some edits and wanted to run a new version of the pipeline and am now getting the following error:
Caused by: java.io.InvalidClassException: org.apache.flink.streaming.api.graph.StreamConfig$NetworkInputConfig; local class incompatible: stream classdesc serialVersionUID = -3137689219135046939, local class serialVersionUID = 3698633776553163849
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2160)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2093)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1655)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
at org.apache.flink.streaming.api.graph.StreamConfig.getInputs(StreamConfig.java:263)
... 21 more
My pom.xml looks something like this:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>pipelines</artifactId>
<version>1.0.0</version>
<packaging>jar</packaging>
<properties>
<beam.version>2.27.0</beam.version>
<flink.version>1.12</flink.version>
<maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
<maven-exec-plugin.version>1.6.0</maven-exec-plugin.version>
<maven-jar-plugin.version>3.2.0</maven-jar-plugin.version>
<maven-shade-plugin.version>3.1.0</maven-shade-plugin.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>${maven-jar-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${maven-shade-plugin.version}</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${project.artifactId}-bundled-${project.version}</finalName>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/LICENSE</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${maven-exec-plugin.version}</version>
<configuration>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
<profiles>
<profile>
<id>flink-runner</id>
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-flink-${flink.version}</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
</dependencies>
</profile>
</profiles>
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>${beam.version}</version>
</dependency>
<!-- note that the pipeline runs on GCP -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>${beam.version}</version>
<exclusions>
<exclusion>
<groupId>com.google.cloud.bigtable</groupId>
<artifactId>bigtable-client-core</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- main output is JDBC/Postgres -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-jdbc</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.4</version>
</dependency>
</dependencies>
</project>
Of course, before submitting the job I built the JAR using mvn clean package -Pflink-runner.
Flink is self-hosted on a GKE instance, and is deployed using the flink:1.12.4-java8 image.
I found this issue which seems to have the same error message, but no clear offender.
Any help or ideas to explore would be greatly appreciated, I have no idea what to investigate.
I found the issue own my own. In the pom.xml above, I specified that the Beam runner was beam-runners-flink-1.12, and the cluster was running Flink 1.12.4
Beam's Flink runner does not specify a patch, and uses Flink 1.12.0
Downgrading the cluster fixed my issue.
Related
I updated my project to Java 17/Wildfly 27/Keycloak 19 and I get the error:
java.lang.NoClassDefFoundError: jakarta/ws/rs/core/MultivaluedMap
The library Maven: jakarta.ws.rs-api is added in the library of the project via Maven: jakarta.ws.rs:jakarta.ws.rs-api:3.1.0
Does somebody know why this happens?
My pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>mainProject</artifactId>
<groupId>ch.company.mainProject</groupId>
<version>4.5.1</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>auth-post</artifactId>
<name>Project Auth Post (Keycloak Setup)</name>
<description>This project is responsible to setup Keycloak.</description>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.10.1</version>
<configuration>
<source>17</source>
<target>17</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>ch.company.project.AuthPostMain</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>ch.company.project.AuthPostMain</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>ch.company.project</groupId>
<artifactId>common</artifactId>
</dependency>
<dependency>
<groupId>org.keycloak</groupId>
<artifactId>keycloak-admin-client-jakarta</artifactId>
<version>19.0.0</version>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-client</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxrs</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jaxb-provider</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.jboss.resteasy</groupId>
<artifactId>resteasy-jackson2-provider</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.jetbrains</groupId>
<artifactId>annotations</artifactId>
</dependency>
<dependency>
<groupId>jakarta.ws.rs</groupId>
<artifactId>jakarta.ws.rs-api</artifactId>
</dependency>
</dependencies>
</project>
Stacktrace:
"C:\Program Files\ojdkbuild\jdk-17.0.2\bin\java.exe" -
Dtenant.config=../e2e-test/src/main/resources/e2e-config "-
javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2022.1.3\lib\idea_rt.jar=55918:C:\Program Files\JetBrains\IntelliJ IDEA 2022.1.3\bin" -Dfile.encoding=UTF-8 -classpath C:\work\project\auth-post\target\classes;C:\work\project\common\target\classes;C:\Users\maku\.m2\repository\org\keycloak\keycloak-admin-client-jakarta\19.0.0\keycloak-admin-client-jakarta-19.0.0.jar;C:\Users\maku\.m2\repository\org\keycloak\keycloak-core\19.0.0\keycloak-core-19.0.0.jar;C:\Users\maku\.m2\repository\org\keycloak\keycloak-common\19.0.0\keycloak-common-19.0.0.jar;C:\Users\maku\.m2\repository\org\jboss\resteasy\resteasy-client\6.2.2.Final\resteasy-client-6.2.2.Final.jar;C:\Users\maku\.m2\repository\org\jboss\resteasy\resteasy-client-api\6.2.2.Final\resteasy-client-api-6.2.2.Final.jar;C:\Users\maku\.m2\repository\org\jboss\resteasy\resteasy-core-spi\6.2.2.Final\resteasy-core-spi-6.2.2.Final.jar;C:\Users\maku\.m2\repository\jakarta\annotation\jakarta.annotation-api\2.1.1\jakarta.annotation-api-2.1.1.jar;C:\Users\maku\.m2\repository\org\jboss\resteasy\resteasy-core\6.2.2.Final\resteasy-core-6.2.2.Final.jar;C:\Users\maku\.m2\repository\org\jboss\jandex\2.4.3.Final\jandex-2.4.3.Final.jar;C:\Users\maku\.m2\repository\jakarta\activation\jakarta.activation-api\2.1.0\jakarta.activation-api-2.1.0.jar;C:\Users\maku\.m2\repository\org\eclipse\angus\angus-activation\1.0.0\angus-activation-1.0.0.jar;C:\Users\maku\.m2\repository\com\ibm\async\asyncutil\0.1.0\asyncutil-0.1.0.jar;C:\Users\maku\.m2\repository\org\jboss\logging\jboss-logging\3.5.0.Final\jboss-logging-3.5.0.Final.jar;C:\Users\maku\.m2\repository\org\apache\httpcomponents\httpclient\4.5.14\httpclient-4.5.14.jar;C:\Users\maku\.m2\repository\org\apache\httpcomponents\httpcore\4.4.16\httpcore-4.4.16.jar;C:\Users\maku\.m2\repository\commons-logging\commons-logging\1.2\commons-logging-1.2.jar;C:\Users\maku\.m2\repository\commons-codec\commons-codec\1.15\commons-codec-1.15.jar;C:\Users\maku\.m2\repository\org\reactivestreams\reactive-streams\1.0.4\reactive-streams-1.0.4.jar;C:\Users\maku\.m2\repository\org\jboss\resteasy\resteasy-jaxrs\3.15.3.Final\resteasy-jaxrs-3.15.3.Final.jar;C:\Users\maku\.m2\repository\org\jboss\spec\javax\ws\rs\jboss-jaxrs-api_2.1_spec\2.0.1.Final\jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar;C:\Users\maku\.m2\repository\org\jboss\spec\javax\xml\bind\jboss-jaxb-api_2.3_spec\2.0.1.Final\jboss-jaxb-api_2.3_spec-2.0.1.Final.jar;C:\Users\maku\.m2\repository\jakarta\validation\jakarta.validation-api\2.0.2\jakarta.validation-api-2.0.2.jar;C:\Users\maku\.m2\repository\org\jboss\spec\javax\annotation\jboss-annotations-api_1.3_spec\2.0.1.Final\jboss-annotations-api_1.3_spec-2.0.1.Final.jar;C:\Users\maku\.m2\repository\com\sun\activation\jakarta.activation\1.2.2\jakarta.activation-1.2.2.jar;C:\Users\maku\.m2\repository\commons-io\commons-io\2.6\commons-io-2.6.jar;C:\Users\maku\.m2\repository\com\github\stephenc\jcip\jcip-annotations\1.0-1\jcip-annotations-1.0-1.jar;C:\Users\maku\.m2\repository\org\jboss\resteasy\resteasy-jaxb-provider\6.2.2.Final\resteasy-jaxb-provider-6.2.2.Final.jar;C:\Users\maku\.m2\repository\jakarta\xml\bind\jakarta.xml.bind-api\4.0.0\jakarta.xml.bind-api-4.0.0.jar;C:\Users\maku\.m2\repository\org\glassfish\jaxb\codemodel\4.0.1\codemodel-4.0.1.jar;C:\Users\maku\.m2\repository\org\glassfish\jaxb\jaxb-core\4.0.1\jaxb-core-4.0.1.jar;C:\Users\maku\.m2\repository\org\glassfish\jaxb\jaxb-jxc\4.0.1\jaxb-jxc-4.0.1.jar;C:\Users\maku\.m2\repository\org\glassfish\jaxb\jaxb-runtime\4.0.1\jaxb-runtime-4.0.1.jar;C:\Users\maku\.m2\repository\org\glassfish\jaxb\txw2\4.0.1\txw2-4.0.1.jar;C:\Users\maku\.m2\repository\org\glassfish\jaxb\jaxb-xjc\4.0.1\jaxb-xjc-4.0.1.jar;C:\Users\maku\.m2\repository\org\glassfish\jaxb\xsom\4.0.1\xsom-4.0.1.jar;C:\Users\maku\.m2\repository\com\sun\istack\istack-commons-runtime\4.1.1\istack-commons-runtime-4.1.1.jar;C:\Users\maku\.m2\repository\com\sun\istack\istack-commons-tools\4.1.1\istack-commons-tools-4.1.1.jar;C:\Users\maku\.m2\repository\com\sun\xml\bind\external\relaxng-datatype\4.0.1\relaxng-datatype-4.0.1.jar;C:\Users\maku\.m2\repository\com\sun\xml\bind\external\rngom\4.0.1\rngom-4.0.1.jar;C:\Users\maku\.m2\repository\org\jboss\resteasy\resteasy-jackson2-provider\6.2.2.Final\resteasy-jackson2-provider-6.2.2.Final.jar;C:\Users\maku\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.14.1\jackson-core-2.14.1.jar;C:\Users\maku\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.13.4.2\jackson-databind-2.13.4.2.jar;C:\Users\maku\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.13.4\jackson-annotations-2.13.4.jar;C:\Users\maku\.m2\repository\com\fasterxml\jackson\jakarta\rs\jackson-jakarta-rs-base\2.13.4\jackson-jakarta-rs-base-2.13.4.jar;C:\Users\maku\.m2\repository\com\fasterxml\jackson\jakarta\rs\jackson-jakarta-rs-json-provider\2.13.4\jackson-jakarta-rs-json-provider-2.13.4.jar;C:\Users\maku\.m2\repository\com\fasterxml\jackson\module\jackson-module-jakarta-xmlbind-annotations\2.13.4\jackson-module-jakarta-xmlbind-annotations-2.13.4.jar;C:\Users\maku\.m2\repository\com\github\java-json-tools\json-patch\1.13\json-patch-1.13.jar;C:\Users\maku\.m2\repository\com\github\java-json-tools\msg-simple\1.2\msg-simple-1.2.jar;C:\Users\maku\.m2\repository\com\github\java-json-tools\btf\1.3\btf-1.3.jar;C:\Users\maku\.m2\repository\com\github\java-json-tools\jackson-coreutils\2.0\jackson-coreutils-2.0.jar ch.company.project.AuthPostMain
Fehler: Hauptklasse ch.company.project.AuthPostMain kann nicht initialisiert werden
Ursache: java.lang.NoClassDefFoundError: jakarta/ws/rs/core/MultivaluedMap
WildFly (and other JEE / Jakarta EE containers) provide their own Java EE / Jakarta EE classes (servlet API, JAX-RS, JPA, etc.). If you bundle them in your application, these classes will be not be used by WildFly. Even if the class files are identical, the classes in your application are not the same as the provided classes - they have different class loaders. Class equality does not just use the fully qualified class name, it also uses the class loader. As a result, you will most likely get a class mismatch. You didn't show the full stack trace, but this is a likely cause of it.
Mark these dependencies as provided, that should usually solve the issue. You will also have to do something in your shade plugin if they still get included.
The same goes for the RESTEasy dependencies, although that's specific to WildFly / JBoss.
Deleted .idea and reimported everything and now it works
I'm trying to debug my application that's running on a remote JVM, but I can't manage to get my dependencies there too.
Here are a few things I tried:
Remote debugger:
I've set up a remote debugger, and it corrects correctly to the JVM but when I then use my application I get a bunch of NoClassDefFoundError errors coming from external libraries that are in my maven dependencies (which makes sense because I building the app with the Intellij building tools and not Maven itself. I have some relocation rules set up so maybe that's the issue?
The JVM itself is running on my local machine.
Maven debugger
There is a maven debugger integrated inside Intellij but it seems like that is just using the remote JVM during the jar packaging but nothing else. Maybe I'm going something wrong there?
I also have a few yaml files in my resources folder that should be correctly included in the debugger.
Here is my project structure:
Here is my POM file for reference:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>me.playbosswar.com</groupId>
<artifactId>commandtimer</artifactId>
<name>CommandTimer</name>
<version>6.0</version>
<description>Plugin</description>
<build>
<finalName>${project.name}_v${project.version}</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.3.1</version>
<configuration>
<outputDirectory>/Users/tristan/Documents/minecraft/servers/1.16.5/plugins</outputDirectory>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<configuration>
<relocations>
<relocation>
<pattern>com.cryptomorin.xseries</pattern>
<shadedPattern>me.playbosswar.com</shadedPattern>
</relocation>
</relocations>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</resource>
</resources>
</build>
<repositories>
<repository>
<id>papermc</id>
<url>https://papermc.io/repo/repository/maven-public/</url>
</repository>
<repository>
<id>placeholderapi</id>
<url>http://repo.extendedclip.com/content/repositories/placeholderapi/</url>
</repository>
<repository>
<id>codemc-snapshots</id>
<url>https://repo.codemc.io/repository/maven-snapshots/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>com.destroystokyo.paper</groupId>
<artifactId>paper-api</artifactId>
<version>1.16.5-R0.1-SNAPSHOT</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>me.clip</groupId>
<artifactId>placeholderapi</artifactId>
<version>2.10.9</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.6</version>
</dependency>
<dependency>
<groupId>com.github.cryptomorin</groupId>
<artifactId>XSeries</artifactId>
<version>7.2.1.2</version>
</dependency>
<dependency>
<groupId>fr.minuskube.inv</groupId>
<artifactId>smart-invs</artifactId>
<version>1.2.7</version>
</dependency>
<dependency>
<groupId>net.wesjd</groupId>
<artifactId>anvilgui</artifactId>
<version>1.5.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.8.1</version>
</dependency>
</dependencies>
</project>
Here is the actual error:
I am developing a generic Spark application that listens to a Kafka stream using Spark and Java.
I am using kafka_2.11-0.10.2.2, spark-2.3.2-bin-hadoop2.7 - I also tried several other kafka/spark combinations before posting this question.
The code fails at loading StringDeserializer class:
SparkConf sparkConf = new SparkConf().setAppName("JavaDirectKafkaWordCount");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(2));
Set<String> topicsSet = new HashSet<>();
topicsSet.add(topics);
Map<String, Object> kafkaParams = new HashMap<>();
kafkaParams.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, brokers);
kafkaParams.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
kafkaParams.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
kafkaParams.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
The error I get is:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/StringDeserializer
From Why does Spark application fail with "Exception in thread "main" java.lang.NoClassDefFoundError: ...StringDeserializer"? it seems that this could be a scala version mismatch issue, but my pom.xml doesn't have that issue:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>yyy.iot.ckc</groupId>
<artifactId>sparkpoc</artifactId>
<version>1.0-SNAPSHOT</version>
<name>sparkpoc</name>
<!-- FIXME change it to the project's website -->
<url>http://www.example.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<java.version>1.8</java.version>
<spark.scala.version>2.11</spark.scala.version>
<spark.version>2.3.2</spark.version>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
<build>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version>
</plugin>
<!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
<!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
</plugin>
<plugin>
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
The submission script I use is:
./bin/spark-submit \
--class "yyy.iot.ckc.KafkaDataModeler" \
--master local[2] \
../sparkpoc/target/sparkpoc-1.0-SNAPSHOT.jar
Can anyone please point me in the right direction as to where I am going wrong?
Spark runs the program as by running an instance of a JVM. So if the libraries (JARs) are not in the classpath of that JVM we run into this runtime exception. The solution is to package all the dependent JARs along with main JAR. The following build script will work for that.
Also, as mentioned in https://stackoverflow.com/a/54583941/1224075 the scope of the spark-core and spark-streaming libraries need to be declared as provided. This is because some of the libraries are implicitly provided by the Spark JVM.
The build section of the POM which worked for me -
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2.1</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
You need to use the Maven Shade Plugin to package the Kafka clients along with your Spark application, then you can submit the shaded Jar, and the Kafka serializers should be found on the classpath.
Also, make sure you set the provided Spark packages
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${spark.scala.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
I've annotated a bad test with #Ignore, both on the method-level and on the class level. When running the test through the command line (I've tried "mvn clean install", "mvn test", "mvn clean install -DskipTests; mvn test"), however, the #Ignore annotation is ignored, the test is run, and- as it is a bad test- it fails.
Here is the test:
public class UserTest extends PersistentTestBase {
#Test
#Ignore
public void testPersistence() {
...
}
}
And here is my pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
...
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
...
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
...
<!-- Testing -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jmock</groupId>
<artifactId>jmock-junit4</artifactId>
<version>2.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jmock</groupId>
<artifactId>jmock-legacy</artifactId>
<version>2.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymock</artifactId>
<version>3.4</version>
<scope>test</scope>
</dependency>
...
<!-- For dep management, see Mykong.com's "How to create a jar file with Maven" -->
<build>
<plugins>
<!-- Necessary to force language level of Java 8 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<junitArtifactName>junit:junit-dep</junitArtifactName>
<useFile>false</useFile>
<trimStackTrace>false</trimStackTrace>
<reuseForks>false</reuseForks>
<forkCount>1</forkCount>
</configuration>
</plugin>
<!-- Packages into a jar, looking for deps in target/dependency-jars -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.2</version>
<configuration>
<excludes>
<exclude>**/log4j.properties</exclude>
</excludes>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>...Application</mainClass>
<classpathPrefix>dependency-jars</classpathPrefix>
</manifest>
</archive>
</configuration>
</plugin>
<!-- Moves all compiled deps to target/dependency-jars -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.5.1</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<includeScope>runtime</includeScope>
<outputDirectory>${project.build.directory}/dependency-jars</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<configuration>
<mainClass>...Application</mainClass>
</configuration>
</plugin>
</plugins>
</build>
Worth noting that the test is skipping when the class is run in Intellij IDEA 14, and that there are some other issues with tests failing on mvn test that are passing in Intellij (but are out of the scope of this question). Thank you for the help!
For those who are working with Junit 5.
#Ignore did not work formaven surefire neither Intellij tests even if I had already included junit-vintage-engine
I used #Disabled and it worked for maven at least!
Wemu posted the solution:
I think you can simply remove:
junit:junit-dep from the
surefire config – wemu Nov 20 at 7:10
Removed the element and all tests passed!
I had a same problem with the JUNIT library. After so many tries, it worked with the Pom.xml below:
<repositories>
<repository>
<id>junit</id>
<url>http://junit.org</url>
</repository>
</repositories>
<dependencies>
<dependency><groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version></dependency>
</dependencies>
For me the solution was to simply update the JUnit version to 4.13:
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>
I'm new into Groovy and I'm trying to write a very simple project using Groovy as language. I was able to write a hello world script, deploy an mvn artifact and execute it.
I'm going beyond that right now and I need to import some external libraries for testing matters but I don't know how to do that.
I would really appreciate if you could give me some tips about this.
Thanks!
You can specify dependencies for the project which have test scope, in which case they will not be used in compile scope, or you can specify dependencies which are only used in certain scenarios (if, for example, you want to write Groovy tests without requiring the rest of your team to do so).
This sample pom.xml has a custom profile element which describes dependencies and executions to run only if the current environment user is "andylopresto". This means that if my team doesn't want to use Groovy in their build, I could still write my test classes in Groovy and have them run as part of the standard mvn test phase on my machine only. In my team members' local machines (or a shared Jenkins instance), the Groovy test files will be ignored (not compiled or run, and the groovy-all dependency will not even be downloaded onto the machine).
Example:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>your.package</groupId>
<artifactId>artifact</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<developers>
<developer>
<id>alopresto</id>
<name>Andy LoPresto</name>
<email>alopresto#example.com</email>
<roles>
<role>developer</role>
<role>owner</role>
</roles>
</developer>
<developer>
<id>jjava</id>
<name>Johnny Java</name>
<email>ihategroovy#example.com</email>
<roles>
<role>sticking to older languages</role>
</roles>
</developer>
</developers>
<dependencyManagement>
<dependencies>
<!-- This dependency is available to all project code -->
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.9</version>
</dependency>
<!-- This dependency is only available during test phase -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<!-- This dependency is expected to be provided outside of Maven -->
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-util</artifactId>
<version>8.0.8</version>
<scope>provided</scope>
</dependency>
</dependencies>
</dependencyManagement>
<profiles>
<profile>
<!-- Custom profile for Groovy tests used only on Andy's computer -->
<id>andyLocalTest</id>
<activation>
<property>
<name>env.USER</name>
<value>andylopresto</value>
</property>
</activation>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.2.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.8</version>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/main/groovy</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-test-source</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/test/groovy</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<compilerId>groovy-eclipse-compiler</compilerId>
<source>1.6</source>
<target>1.6</target>
<includes>
<include>**/*.java</include>
<include>**/*.groovy</include>
</includes>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-compiler</artifactId>
<version>2.8.0-01</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-batch</artifactId>
<version>2.1.5-03</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</profile>
</profiles>