flink: scala version conflict? - java

Am attempting to compile the kafka sample from here in IntelliJ. After much fussing with dependencies have run into this issue that I can't seem to get past:
15/10/25 12:36:34 ERROR actor.ActorSystemImpl: Uncaught fatal error from thread [flink-akka.actor.default-dispatcher-4] shutting down ActorSystem [flink]
java.lang.NoClassDefFoundError: scala/runtime/AbstractPartialFunction$mcVL$sp
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.flink.runtime.jobmanager.MemoryArchivist.handleMessage(MemoryArchivist.scala:80)
at org.apache.flink.runtime.FlinkActor$class.receive(FlinkActor.scala:32)
at org.apache.flink.runtime.jobmanager.MemoryArchivist.org$apache$flink$runtime$LogMessages$$super$receive(MemoryArchivist.scala:59)
at org.apache.flink.runtime.LogMessages$class.receive(LogMessages.scala:26)
at org.apache.flink.runtime.jobmanager.MemoryArchivist.receive(MemoryArchivist.scala:59)
at akka.actor.ActorCell.newActor(ActorCell.scala:567)
at akka.actor.ActorCell.create(ActorCell.scala:587)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:460)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:482)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:282)
at akka.dispatch.Mailbox.run(Mailbox.scala:223)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: scala.runtime.AbstractPartialFunction$mcVL$sp
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 28 more
I've run across a few notions that suggest this is an issue with the scala version. Current library list:
flink-runtime-1.0-SNAPSHOT
flink-streaming-java-1.0-SNAPSHOT
flink-connector-kafka-1.0-SNAPSHOT
flink-java8-1.0-SNAPSHOT
flink-core-1.0-SNAPSHOT
flink-java-1.0-SNAPSHOT
org.apache.hadoop:hadoop-core:1.2.1
flink-clients-1.0-SNAPSHOT
org.apache.kafka:kafka-clients:0.8.2.2
org.apache.kafka:kafka_2.11:0.8.2.2
flink-optimizer-1.0-SNAPSHOT
org.apache.sling:org.apache.sling.commons.json:2.0.6
de.javakaffee:kryo-serializers:0.28
com.github.scopt:scopt_2.11:3.3.0
org.clapper:grizzled-slf4j_2.9.0:0.6.6
com.typesafe.akka:akka-osgi_2.11:2.4.0
com.typesafe.akka:akka-slf4j_2.11:2.4.0
Suggestions on where I've run astray?

The problem is indeed a Scala version mismatch. You're mixing dependencies which are built for Scala 2.11, e.g. org.apache.kafka:kafka_2.11:0.8.2.2 with Flink dependencies which are built by default for Scala 2.10.
One of the dependencies built for Scala 2.11 pulls in the scala-library:2.11 jar which replaces the scala-library:2.10 dependency required by the Flink dependencies. You either use the binaries built for Scala 2.10 for the non-Flink dependencies or you build and install Flink using Scala 2.11. See https://ci.apache.org/projects/flink/flink-docs-master/setup/building.html#build-flink-for-a-specific-scala-version for how to build Flink with different Scala version.
Kafka example
If you just want to bump the version of the referenced Kafka example to 0.10-SNAPSHOT you have to change the Flink version in the pom.xml file and you have to use the FlinkKafkaProducer instead of the KafkaSink in the WriteIntoKafka.java file. You don't need the SimpleStringSchema then anymore. That is all you have to change (no additional dependencies are required).

Related

Travis CI: java.lang.UnsupportedClassVersionError

I am trying to setup CI for https://github.com/amCap1712/pljava using Travis. But it fails with following log:
find pljava-packaging -name \*.jar | sudo xargs java -jar
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/gjt/cuspy/JarX has
been compiled by a more recent version of the Java Runtime (class file version 53.0), this
version of the Java Runtime only recognizes class file versions up to 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)
The complete build log and travis config is available at https://travis-ci.org/github/amCap1712/pljava/builds/661687014 . I am using OpenJDK9 for this build but it fails for OpenJDK14 as well.
How to fix the error ?

Class not found after copying libraries to libraries folder (NetBeans)

New to Java. As a DBA it's not the world I normally live in, but I IBM's CDC offering provides a set of libraries for any real automation.
https://www.ibm.com/support/knowledgecenter/en/SSTRGZ_11.3.0/com.ibm.cdcdoc.chcclp.doc/tasks/setupembeddedscript.html
I can import and run my application so far if I reference the .jar files from their absolute path. i.e. where the IBM CDC application was installed. But if I copy the .jar files to my libraries folder in NetBeans, I get the below error:
The import commands don't error and everything seems fine until I run it. Any advise would be appreciated.
Exception in thread "main" java.lang.NoClassDefFoundError: org/eclipse/osgi/util/NLS
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.ibm.replication.cdc.scripting.EmbeddedScript.executeWithStatus(EmbeddedScript.java:231)
at com.ibm.replication.cdc.scripting.EmbeddedScript.execute(EmbeddedScript.java:159)
at cdcinterface.CDCInterface.main(CDCInterface.java:18)
Caused by: java.lang.ClassNotFoundException: org.eclipse.osgi.util.NLS
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 15 more
C:\Users\roberdan\AppData\Local\NetBeans\Cache\8.2\executor-snippets\run.xml:53: Java returned: 1
BUILD FAILED (total time: 0 seconds)
This is line 53 of run.xml:
<java classpath="${classpath-translated}" classname="${classname}" dir="${work.dir}" jvm="${platform.java}" fork="true" failonerror="${java.failonerror}">
<jvmarg value="-Dfile.encoding=${encoding}"/>
<redirector inputencoding="${encoding}" outputencoding="${encoding}" errorencoding="${encoding}"/>
<jvmarg line="${run.jvmargs}" />
<arg line="${application.args}" />
<syspropertyset>
<propertyref prefix="run-sys-prop."/>
<mapper from="run-sys-prop.*" to="*" type="glob"/>
</syspropertyset>
</java>
Turns out the issues was with the IBM classes themselves. You can get the classes either from the server install, or the client install. I copied them from the client install but should have been from the server. Once I did that, all was good.

Java Exceptions error when trying to execute .jar on Debian Jessie

A teacher of mine created a program on java around 2005, he has always used MacOS. He sent the .jar to me and a classmate, both linux (me Debian 8 and him Ubuntu) and we can't execute the program because it returns a lot of exceptions.
At first I thought it could have been because the program was too old but my teacher can run it in his 1 year old updated mac perfectly fine.
Checked and Debian comes with OpenJDK as default. Havent installed anything, is there anything I should be installing for it to run?
PS: if I double click the .jar I can see the folders.
The exceptions are:
Exception in thread "main" java.lang.NoClassDefFoundError: org/jfree/chart/ChartPanel
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
at java.lang.Class.getMethod0(Class.java:2866)
at java.lang.Class.getMethod(Class.java:1676)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.jfree.chart.ChartPanel
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 18 more
I don't know anything about java so please if you can try to be as basic as you can. I am also new to Debian, so I really don't know if there's a package missing from the default installation.
Edit:
The MANIFEST.MF file is
Manifest-Version: 1.0
Ant-Version: Apache Ant 1.8.2
Created-By: 1.6.0_24-b07-334-10M3326 (Apple Inc.)
Class-Path: lib/Jama-1.0.2-ed.jar lib/jfreechart-1.0.6.jar lib/ant.jar
lib/jcommon-1.0.10.jar
X-COMMENT: Main-Class will be added automatically by build
Main-Class: com.cmcweb.electronicstructure.CompleteDOSPlotter
You have to download JFreeChart.jar libary (http://www.java2s.com/Code/Jar/j/Downloadjfreechartjar.htm) because your teacher didn't include external classes in his jar file.
Put JFreeChart.jar into /your_linux_path/jre/lib/ext folder and after that it should works.
P.S. The same you have to do with Jama.jar and jcommon.jar
Make sure you have all jars with the version from the manifest file downloaded in a lib directory. Lib directory should be placed in the same directory as your jar, the names of dependency jars should be exactly the same as these from classpath in manifest.
So in your case, it may not work with the newest JFreeChart version. You need to find and download version 1.0.6

Apache Ant: NoClassDefFoundError when using Junit

I just can't seem to get my junit tests run by Ant. Here's my setup:
build.xml
http://pastebin.com/c7n4cdJ6
What am I trying to do? I run ant main or ant package respectively. The compile target compiles my code. First the classes of my webapp and afterwards my test classes:
javac: source: webapp java files, destination: ${web.classes.dir}
javac: source: test java files, destination: ${web.testclasses.dir}
For dependency management I use Apache Ivy, which is working fine.
So let's take a look at my junit target. You can see that the classpath is defined in junit.classpath. It contains:
My Ivy dependencies with the test scope: <path refid="test.path"/>
The compiled class files of my tests: <pathelement location="${web.testclasses.dir}"/>
The compiled class files of my webapp: <pathelement location="${web.classes.dir}"/>
Also it contains a dirset of my compiled test classes (you see I was desperate).
So when I run the ant script it compiles well, the class files are all there. But when it comes to junit target and the first test is being executed, I get this error in the console:
[junit] Running CreateStopwordSqliteDBTest
[junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
BUILD FAILED
In the result xml file I get further information:
<error message="CreateStopwordSqliteDBTest (wrong name: de/mycompany/myproduct/CreateStopwordSqliteDBTest)" type="java.lang.NoClassDefFoundError">
java.lang.NoClassDefFoundError: CreateStopwordSqliteDBTest (wrong name: de/mycompany/myproduct/CreateStopwordSqliteDBTest)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
</error>
NoClassDefFoundError tells me that the class could be found at compile time but not on runtime. Please note that the affected class is the actual JUnit test. Why can't it be found even though I made sure it is part of my junit classpath?
Fyi: When I remove the lines 131 - 133 from my build.xml (the dirset in my junit classpath), I would get a ClassNotFoundException:
<error message="CreateStopwordSqliteDBTest" type="java.lang.ClassNotFoundException">
java.lang.ClassNotFoundException: CreateStopwordSqliteDBTest
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
</error>
Could you help me out here?
The source directories are wrong. Here are the correct properties:
<property name="src.dir" location="${basedir}/src" />
<property name="testsrc.dir" location="${basedir}/testsrc" />

Java Errors starting from ClassNotFoundException

I have the following errors when i try to run my Netbeans aplication of a map reduce algorithm:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Level
at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:349)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1121)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
at org.apache.hadoop.security.Groups.<init>(Groups.java:54)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:142)
at
org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:243)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation
.java:216)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation
.java:207)
at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroup
Information.java:286)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroup
Information.java:500)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroup
Information.java:483)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:73)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:133)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:138)
at cloudex.CloudEx.main(CloudEx.java:33)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 19 more
Could you please give me a hint on how to get rid of it? i do not know what it generates it
You need to add the log4j jar to your classpath.
Either add the log4j class to your classpath as #smcg has stated, or add the following into your project pom.xml if you're using maven.
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
Download Apache log4j *.jar file at log4j. Import external *.jar file to your project.

Categories

Resources