Exception in thread "main" java.lang.NoClassDefFoundError: com/google/protobuf/ServiceException
at org.apache.hadoop.ipc.ProtobufRpcEngine.<clinit> (ProtobufRpcEngine.java:73)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
I got the above error when I try to run the program using Naive Bayes Classifier using Map reduce.. I have imported all the necessary jar files... I already surfed for many answers and come to know that its because of protobuf version mismatch but I still have the correct version of jar....
Hadoop version : 2.6.0
Compiled with protoc : 2.5.0
Pls can any one help me out...... I am madly in need to execute it ....
Related
I have built a Java 1.8 application under Maven in Eclipse that allows me to generate BIRT reports in a programmatic way without the need for the UI. Within Eclipse everything works fine. If I compile the code into a JAR file and try to run it however I run into problems with java.lang.NoClassDefFoundErrors. Clearly I have a missing dependancy but I don't know how to resolve the problem.
The error is
java -cp target/* com.resonancex.scheduler.Scheduler
Establishing conection...
Connected
Got products to report
ProductId=1 ProductType=BREN Isin=XS1939207996 Template=BREN_MU
* birtEngine EngineConfig: org.eclipse.birt.report.engine.api.EngineConfig.emitterConfigs={html=org.eclipse.birt.report.engine.api.HTMLEmitterConfig#71bbf57e};
* Birt Engine Startring ....
Exception in thread "main" java.lang.NoClassDefFoundError: org/eclipse/core/runtime/CoreException
at org.eclipse.birt.core.framework.Platform.createPlatformLauncher(Platform.java:115)
at org.eclipse.birt.core.framework.Platform.startup(Platform.java:74)
at com.resonancex.scheduler.BirtEngine.start(BirtEngine.java:33)
at com.resonancex.scheduler.BirtEngine.getBirtEngine(BirtEngine.java:47)
at com.resonancex.scheduler.BIRTReportGenerator.setReportCustom(BIRTReportGenerator.java:67)
at com.resonancex.scheduler.Scheduler.generateAllReports(Scheduler.java:68)
at com.resonancex.scheduler.Scheduler.main(Scheduler.java:79)
Caused by: java.lang.ClassNotFoundException: org.eclipse.core.runtime.CoreException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I added the library org.eclipse.birt.runtime_4.8.0-20180626-4.8.0 which appears to be the required BIRT runtime. Can anyone give me a few pointers as to what else I may need?
Just in case anyone else tries to do this I thought I would list the libraries I needed to install to get standalone BIRT report generation running running. The list is as follows. Obviously this list may change over time so keep that in mind:
axis-1.4.jar
batik-anim-1.7.jar
batik-awt-util-1.7.jar
batik-bridge-1.7.jar
batik-css-1.7.jar
batik-dom-1.7.jar
batik-gvt-1.7.jar
batik-i18n-1.9.jar
batik-parser-1.7.jar
batik-script-1.7.jar
batik-svg-dom-1.7.jar
batik-transcoder-1.7.jar
batik-util-1.8.jar
batik-xml-1.7.jar
hibernate-agroal-5.4.9.Final.jar
icu4j-4.8.jar
itext-2.1.7.jar
mssql-jdbc-7.4.1.jre8.jar
org.apache.batik.css-1.6.0.jar
org.eclipse.birt.runtime_4.8.0-20180626-4.8.0.jar
org.eclipse.core.runtime_3.14.0.v20180417-0825-4.8.0.jar
org.eclipse.datatools.connectivity.oda-3.5.0.201603142002.jar
org.eclipse.datatools.connectivity.oda.consumer-3.3.0.201603142002.jar
org.eclipse.emf.common-2.16.0.jar
org.eclipse.emf.ecore-2.11.1.jar
org.eclipse.emf.ecore.xmi-2.11.1.jar
org.eclipse.equinox.common_3.10.0.v20180412-1130-4.8.0.jar
org.eclipse.equinox.registry_3.8.0.v20180426-1327-4.8.0.jar
org.eclipse.osgi_3.13.0.v20180409-1500-4.8.0.jar
org.mozilla.javascript-1.7.5.v201504281450.jar
org.w3c.dom.smil_1.0.1.v200903091627-4.8.0.jar
org.w3c.dom.svg-1.1.0.jar
sac-1.3.jar
xml-apis-1.4.01.jar
I have extracted a jar from a Maven project that runs the MapReduce job. However, I keep receiving the error "java.lang.ClassNotFoundException". The things that I have tried to use to repair this is:
Configured the classpath
Tried doing job.setJar(.jar)
Attempted job.setJarbyClass(.class)
Changing JobConf path file
Caress Hadoop and tell it everything is going to be okay
I extracted the jar file from Maven and transferred it to a Linux server, and running it from there.
The full error message is:
Exception in thread "main" java.lang.ClassNotFoundException: BLAMapAttempt2
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
If there is any missing information needed to answer this question, please let me know, and thank you for reading.
When exporting the jar out of Eclipse, I unchecked "classpath" because I was not running it locally. This fixed my error.
We are trying to execute basic Hbase example on hortonworks sandbox (2.3).
hadoop jar /usr/hdp/2.3.0.0-2557/hbase/lib/hbase-examples.jar org.apache.hadoop.hbase.mapreduce.IndexBuilder
We are getting below exception after executing this program.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
at org.apache.hadoop.hbase.mapreduce.IndexBuilder.<clinit>(IndexBuilder.java:67)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 5 more
Based on this error we tried to set the Hadoop classpath in Hbase-env.sh.
/usr/hdp/2.3.0.0-2557/hbase/lib/hbase-client-1.1.1.2.3.0.0-2557.jar:/usr/hdp/2.3.0.0-2557/hbase/lib/hbase-common-1.1.1.2.3.0.0-2557.jar:/usr/hdp/2.3.0.0-2557/hbase/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.3.0.0-2557/hbase/lib/guava-12.0.1.jar:$/usr/hdp/2.3.0.0-2557/hbase/lib/zookeeper.jar:/usr/hdp/2.3.0.0-2557/hbase/lib/hbase-protocol-1.1.1.2.3.0.0-2557.jar:/usr/hdp/2.3.0.0-2557/hbase/lib/commons-configuration-1.6.jar:/usr/hdp/2.3.0.0-2557/hbase/lib/hadoop-common.jar:/usr/hdp/2.3.0.0-2557/hbase/lib/hbase-0.94.27.jar
But still getting the same error.
Instead of manually adding jars into classpath you can directly use below command.
$(hbase classpath) recursively search in hortonworks hadoop folders and finds the required jars from sandbox.
HADOOP_CLASSPATH=$(hbase classpath):/usr/hdp/2.3.0.0-2557/hbase/conf hadoop jar /usr/hdp/2.3.0.0-2557/hbase/lib/hbase-examples.jar org.apache.hadoop.hbase.mapreduce.IndexBuilder
When I face NoClassDefFoundError error with mapreduce, I add jar using one of the jar class in JobBuilder to resolve it.
e.g.
Job job = new Job(conf);
job.setJarByClass(org.apache.hadoop.hbase.util.Bytes.class);
Supply jars using libjars parameter to your job-
e.g.
LIB=hbase-x.x.x.jar
hadoop jar /usr/hdp/2.3.0.0-2557/hbase/lib/hbase-examples.jar org.apache.hadoop.hbase.mapreduce.IndexBuilder -libjars ${LIB}
you can also add jar to HADOOP_CLASSPATH variable before launch job.
Is all the latest code included in the jar? Use a java decompiler such as jd-gui to look inside the jar file to make sure this class you are referencing is actually there. Also check that the necessary import statements are present in the Java class.
I'm working on a Java project running with Hadoop 0.20.1 and I'm trying to migrate to Hadoop 2.6.0. Once I've changed the corresponding Hadoop jar files in the project, I obtain the following error when submitting a job:
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:557)
at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)
at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:187)
at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:174)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:108)
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:285)
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:131)
at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:536)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
I've read it could be a problem related to Hadoop binaries, but I've built them myself, placed them in "c:\hadoop\bin" and the environment variable HADOOP_HOME has the right value.
I'm running my project on Eclipse, on a machine with Windows 7 64 bits and Java 8.
Can anyone help me with this?
Thanks!
I've finally solved my problem. I had Java 8 32 bits installed instead of the 64 bits version. I installed the right version and Hadoop jobs submitted perfectly.
TL/DR : are mahout 0.9 examples compatible with hadoop 2.4 ?
My problem:
I would like to classify a bunch of documents using Mahout 0.9. To do so, I'm following the example described here.
I'm on windows and trying to go full native (ie, no cygwin). I already dispose of a local hadoop 2.4.1 cluster.
I downloaded the mahout sources and compiled it according to the wiki :
mvn "-Dhadoop2.version=2.4.1" -DskipTests clean install
I then tried to execute the example with the following example :
hadoop jar $Env:mahout_home/examples/target/mahout-examples-0.9-job.jar org.apache.mahout.driver.MahoutDriver seqdirectory -i Decomposition -o output
It all seems to work : I'm getting logs showing the mapreduce job begins to run. However, I quickly get the following errors :
Error: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.ja
va:166)
at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.<init>(CombineFileRecordReader.java:126)
at org.apache.mahout.text.MultipleTextFileInputFormat.createRecordReader(MultipleTextFileInputFormat.java:43)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:492)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:735)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.initNextRecordReader(CombineFileRecordReader.ja
va:157)
... 10 more
Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but c
lass was expected
at org.apache.mahout.text.WholeFileRecordReader.<init>(WholeFileRecordReader.java:59)
... 15 more
According to the various links I found, it seems to come from code intented for Hadoop 1.0.
Am I missing something, or are mahout provided examples not suited for a Hadoop 2.4 cluster ?
The problem is because the object TaskAttemptContext is a Interface in the version 2.4 of Hadoop and the job was expecting a class (version 1.1.2 Hadoop). TaskAttemptContext has been changed in version 2.0.