Error running Jar file in Hadoop's MapReduce program - java

I'm trying to compile my code into a JAR file for use with Hadoop's MapReduce. My main class is VectorMaker.java and the director structure is as follows.
RandomForestVectors
/bin
/lib
/hadoop-core-1.2.0.jar
/mahout-core-0.7.jar
/mahout-math-0.7.jar
/opencsv-2.3.jar
/VectorMaker.java
These are the commands I'm using to make my JAR file.
javac -classpath "./lib/*" -d ./bin ./VectorMaker.java
jar cf VectorMaker.jar -C "./bin/" . &
This is the command I used to try and run my JAR file as Hadoop MapReduce program.
hadoop jar VectorMaker.jar VectorMaker user/starmine/AlphaDefault/mahout/random_forest/prevectors /user/starmine/AlphaDefault/mahout/random_forest/test1
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/math/VectorWritable
at VectorMaker.main(VectorMaker.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.math.VectorWritable
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
I know I need to somehow make my JAR file contain all my dependencies but I'm not sure how.

Make sure you have all your dependencies in your classpath, including all the Mahout libraries. Alternatively you can specify the path through -classpath while compiling you program.
javac -classpath -classpath $MAHOUT_HOME/lib/*

Related

what is the alternative for Java Jar command to run a jar with scala class [duplicate]

I got some problems with the java. Check it out.
sebastian#sebastian-desktop:~/scaaaaaaaaala$ java -cp /home/sebastian/.m2/repository/org/scala-lang/scala-library/2.8.0.RC3/scala-library-2.8.0.RC3.jar:target/scaaaaaaaaala-1.0.jar scaaalaaa.App
Hello World!
That's cool, right, but how bout this:
sebastian#sebastian-desktop:~/scaaaaaaaaala$ java -cp /home/sebastian/.m2/repository/org/scala-lang/scala-library/2.8.0.RC3/scala-library-2.8.0.RC3.jar -jar target/scaaaaaaaaala-1.0.jar
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Application
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at scaaalaaa.App.main(App.scala)
Caused by: java.lang.ClassNotFoundException: scala.Application
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 13 more
Wat the heck? Any idea why the first works and not the 2nd? How do I -jar my scala??
My thanks in advance, brother.
If you define -jar the -classpath is ignored:
Java manual:
-jar When you use this option, the JAR file is the source of all user
classes, and other user class path
settings are ignored.
You can define the classpath dependencies in the Manifest metadata.
The easiest way to start your app is using the scala scripts:
scala -classpath target/scaaaaaaaaala-1.0.jar scaaalaaa.App Hello World!
Simply running
scala scaaaaaaaaala-1.0.jar
works for me with 2.11.6
use sbt-assembly sbt plugin to produce a jar containing all dependencies
sbt-assembly github
e.g. add a line in project/plugins.sbt:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10")
adjust your build.sbt e.g. with
mainClass in (assembly) := Some("mypackage.mysubpackage.MyMainClass")
assemblyJarName in assembly := "my_assembly.jar"
then build your assembled jar with
sbt assembly
and launch it with
java -jar ./target/scala-2.12/my_assembly.jar
Et voilĂ , no class-path this & thats needed any more. only 1 jar.
For an executable jar, the classpath should be in the jar's manifest. For help on doing that, look through the answers to Stackoverflow: How to create an executable jar with dependancy jars.
Below is the way to execute the Scala Executable Jar
We need scala installed in your system.
Here first will give the jar path and jar name
If your code needs any dependency jar then keep all jar in one directory and give a path of this in command like below after the ":".
At last give your class/object name.
scala -cp "/your/jar/path/jar_name.jar:/your/dependency/jar/path/*"
SampleCode

Java file runs but not Jar file

I'm trying to create a jar file for one of my files (Worker.java). The file is in the beatDB package in the src/ folder.
To create the jar I ran jar cfm InitializeInstanceWorker.jar Manifest.txt -C src/ . and my Manifest.txt contains: Main-Class: Main-Class: beatDB.Worker \n with a new line at the end.
When I call java beatDB/Worker the file runs, but then when I try to run the jar file, java.lang.ClassDefNotFoundException:
Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentialsProvider
Caused by: java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
I have aws-java-sdk-1.7.7.jar in my lib folder, which is on the classpath, which I set in my .bash_profile file.
Does anyone know why running the java file directly works, but the jar file does not run?
The ways that you can run your jar/main class are:
java -cp InitializeInstanceWorker.jar;lib\aws-java-sdk-1.7.7.jar beatDB.Worker
Or
java -cp lib\aws-java-sdk-1.7.7.jar beatDB.Worker -jar InitializeInstanceWorker.jar`
Or, if you add the attribute Class-Path: lib/ in your jar, you will be able to run it by:
java -jar InitializeInstanceWorker.jar
Hope it helps!

How to set Lucene with following error on Mac?

I am having trouble on setting up classpath on Mac. The following is the code I have attempted,
First I exported the jars using terminal
abc-MacBook-Pro:~ abc$ export
CLASSPATH=$CLASSPATH:/Users/abc/Desktop/IR/luceneJar/demo/lucene-demo-4.0.0.jar: export
CLASSPATH=$CLASSPATH:/Users/abc/Desktop/IR/luceneJar/core/lucene-core-4.0.0.jar: export
CLASSPATH=$CLASSPATH:/Users/abc/Desktop/IR/luceneJar/queryparser/lucene-queryparser-4.0.0.jar: export
CLASSPATH=$CLASSPATH:/Users/abc/Desktop/IR/luceneJar/analysis/common/lucene-analyzers-common-4.0.0.jar
Then when I echoed the classpath, following is the result
abc-MacBook-Pro:~ abc$ echo $CLASSPATH
/Users/abc/Desktop/IR/lucene-4.0.0 2/core/lucene-core 4.0.0.jar:
/Users/abc/Desktop/IR/lucene-4.0.0 2/demo/lucene-demo-4.0.0.jar:
/Users/abc/Desktop/IR/luceneJar/analysis/common/lucene-analyzers-common-4.0.0.jar
The error I am having is following when I tried to function the index file
abc-MacBook-Pro:~ abc$ java org.apache.lucene.demo.IndexFiles -docs
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/lucene/demo/IndexFiles
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.demo.IndexFiles
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Please let me know about the above error, thank you in advance. I am quite new with Lucene Java.
This is common Java classpath problem, nothing related to lucene (or mac) here.
Try this
1) Create lib folder and move all your lucene jars to lib
- lets say your project has two folders now lib and src, lib has all jars and src has yourJavaFile.java
2) Run below commands for compilation and execution from your src
- javac -cp ".:../lib/*" yourJavaFile.java
- java -cp ".:../lib/*" yourJavaFile

How do I add a .jar file to the classpath on Mac OS X?

I'm starting to become pretty irritated, because EVERY time I try to add a JAR file to the classpath to run a program (which compiles fine in Eclipse) I get a "java.lang.NoClassDefFoundError" exception.
The file swt.jar and GUI.class (main class) are in the same directory. If I issue this command: "java -cp swt.jar GUI", I get this error message:
Exception in thread "main" java.lang.NoClassDefFoundError: GUI Caused
by: java.lang.ClassNotFoundException: GUI at
java.net.URLClassLoader$1.run(URLClassLoader.java:202) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:190) at
java.lang.ClassLoader.loadClass(ClassLoader.java:306) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at
java.lang.ClassLoader.loadClass(ClassLoader.java:247)
If I formulate the command like this: "java GUI -cp swt.jar", it'll throw this exception:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/eclipse/swt/widgets/Display at GUI.main(GUI.java:9) Caused by:
java.lang.ClassNotFoundException: org.eclipse.swt.widgets.Display at
java.net.URLClassLoader$1.run(URLClassLoader.java:202) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:190) at
java.lang.ClassLoader.loadClass(ClassLoader.java:306) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at
java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 1 more
I just want to know how I can add .jar files to the runtime classpath on mac.
All of these commands are being issued from the Terminal on a 64-bit MacBook running Mac OS X Lion.
The problem, I think, is that you are specifying swt.jar as the classpath, but the file you're using is not in the jar. You neeed to specify both the jar and this dir as the classpath.
e.g.
java -cp swt.jar:./ GUI
This one will work, with modifications:
If I formulate the command like this: "java GUI -cp swt.jar"
You just didn't go far enough. You have to add ALL the 3rd party JARs, including the ones for Eclipse SWT and all your other dependencies.

problem related to MANIFEST.MF in jar

I have created jar in folder /usr/local/bin/niidle.jar.
And I have cerated MANIFEST.MF in jar as follows:
Manifest-Version: 1.0
Main-Class: com.ensarm.niidle.web.scraper.NiidleScrapeManager
Class-Path: hector-0.6.0-17.jar
And my 'hector-0.6.0-17.jar' file is present in following folder:
/Projects/EnwelibDatedOct13/Niidle/lib/hector-0.6.0-17.jar
And I have added this 'hector-0.6.0-17.jar' in niidle.jar folder as follows:
niidle.jar/lib/hector-0.6.0-17.jar
And when I run this using command:
java -jar /usr/local/bin/niidle.jar arguments....
then I am getting error message as follows:
Exception in thread "main" java.lang.NoClassDefFoundError: me/prettyprint/hector/api/Serializer
at com.ensarm.niidle.web.scraper.NiidleScrapeManager.main(NiidleScrapeManager.java:21)
Caused by: java.lang.ClassNotFoundException: me.prettyprint.hector.api.Serializer
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 1 more
And When I run again using different command as follows:
java -cp /Projects/EnwelibDatedOct13/Niidle/lib/hector-0.6.0-17.jar -jar /usr/local/bin/niidle.jar arguments.....
---------------Then also I am getting the same error message:--
Exception in thread "main" java.lang.NoClassDefFoundError: me/prettyprint/hector/api/Serializer
at com.ensarm.niidle.web.scraper.NiidleScrapeManager.main(NiidleScrapeManager.java:21)
Caused by: java.lang.ClassNotFoundException: me.prettyprint.hector.api.Serializer
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 1 more-------------
so tell me what is solution for that.or any modification in MANIFEST.MF in jar
java -cp /Projects/EnwelibDatedOct13/Niidle/lib/hector-0.6.0-17.jar -jar /usr/local/bin/niidle.jar arguments.....
You can't use the -cp switch together with the -jar switch.
From http://mindprod.com/jgloss/classpath.html:
If you use the -jar option, java.exe ignores the classpath. It will only look in that jar!! Try using the manifest Class-Path instead to get Java to look in auxiliary jars
And I have added this 'hector-0.6.0-17.jar' in niidle.jar folder as follows:
To my knowledge, you can't nest .jar files this way.
Since you have Class-Path: hector-0.6.0-17.jar you need to have the needle.jar and the hector-0.6.0-17.jar side by side and run niidle.jar from their common directory (using the -jar switch if you like).
Line niidle.jar/lib/hector-0.6.0-17.jar is confusing. Is niidle.jar directory or file?
I'd suggest you to post your full manifest, directory structure, command line you are using and current working directory.
But generally you should write in your manifest:
Class-Path: hector-0.6.0-17.jar
then put all jars in one directory. Let's call it foo.
Now open command prompt, go to foo and run
java -jar niidle.jar args...
And I believe that everything will work. Then if you wish play with location of your jars.
For setting up "jars within jars" take a look at this question:
Classpath including JAR within a JAR

Categories

Resources