I am following the documentation found at this link
https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Usage
When i try to compile for WordCount.java and create a jar, I get the following error
bin/hadoop com.sun.tools.javac.Main WordCount.java
Error: Could not find or load main class com.sun.tools.javac.Main
I have verified my $JAVA_HOME and $HADOOP_CLASSPATH in the hadoop-env.sh file and also verified to see if I have the jdk
Here are the contents from hadoop-env.sh
export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/"
.......
.........
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH="$JAVA_HOME/lib/tools.jar"
else
export HADOOP_CLASSPATH=$f
fi
I am not sure the reason behind error or if I am missing another key configuration?
This doesn't make sense in that loop... Nor does checking the existence of the variable first
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH="$JAVA_HOME/lib/tools.jar"
else
You need to set HADOOP_CLASSPATH="$JAVA_HOME/lib/tools.jar", as the documentation says for that class to be found. And that class is only available in the JDK
But, you could just run javac command to compile code. Not sure why the docs have you calling that class.
How to compile a Hadoop program
$ javac -classpath ${HADOOP_CLASSPATH} -d WordCount/ WordCount.java
To create jar:
$ jar -cvf WordCount.jar -C WordCount/ .
To run:
$ hadoop jar WordCount.jar WordCount input/ output
Suggestion Please use Maven/Gradle to create proper JAR files, and an IDE to write code.
P.S. Not many people actually write plain MapReduce
Related
I am trying to run the WordCount example. But I am facing issues with compiling the program.
I get the error:
error: package org.apache.hadoop.mapred does not exist
after executing:
javac -classpath /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.3.jar -d wordcount_classes WordCount.java
I set up hadoop using this tutorial. I also looked this up on stackoverflow : question and executed the bin/hadoop classpath command in /usr/local/hadoop. This is the output I obtained:
/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/* :/usr/local/hadoop/share/hadoop/common/* :/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/* :/usr/local/hadoop/share/hadoop/hdfs/* :/usr/local/hadoop/share/hadoop/yarn/lib/* :/usr/local/hadoop/share/hadoop/yarn/* :/usr/local/hadoop/share/hadoop/mapreduce/lib/* :/usr/local/hadoop/share/hadoop/mapreduce/* :/contrib/capacity-scheduler/*.jar
But I don't know what to make of it or what my next step should be! Please help!
You're trying to compile the source code using one of the many hadoop dependency jars (hadoop-common-x.x.x.jar). The jar that contains the mapred package noted in the error message is the hadoop-mapreduce-client-core jar.
I suggest you use a build tool such as Maven or Gradle to build your source code as it will manage transitive dependencies for you.
Alternatively to proceed with your manual invocation of javac, try something like this (untested):
javac -cp '/usr/local/hadoop/share/hadoop/common/*' \
-cp '/usr/local/hadoop/share/hadoop/hdfs/lib/*' \
-cp '/usr/local/hadoop/share/hadoop/hdfs/*' \
-cp '/usr/local/hadoop/share/hadoop/yarn/lib/*' \
-cp '/usr/local/hadoop/share/hadoop/yarn/*' \
-cp '/usr/local/hadoop/share/hadoop/mapreduce/lib/*' \
-cp '/usr/local/hadoop/share/hadoop/mapreduce/*' \
-d wordcount_classes WordCount.java
I'm running my hbase program using
java -classpath run.jar com.mycompany.app.HBaseImporter test2
/home/rahulko/Downloads/my-app/xaa
I have specified the HADOOP_CLASSPATH in hadoop.env.sh like this
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
else
export HADOOP_CLASSPATH=$f
fi
done
export HADOOP_CLASSPATH="$HBASE_HOME/lib/hbase-client-0.98.18-hadoop2.jar:\
$HBASE_HOME/lib/hbase-common-0.98.18-hadoop2.jar:\
$HBASE_HOME/lib/protobuf-java-2.5.0.jar:\
$HBASE_HOME/lib/guava-12.0.1.jar:\
$HBASE_HOME/lib/zookeeper-3.4.6.jar:\
$HBASE_HOME/lib/hbase-protocol-0.98.18-hadoop2.jar"
I have also specified in bashrc
export CLASSPATH=$CLASSPATH:/usr/local/hbase1/lib/*:/usr/local/hadoop/share/hadoop/common/*
But I'm still getting java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/MasterNotRunningException
But when I run from eclipse the program runs successfully
Solved it using
java -cp "run.jar:/usr/local/hbase1/lib/*"
com.mycompany.app.HBaseImporter test2
/home/rahulko/Downloads/my-app/xaa
mkdir like libs and put you dependents jar into this
and run this command:
java -Djava.ext.dirs=/xxx/xxx/libs -jar /xxx/xxx/you program jar
that can auto use the dependents jar
I am trying to learn MapReduce from the official documentation. To make a jar file for WordCount class, the documentation says to run the following command:
javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar -d wordcount_classes WordCount.java
But, I found that my Hadoop directory has no core.jar present. I suppose my Hadoop installation is alright as I can execute the Hadoop shell script from the Bin folder.
If you trying with that:
javac -classpath `hadoop classpath` -d wordcount_classes WordCount.java
Isn't the best practice, I think, but work for me.
Check in your hadoop-1.2.1 folder (as in my case), which you unzipped in "Prepare to Start Cluster" of single node setup. There you would find hadoop-1.2.1-core.jar
That is the file being used to compile here.
I am new to Android.I use this link to run the NDK Project.
I follow all these steps from the given link.However during compilation in the command prompt it shows error like:
$ javah com.hello.LibC
error: cannot access com.hello.LibC
class file for com.hello.LibC not found
javadoc: error - Class com.hello.LibC not found.
Error: No classes were specified on the command line. Try -help.
please help me.
Thanks in Advance.
you need to specify the classpath using -classpath PATH option when running javah inside bin directory of your project:
$ javah -classpath classes/ com.hello.LibC
also you can specify the output directory using option -d PATH
$ javah -d ../jni -classpath classes/ com.hello.LibC
I'm trying to compile a simple WordCount.java map-reduce example on a linux (CentOS) installation of Cloudera 4. I keep hitting compiler errors when I reference any of the hadoop classes, but I can't figure out which jars of the hundreds under /usr/lib/hadoop I need to add to my classpath to get things to compile. Any help would be greatly appreciated! What I'd like most is a java file for word count (just in case the one I found is bad for some reason) along with the associated command to compile and run it.
I am trying to do this using just javac rather than Eclipse. My main issue either way is what exactly are the Hadoop libraries from the Cloudera 4 install which I need to include in order to get the classic WordCount example to compile. Basically, I need to put the Java MapReduce API classes (Mapper, Reducer, etc.) in my classpath.
I have a script that builds my hadoop classes. Try:
#!/bin/bash
program=`echo $1 | awk -F "." '{print $1}'`
if [ ! -d "${program}_classes" ]
then mkdir ${program}_classes/;
fi
javac -classpath /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1.jar:/usr/lib/hadoop/client/h\
adoop-mapreduce-client-core-2.0.0-cdh4.0.1.jar -d ${program}_classes/ $1
jar -cvf ${program}.jar -C ${program}_classes/ .;
You were probably missing the key jars:
/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1.jar
and
/usr/lib/hadoop/client/hadoop-mapreduce-client-core-2.0.0-cdh4.0.1.jar
If you are running the Cloudera CDH4 Virtual Machine then the following should get you running:
javac -classpath /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/client/hadoop-mapreduce-client-core-2.0.0-cdh4.0.0.jar -d wordcount_classes WordCount.java
Or you can export environment:
export JAVA_HOME=/usr/java/default
export PATH=${JAVA_HOME}/bin:${PATH}
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
and use the commands below:
$ bin/hadoop com.sun.tools.javac.Main WordCount.java
$ jar cf wc.jar WordCount*.class
If you are using Eclipse please do add Hadoop packages. you may get it from java2s or any similar sites. I couldn't say without know anything about what you did till now.