Develop Reference

Hadoop WordCount error

I am following the documentation found at this link
https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Usage
When i try to compile for WordCount.java and create a jar, I get the following error
bin/hadoop com.sun.tools.javac.Main WordCount.java
Error: Could not find or load main class com.sun.tools.javac.Main
I have verified my $JAVA_HOME and $HADOOP_CLASSPATH in the hadoop-env.sh file and also verified to see if I have the jdk
Here are the contents from hadoop-env.sh
export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/"
.......
.........
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH="$JAVA_HOME/lib/tools.jar"
else
export HADOOP_CLASSPATH=$f
fi
I am not sure the reason behind error or if I am missing another key configuration?

This doesn't make sense in that loop... Nor does checking the existence of the variable first
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH="$JAVA_HOME/lib/tools.jar"
else
You need to set HADOOP_CLASSPATH="$JAVA_HOME/lib/tools.jar", as the documentation says for that class to be found. And that class is only available in the JDK
But, you could just run javac command to compile code. Not sure why the docs have you calling that class.
How to compile a Hadoop program
$ javac -classpath ${HADOOP_CLASSPATH} -d WordCount/ WordCount.java
To create jar:
$ jar -cvf WordCount.jar -C WordCount/ .
To run:
$ hadoop jar WordCount.jar WordCount input/ output
Suggestion Please use Maven/Gradle to create proper JAR files, and an IDE to write code.
P.S. Not many people actually write plain MapReduce

Error: Could not find or load main class Swagger Codegen

I left an open issue for here
I´m trying to create a custom codegen, I managed to make it work by putting the files inside of the codegen project but I want it to work like this: https://github.com/swagger-api/swagger-codegen#making-your-own-codegen-modules
I haven't modified the autogenerated project at all but I keep getting:
Error: Could not find or load main class io.swagger.codegen.SwaggerCodegen
This is the command line:
java -cp output/myLibrary/target/myCustomCodegen-swagger-codegen-1.0.0.jar:swagger-codegen-cli-2.1.6.jar io.swagger.codegen.SwaggerCodegen generate -i https://watson-api-explorer.mybluemix.net/listings/conversation-v1-experimental.json -l com.my.company.codegen.Mycustomcodegengenerator -o outputlocation
I got the jar from here https://mvnrepository.com/artifact/io.swagger/swagger-codegen-project/2.1.6
This is what I'm doing:
Run java -jar swagger-codegen-cli-2.1.6.jar meta \ -o output/myLibrary -n myClientCodegen -p com.my.company.codegen to create costume codegen
Run mvn package in output/myLibrary
Run java -cp output/myLibrary/target/myCustomCodegen-swagger-codegen-1.0.0.jar:swagger-codegen-cli-2.1.6.jar io.swagger.codegen.SwaggerCodegen generate -i https://watson-api-explorer.mybluemix.net/listings/conversation-v1-experimental.json -l com.my.company.codegen.Mycustomcodegengenerator -o outputlocation in the folder that contains both swagger-codege-cli-2.1.6.jar and the output folder
It does find the class if I remove the first part, but doesn´t find the new language:
java -cp swagger-codegen-cli-2.1.6.jar io.swagger.codegen.SwaggerCodegen generate -i https://watson-api-explorer.mybluemix.net/listings/conversation-v1-experimental.json -l com.my.company.codegen.Mycustomcodegengenerator -o outputlocation
I have looked at the answers for "Error: Could not find or load main class" problems but haven't manage to fix it.
Here is a link to the jar

For Windows, change the colon (:) to a semicolon (;) - between the jars in the class path. So instead of
java -cp output/myLibrary/target/myCustomCodegen-swagger-codegen-1.0.0.jar:swagger-codegen-cli-2.1.6.jar io.swagger.codegen.SwaggerCodegen generate -i https://watson-api-explorer.mybluemix.net/listings/conversation-v1-experimental.json -l com.my.company.codegen.Mycustomcodegengenerator -o outputlocation
It should be
java -cp output/myLibrary/target/myCustomCodegen-swagger-codegen-1.0.0.jar;swagger-codegen-cli-2.1.6.jar io.swagger.codegen.SwaggerCodegen generate -i https://watson-api-explorer.mybluemix.net/listings/conversation-v1-experimental.json -l com.my.company.codegen.Mycustomcodegengenerator -o outputlocation
Multiple class paths need to be separated by a semicolon. http://docs.oracle.com/javase/7/docs/technotes/tools/windows/classpath.html

The problem is that you don't specify that correct path of swagger-codegen-2.1.6.jar in your call. Thats why it can't find the main-class.
If you are inside the root project swagger-codegen you should specify it like this: modules/swagger-codegen-cli/target/swagger-codegen-cli.jar
~$ cd ~/git/swagger-codegen # go into your root project
~/git/swagger-codegen$ # ... do the steps you described
~/git/swagger-codegen$ java -cp \
output/myLibrary/target/myClientCodegen-swagger-codegen-1.0.0.jar:modules/swagger-codegen-cli/target/swagger-codegen-cli.jar \
io.swagger.codegen.SwaggerCodegen \
generate -i https://watson-api-explorer.mybluemix.net/listings/conversation-v1-experimental.json \
-l com.my.company.codegen.Mycustomcodegengenerator \
-o outputlocation
Or as a one-liner:
~/git/swagger-codegen$ java -cp output/myLibrary/target/myClientCodegen-swagger-codegen-1.0.0.jar:modules/swagger-codegen-cli/target/swagger-codegen-cli.jar io.swagger.codegen.SwaggerCodegen generate -i https://watson-api-explorer.mybluemix.net/listings/conversation-v1-experimental.json -l com.my.company.codegen.Mycustomcodegengenerator -o outputlocation
Update 1
I'm pretty sure when you construct the classpath with -cp you have a mistake with the swagger-codegen-cli-2.1.6.jar. Please test the following.
Copy both (myClientCodegen-swagger-codegen-1.0.0.jar and swagger-codegen-cli-2.1.6.jar) jars into the same folder. Then go into this folder and try the following:
javap -cp myCustomCodegen-swagger-codegen-1.0.0.jar:swagger-codegen-cli-2.1.6.jar io.swagger.codegen.SwaggerCodegen
javap checks whether the main class io.swagger.codegen.SwaggerCodegen is available. On my machine it prints this:
Compiled from "SwaggerCodegen.java"
public class io.swagger.codegen.SwaggerCodegen {
public io.swagger.codegen.SwaggerCodegen();
public static void main(java.lang.String[]);
}

I had this similar problem when running as Spring Boot app in Eclipse 4.19 in macOS 10.14:
"Error: Could not find or load main class
io.swagger.Swagger2SpringBoot"
…even as I was staring at this very class containing the main method. My solution was issuing a "maven update project". This, like in other situations with odd Java symptoms, fixed the issue.

MapReduce: executing WordCount v1.0

I am trying to learn MapReduce from the official documentation. To make a jar file for WordCount class, the documentation says to run the following command:
javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar -d wordcount_classes WordCount.java
But, I found that my Hadoop directory has no core.jar present. I suppose my Hadoop installation is alright as I can execute the Hadoop shell script from the Bin folder.

If you trying with that:
javac -classpath `hadoop classpath` -d wordcount_classes WordCount.java
Isn't the best practice, I think, but work for me.

Check in your hadoop-1.2.1 folder (as in my case), which you unzipped in "Prepare to Start Cluster" of single node setup. There you would find hadoop-1.2.1-core.jar
That is the file being used to compile here.

how to execute jar file in hadoop platform in ubuntu?

I can create Mapreduce programme, so i can configure hadoop in eclipse. after i can create 1. mapper, 2. reducer, 3.mapreducerDriver
after i can create jar file with help of Makefile in shell command prompt,
after i can use
this command
hadoop jar $ {JarFile} $ {MainFunc} input output
make file
JarFile = "Sample-0.1.jar"
MainFunc = "mypack.Mapreduce"
LocalOutDir = "/ tmp / output"
after i use
jar-cvf $ {Sample-0.1.jar}-C bin /.
jar file created , finally i can write this command.
hadoop jar $ {Sample-0.1.jar} $ {mypack.Mapreduce} input output
finally i get like this error will come in command prompt.
bash: ${mypack.Mapreduce}: bad substitution
how can i solve this problem . pleasae help me
now i find sollution
hadoop jar $ {Sample-0.1.jar} mypack.Mapreduce input output
then hadoop will be run .

Write a script like compile.sh
$ mkdir wordcount_classes
$ javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar -d wordcount_classes WordCount.java
$ jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .
For reference: http://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html

How do you get WordCount.java to compile on Cloudera 4?

I'm trying to compile a simple WordCount.java map-reduce example on a linux (CentOS) installation of Cloudera 4. I keep hitting compiler errors when I reference any of the hadoop classes, but I can't figure out which jars of the hundreds under /usr/lib/hadoop I need to add to my classpath to get things to compile. Any help would be greatly appreciated! What I'd like most is a java file for word count (just in case the one I found is bad for some reason) along with the associated command to compile and run it.
I am trying to do this using just javac rather than Eclipse. My main issue either way is what exactly are the Hadoop libraries from the Cloudera 4 install which I need to include in order to get the classic WordCount example to compile. Basically, I need to put the Java MapReduce API classes (Mapper, Reducer, etc.) in my classpath.

I have a script that builds my hadoop classes. Try:
#!/bin/bash
program=`echo $1 | awk -F "." '{print $1}'`
if [ ! -d "${program}_classes" ]
then mkdir ${program}_classes/;
fi
javac -classpath /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1.jar:/usr/lib/hadoop/client/h\
adoop-mapreduce-client-core-2.0.0-cdh4.0.1.jar -d ${program}_classes/ $1
jar -cvf ${program}.jar -C ${program}_classes/ .;
You were probably missing the key jars:
/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.1.jar
and
/usr/lib/hadoop/client/hadoop-mapreduce-client-core-2.0.0-cdh4.0.1.jar

If you are running the Cloudera CDH4 Virtual Machine then the following should get you running:
javac -classpath /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.0.0.jar:/usr/lib/hadoop/client/hadoop-mapreduce-client-core-2.0.0-cdh4.0.0.jar -d wordcount_classes WordCount.java

Or you can export environment:
export JAVA_HOME=/usr/java/default
export PATH=${JAVA_HOME}/bin:${PATH}
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
and use the commands below:
$ bin/hadoop com.sun.tools.javac.Main WordCount.java
$ jar cf wc.jar WordCount*.class

If you are using Eclipse please do add Hadoop packages. you may get it from java2s or any similar sites. I couldn't say without know anything about what you did till now.