I am trying to run a MapReduce job to scan a HBase table. Currently I am using the version 0.94.6 of HBase that comes with Cloudera 4.4. At some point in my program I use Scan(), and I properly import it with:
import org.apache.hadoop.hbase.client.Scan;
It compiles well and I am able to create a jar file too. I do it by passing the hbase classpath as the value for the -cp option. When running the program, I obtain the following message:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/Scan
I run the code using:
hadoop jar my_program.jar MyJobClass -libjars <list_of_jars>
where list_of_jars contains /opt/cloudera/parcels/CDH/lib/hbase/hbase.jar. Just to double-check, I confirmed that hbase.jar contains Scan. I do it with:
jar tf /opt/cloudera/parcels/CDH/lib/hbase/hbase.jar
And I can see the line:
org/apache/hadoop/hbase/client/Scan.class
in the output. All looks ok to me. I don't understand why is saying that Scan is not defined. I pass the correct jar, and it contains the class.
Any help is appreciated.
Setting the HADOOP_CLASSPATH variable fixed the issue:
export HADOOP_CLASSPATH=`/usr/bin/hbase classpath`
Related
Problem explanation
I have recently been trying to use Apache Jena with Java (rather than on the command line). I wrote a simple script to convert read and write differetn RDF format types, as so
import org.apache.jena.riot.RDFDataMgr;
import org.apache.jena.query.Dataset;
import org.apache.jena.riot.Lang;
public class Go_NT
{
public static void main(String[] args)
{
Dataset dataset = RDFDataMgr.loadDataset("triail.nq");
RDFDataMgr.write(System.out, dataset, Lang.NTRIPLES);
}
}
triail.nq is a test nquads file containing 81 quads.
I invoked it as so:
javac -cp "/mnt/e/Tráchtas/apache-jena-3.17.0/lib/*" Go_NT.java
java Go_NT
It compiles without error, but when I run it, it returns an error
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/jena/riot/RDFDataMgr
at Go_NT.main(Go_NT.java:9)
Caused by: java.lang.ClassNotFoundException: org.apache.jena.riot.RDFDataMgr
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 1 more
What I have tried
I have looked around and seen that this error occurs almost always because a necessary .jar file is not included, so a class referenced by the code cannot be loaded. The solution to these other issues was to include all of /apache-jena-3.17.0/lib/* . Oddly enough, that has not worked for me--I do include all of the contents of lib/ in my classpath, but I am still seeing the error.
System notes
I am running Jena 3.17.0, using the default Linux binaries available here (https://jena.apache.org/download/index.cgi). I have not added or removed any other Jena modules.
I am running this in the Windows Subsystem for Linux (version 2) with Ubuntu 20.04.
If any of you have any insight into what could be causing this, I would greatly appreciate it!
Based on the comment by vvs, the link https://howtodoinjava.com/java-examples/set-classpath-command-line/ helped out a lot. There were 2 issues: I needed to include the classpath in the java command, not just javac. I also needed to include the current directory where the output of javac would be.
I fixed this by setting the CLASSPATH variable, and then adding all the needed directories to that. (You could also do this by adding the classpath into the -cp argument). Note that the : separates different directories.
In short, here is what I did:
export CLASSPATH=/mnt/e/Tráchtas/apache-jena-3.17.0/lib/*:.
javac Go_NT.java
java Go_NT
Note that you need to re-assign CLASSPATH each time you open a new terminal.
I was trying to run the basic WordCount example of Apache MapReduce 2.7 from here:
https://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v1.0
I put the input files at : /user/hadoopLearning/WordCount/input/
Output path : /user/hadoopLearning/WordCount/output/
then I ran the following command :
hadoop jar wc.jar WordCount /user/hadoopLearning/WordCount/input/file01 /user/hadoopLearning/WordCount/output
However on running I am getting the following error:
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: **Output directory** hdfs://sandbox.hortonworks.com:8020/user/hadoopLearning/WordCount/**input**/file01 already exists
I haven't written a single piece of code and copied everything from above location from Apache's website.
I understand the error , but the if we closely look at the error it says the output directory already exists and in the stack trace it gives the path of input directory.
Please can anyone help me. I am a beginner in the field of hadoop. Thanks in advance.
You're trying to create a file which already exists.
HDFS doesn't allow that.
replace your output path ('/user/hadoopLearning/WordCount/output'), with something else.
try this command
hadoop jar wc.jar WordCount /user/hadoopLearning/WordCount/input/file01 /user/hadoopLearning/WordCount/new_output_path
so I've tried many of the solutions that are present in this website but none could help me.
The problem:
I have my project files structured like this:
cnv/webserver/aplication/insProj.java
cnv/webserver/aplication/IntFactorization.java
and both are in the package webserver.aplication.
to compile and run i do this:
starting at the folder cnv and with the $CLASSPATH=/home/ll/Documents/cnv
cd webserver/aplication
javac -source 1.4 insProj.java
javac IntFactorization.java
cd ..
cd ..
java -XX:-UseSplitVerifier webserver.aplication.insProj ./webserver/aplication/IntFactorization.class
java -XX:-UseSplitVerifier webserver.aplication.IntFactorization 5
When executing this last instruction, it returns an exception:
Factoring 5...
Exception in thread "main" java.lang.NoClassDefFoundError: webserver.aplication.insProj
at webserver.aplication.IntFactorization.calcPrimeFactors(IntFactorization.java:22)
at webserver.aplication.IntFactorization.main(IntFactorization.java:59)
The instrumentations i'm making is incrementing a counter when a given method is executed and creating a file when the class ends execution. I know this works because i've tested outside these packages and it works fine.
It seems BIT doesn't work well with packages. In the end, I removed insProj from any package and inserted its folder in the classpath. Then it began to work.
i am using Linux ubuntu and i have created a java program named hola.java which is the following program code, this program works perfectly
import javax.swing.*;
import java.awt.*;
public class hola extends JFrame {
JButton b1 = new JButton("presionar");
hola(){
super("Botones");
setSize(250,250);
setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
FlowLayout flo=new FlowLayout();
setLayout(flo);
add(b1);
setVisible(true);
}
public static void main(String[] args)
{
hola bt = new hola();
}
}
This java program works perfectly when it runs
Now I created a jar file of this program using in command line:
jar cf hola.jar hola.class
This creates a Jar file named hola.jar
I even wrote Main-Class: hola in the manifest.mf file.
When I try to run this by:
java -jar hola.jar
I get an error: An unexpected error occurred while trying to open file hola.jar
Please tell me how to run jar file so that I get an output :'( , what could be the possible reason that i can not run this program as a jar file, even the program works perfectly using "java hola.java"
This error mostly indicates invalid MANIFEST.MF file. Perhaps long line, missing final line terminator, accidental empty line in the middle, ... many things can go wrong. Using -cp just goes around the problem, but does not fix the root cause.
To run a java file within a jar file, you don't need to open it. You simply need to ensure your classpath is having the given jar file
If the class is within a package then you can run using
java -cp hola.jar package.hola
If the class is not in a package then simply use
java -cp hola.jar hola
If you are not within the directory where hola.jar is located, then you can try following:
In case of within package
java -cp /locationOfJar/hola.jar package.hola
or In case of not in package
java -cp /locationOfJar/hola.jar hola
Error can also happen with > 65535 files in the .jar on some OS
I experienced same error message attempting to launch via -jar:
$ java -jar app.jar
Error: An unexpected error occurred while trying to open file: app.jar
Root cause is that my sbt-assembly output .jar file contains more than 65535 entries.
$ zipinfo app.jar |wc -l
65543
Solution: Short term solution was removal of older dependencies to lower the number of assembled .class files below the 16 bit file count limit.
Long term solution will involve testing Zip64 jvm support on the target OS. Unsure of why the zip64 auto negotiation isn't occurring automatically.
This issue is reproducible using sbt-assembly 15.0, openjdk version "11.0.8" on MacOSX 10.15.7.
Start of the code review:
package java.util.zip;
...
public
class ZipOutputStream extends DeflaterOutputStream implements ZipConstants {
/**
* Whether to use ZIP64 for zip files with more than 64k entries.
* Until ZIP64 support in zip implementations is ubiquitous, this
* system property allows the creation of zip files which can be
* read by legacy zip implementations which tolerate "incorrect"
* total entry count fields, such as the ones in jdk6, and even
* some in jdk7.
*/
private static final boolean inhibitZip64 =
Boolean.parseBoolean(
GetPropertyAction.privilegedGetProperty("jdk.util.zip.inhibitZip64"));
I am continuing to have trouble with the import.bat file for the Neo4j batch importer. I started a new thread as the original problem was resolved.
from the command prompt I run
import.bat test.db sample\nodes.csv sample\rels.csv
With some variations on the path listing for the files, including absolute paths. I continue to get the following error message
The system cannot find the path specified.
Error: Could not find or load main class org.neo4j.batchimport.Importer
I also tried running import.sh from Cygwin and in my Debian VM but keep getting the error
Error: Could not find or load main class org.neo4j.batchimport.Importer
What am I doing wrong?
Please download the zip-file, not the github clone.
This is a pre-build binary as outlined in the readme, that doesn't require that you have to have maven installed to build it.