Hadoop with mongoDB : NoClassDefFoundError MongoConfigUtil - java

I'm learning how to write a map / reduce job in hadoop with mongodb data as input. So I followed this example, but I got following error :
Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/hadoop/util/MongoConfigUtil
at WordCount.main(WordCount.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.util.MongoConfigUtil
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
I've been searching for hours without any result. Any one can help me please.

it means mongo-hadoop-core.jar is available at compile time but not at run time.
Try running your application with dependency jars added in classpath
Example : java -cp mongo-hadoop-core.jar<++other dependencies> MainClass
EDIT 1
If you're running using hadoop shell
check the classpath by executing hadoop classpath it will print the dir/jars in the classpath.
If the dependent jars are not in the classpath add them in the classpath using export command then execute hadoop jar yourjar.jar mainClass
EDIT 2
make use of libjars option
hadoop jar myjar.jar mainClass -libjars ${LIBJARS}

I can see this link examples folder structure is maven. I CANT see pom.xml in that link.
We can set maven scope from provided to runtime
You need to write assembly.xml (to package your application lib and related dependencies in tar file) and need to refer it from pom.xml to package mongo-hadoop-core.jar along with other dependencies which are NOT installed in cluster.
export HADOOP_CLASSPATH=`hadoop classpath`:`hbase classpath`
hadoop jar .... -cp $HADOOP_CLASSPATH MainClass
If you unzip the tar file mentioned above and you can refer the classpath for example :
hadoop jar .... ../lib/* main classs
where lib folder contails all your dependencies like your mongodb also.
If mongo db and related components are installed in the cluster we can mention the classpath like below example.
Also see this answer how they have used libjars

Related

java.lang.NoClassDefFoundError on Docker

When The Program is Ran in the Local PC, it is working fine, but if it is running in the docker throws the following error:
Exception in thread "Thread-16" java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/Serializer
at org.eclipse.kura.example.configurable.ConfigurableExample$1.run(ConfigurableExample.java:52)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.Serializer cannot be found by org.eclipse.kura.example.configurable_1.0.0.202201251713
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:484)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:150)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 2 more
But this is working fine in the LocalMachine.
What is the issue ?
This may be debugged in following ways:
Ensure that you have all the required dependencies inside your docker container. You may want to check your docker base image first.
Check if you have set the classpath appropriately to include that jar file (kafka-clients).
Validate your entrypoint/cmd which you use to start the service to check if it has the appropriate classpath.
Check your classloader and see why it is not able to load that class.
If your executable is a fat jar file, check if that jar file contains the kafka-clients library where that class is present.

Javaagent throwing java.lang.NoClassDefFoundError: java/sql/PreparedStatement with java 15

Java agent is throwing this error when running with a java 15 app, if anybody can shed a light on why this might be coming, it will be very helpful.
java.lang.NoClassDefFoundError: java/sql/PreparedStatement
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3325)
at java.base/java.lang.Class.getDeclaredMethods(Class.java:2466)
.
.
.
.
.
Caused by: java.lang.ClassNotFoundException: java.sql.PreparedStatement
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:435)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 19 more
Class not found Exception happens when JVM Doesn't find the class files on the classpath. Either the auto deployment feature(auto-deployment="true") is erroneous or you don't have class file(sql jar) on the classpath. If you are using Maven, Just check if sql jar(prepared statement class file and sql jar file) is on the classpath. If class file not on classpath, Just add the class file manually on classpath. It should resolve the error.
It means java.sql.PreparedStatement is available at compile time but not available at Runtime.
To solve the issue add the sql jar to classpath, if you are using maven add dependency in pom.xml.

Sonar properties files

I have a project that has the following package structure
src/main/proj
-com.company.package
-appName
-morepackages
-appVersion2
-morepackages
sonar-runner.properties
sonarBuild.sh
sonar-runner-project2.properties
sonarBuildProject2.sh
As it stands, with the default properties file i can run the sonar report and it will generate a review for the whole project
sonar.projectKey=KEY
sonar.projectName=PROJNAME
sonar.projectVersion=1.0
sonar.host.url=someurl
#Path for sonar sources
sonar.sources=src/main/java
#Path for libraries
sonar.libraries=target/lib/*.jar
#Path for binaries
sonar.binaries=target/classes
#--------Default database
sonar.jdbc.url=someurl
sonar.jdbc.driver=com.mysql.jdbc.Driver
#----- Global database settings
sonar.jdbc.username=sonar
sonar.jdbc.password=sonar
#----- Default directory layout
sonar.java.source=1.6
sonar.java.target=1.6
sonar.sourceEncoding=UTF-8
Ideally however I would like to separate the two reports so i have one for the original package and one for the appVersion2 package. Now - as mentioned above I have created a separate properties file for each. Should i just point sonar.sources to the respective packages for each job to analyse or is there a better way to do this all in one property file?
Thanks
EDIT
My multi module attempt properties file looks as follows
sonar.projectKey=rootkey
sonar.projectName=rootname
sonar.projectVersion=1.0
sonar.host.url=rooturl
sonar.sources=src/main/java/
sonar.libraries=target/lib/*.jar
sonar.modules=module1,module2
sonar.sourceEncoding=UTF-8
#----- Global database settings
sonar.jdbc.username=user
sonar.jdbc.password=pass
sonar.java.source=1.7
sonar.java.target=1.7
#--------Default database
sonar.jdbc.url=url
sonar.jdbc.driver=com.mysql.jdbc.Driver
module1.sonar.projectName=proja
module2.sonar.projectName=projb
module1.sonar.projectKey=projakey
module2.sonar.projectKey=projbkey
#Path for sonar sources
module1.sonar.sources=src/main/java/app1code
module2.sonar.sources=src/main/java/app2code
#Path for binaries
module1.sonar.binaries=target/classes/app1binaries
module2.sonar.binaries=target/classes/app2binaries
I get an error in the output saying....
Exception in thread "main" org.sonar.runner.RunnerException: org.sonar.runner.RunnerException: The base directory of the module 'module1' does not exist: patthtoapp/module1
at org.sonar.runner.Runner.delegateExecution(Runner.java:288)
at org.sonar.runner.Runner.execute(Runner.java:151)
at org.sonar.runner.Main.execute(Main.java:84)
at org.sonar.runner.Main.main(Main.java:56)
Caused by: org.sonar.runner.RunnerException: The base directory of the module 'module1' does not exist: pathtoapp/module1
at org.sonar.runner.internal.batch.SonarProjectBuilder.setProjectBaseDir(SonarProjectBuilder.java:279)
at org.sonar.runner.internal.batch.SonarProjectBuilder.loadChildProject(SonarProjectBuilder.java:191)
at org.sonar.runner.internal.batch.SonarProjectBuilder.defineChildren(SonarProjectBuilder.java:169)
at org.sonar.runner.internal.batch.SonarProjectBuilder.generateProjectDefinition(SonarProjectBuilder.java:122)
at org.sonar.runner.internal.batch.Launcher.execute(Launcher.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.sonar.runner.Runner.delegateExecution(Runner.java:285)
You have to specify the projectBaseDir if the module name doesn't match you module directory.
Since both your module are located in ".", you can simply add the following to your sonar-project properties:
module1.sonar.projectBaseDir=.
module2.sonar.projectBaseDir=.
Sonar will handle your modules as components of the project:
EDIT
If both of your modules are located in the same source directory, define the same source folder for both and exclude the unwanted packages with sonar.exclusions:
module1.sonar.sources=src/main/java
module1.sonar.exclusions=app2code/**/*
module2.sonar.sources=src/main/java
module2.sonar.exclusions=app1code/**/*
More details about file exclusion
You can define a Multi-module project structure, then you can set the configuration for sonar in one properties file in the root folder of your project, (Way #1)
Do the build job on Jenkins first without Sonar configured. Then add Sonar, and run a build job again. Should fix the problem

First tomcat application - 'ant install' error "NoClassDefFound"

JSP/servlets are something I would like to learn, so I set about creating my first tomcat application.
I have gone through the apache tutorial here, and progressed to trying to install my application in tomcat (simple test html file with appropriate dir structure). Now, I realise its very wishful thinking hoping to use the stock build.xml provided by the tutorial but I need to start somewhere, Im not sure how to write one myself just yet. I have looked for examples, but they dont seem to have 'install' targets, perhaps they are not web applications.
My project builds, but does not install onto tomcat via ant. When I attempt to 'ant install' my console outputs...
ant install
Buildfile: /home/mark/svn/tomcatapp/build.xml
Trying to override old definition of datatype resources
prepare:
compile:
[javac] /home/mark/svn/tomcatapp/build.xml:299: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
install:
BUILD FAILED
/home/mark/svn/tomcatapp/build.xml:375: java.lang.NoClassDefFoundError: org/apache/tomcat/util/buf/B2CConverter
at org.apache.catalina.util.Base64.encode(Base64.java:173)
at org.apache.catalina.ant.AbstractCatalinaTask.execute(AbstractCatalinaTask.java:204)
at org.apache.catalina.ant.DeployTask.execute(DeployTask.java:211)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:392)
at org.apache.tools.ant.Target.performTasks(Target.java:413)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
at org.apache.tools.ant.Main.runBuild(Main.java:811)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
Caused by: java.lang.ClassNotFoundException: org.apache.tomcat.util.buf.B2CConverter
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 19 more
Total time: 0 seconds
Why would a class fail to be found, would it not be contained within the tomcat install? There is a similar question, but I am not attempting to use the manager app, and I am not sure how to edit the class path - would I need to have compiled a .java class to create one for the project?
I am sure a more simple build file would suffice, but I am not sure which components are relevant. Sorry for the potentially very stupid questions.
Here is my install target from my build.xml file.
<target name="install" depends="compile"
description="Install application to servlet container">
<deploy url="${manager.url}"
username="${manager.username}"
password="${manager.password}"
path="${app.path}"
localWar="file://${build.home}"/>
</target>
If you're using the sample build.xml , change
localWar="file://${build.home}"
to
war="${dist.home}/${app.name}-${app.version}.war"
If you have catalina-ant.jar in ANT_HOME/lib. Try removing it from this location.
Restart tomcat.
That has worked for me.
Short answer:
copy file tomcat-util.jar from CATALINA_HOME/lib to ANT_HOME/lib OR
when issuing ant commands, add -lib option with a path to that file,
for example ant -lib C:\Tomcat8\lib\tomcat-util.jar install (provided CATALINA_HOME points to C:\Tomcat8)
Explanation:
I have encountered similar problem with Tomcat 8.0.36. According to error message class org.apache.tomcat.util.codec.binary.Base64 was not found. ant install command from the tutorial executes deploy task, that is not a standard task in Ant, but a custom task provided by Tomcat. Deploy task is implemented by DeployTask class (full name of the class is org.apache.catalina.ant.DeployTask) which is placed in CATALINA_HOME/lib/catalina-ant.jar. Custom tasks (like DeployTask) usually are dependent on external libraries, so you need to add those libraries to Ant's classpath. For more information see Optional Tasks in Ant.
You have hit https://issues.apache.org/bugzilla/show_bug.cgi?id=52148
Upgrade to the latest Tomcat 7 release and the problem will go away.
I am also running a new version of Tomcat (7.0.32) and encountered the same error while trying to follow the tutorial. I know this isn't the best solution, but perhaps it will be helpful in your debugging: When I use the -lib option with ant I was able to make it pull in the jar files with the needed classes:
ant -lib /usr/local/apache-tomcat-7.0.32/lib/ -lib /usr/local/apache-tomcat-7.0.32/bin/ install (In my case /usr/local/apache-tomcat-7.0.32 is $CATALINA_HOME)

How to run a maven-packaged clojure application from the jar

I have the following contents in src/main/clojure/za/co/pb/maven_test/test.clj file:
(ns za.co.pb.maven-test.test
(:gen-class))
(defn -main []
(println "Hello world!"))
I also have a POM that has the necesary dependencies on clojure-maven-plugin with the compile execution.
If I execute a mvn package command, I get a target/maven-test-1.0-SNAPSHOT.jar file and if I look in the classes folder I have these files in the folder target/classes/za/co/pb/maven_test:
test.class
test.clj
test__init.class
test$loading__4410__auto__.class
test$_main.class
This, as far as I know, is appropriate.
However, when I run this command:
java -cp target\app-1.0-SNAPSHOT.jar za.co.pb.maven_test.test
I get this:
Exception in thread "main" java.lang.NoClassDefFoundError: clojure/lang/IFn
Caused by: java.lang.ClassNotFoundException: clojure.lang.IFn
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: za.co.pb.maven_test.test. Program will exit.
You don't have the clojure jars in the classpath. You can either embed them as per Stuart's response, or if you don't like having all the dependencies embedded in a single jar, you can use the dependency plugin and the jar plugin to get this working nicely.
See: http://groups.google.com/group/enclojure/msg/87159854fcb0e708 for a summary (note, there's a typo in the linked post, the package should be called "foo").
You need to generate a JAR file file that includes all the dependencies of your project. The Maven Assembly Plugin does this for you, using the built-in jar-with-dependencies descriptor.
If you are using leiningen, after generating the jar with lein uberjar, do not use java -jar target/your-name.jar, but use java -jar target/your-name-standalone.jar.
Same thing for boot.
Are all dependencies available ? I'm not familiar with the clojure.lang.IFn class. Is it from an external library? If it is did you marked the dependency with the provided scope?

Categories

Resources