I am starting spark-shell (of spark 2.2) and added bunch of jars in spark-shell command (from Ignite 2.1 directory).
Still getting error:
Can't load log handler "org.apache.ignite.logger.java.JavaLoggerFileHandler"
Also followed recommendation from here:
https://apacheignite.readme.io/v1.2/docs/installation--deployment
# Optionally set IGNITE_HOME here.
# IGNITE_HOME=/path/to/ignite
IGNITE_LIBS="${IGNITE_HOME}/libs/*"
for file in ${IGNITE_HOME}/libs/*
do
if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ]; then
IGNITE_LIBS=${IGNITE_LIBS}:${file}/*
fi
done
export SPARK_CLASSPATH=$IGNITE_LIBS
Also set logging to only ERROR as well but still getting error:
Can't load log handler "org.apache.ignite.logger.java.JavaLoggerFileHandler"
java.lang.ClassNotFoundException: org.apache.ignite.logger.java.JavaLoggerFileHandler
java.lang.ClassNotFoundException: org.apache.ignite.logger.java.JavaLoggerFileHandler
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.util.logging.LogManager$5.run(LogManager.java:965)
at java.security.AccessController.doPrivileged(Native Method)
Looks like you use documentation for an old Ignite version 1.2, while you use Ignite 2.1. Check the documentation for the latest version here: https://apacheignite-fs.readme.io/v2.2/docs/installation-deployment
Also, please make sure that have configured IGNITE_HOME in your environment. JavaLoggerFileHandler placed in the ignite-core module, looks like spark classpath doesn't see any Ignite lib at all.
The documentation describe the issue here:
https://apacheignite-fs.readme.io/v2.2/docs/troubleshooting
This issue appears when you do not have any loggers included in classpath and Ignite tries to use standard Java logging. By default Spark loads all user jar files using separate class loader. Java logging framework, on the other hand, uses application class loader to initialize log handlers. To resolve this, you can either add ignite-log4j module to the list of the used jars so that Ignite would use Log4j as a logging subsystem, or alter default Spark classpath as described
Related
I have built a Java 1.8 application under Maven in Eclipse that allows me to generate BIRT reports in a programmatic way without the need for the UI. Within Eclipse everything works fine. If I compile the code into a JAR file and try to run it however I run into problems with java.lang.NoClassDefFoundErrors. Clearly I have a missing dependancy but I don't know how to resolve the problem.
The error is
java -cp target/* com.resonancex.scheduler.Scheduler
Establishing conection...
Connected
Got products to report
ProductId=1 ProductType=BREN Isin=XS1939207996 Template=BREN_MU
* birtEngine EngineConfig: org.eclipse.birt.report.engine.api.EngineConfig.emitterConfigs={html=org.eclipse.birt.report.engine.api.HTMLEmitterConfig#71bbf57e};
* Birt Engine Startring ....
Exception in thread "main" java.lang.NoClassDefFoundError: org/eclipse/core/runtime/CoreException
at org.eclipse.birt.core.framework.Platform.createPlatformLauncher(Platform.java:115)
at org.eclipse.birt.core.framework.Platform.startup(Platform.java:74)
at com.resonancex.scheduler.BirtEngine.start(BirtEngine.java:33)
at com.resonancex.scheduler.BirtEngine.getBirtEngine(BirtEngine.java:47)
at com.resonancex.scheduler.BIRTReportGenerator.setReportCustom(BIRTReportGenerator.java:67)
at com.resonancex.scheduler.Scheduler.generateAllReports(Scheduler.java:68)
at com.resonancex.scheduler.Scheduler.main(Scheduler.java:79)
Caused by: java.lang.ClassNotFoundException: org.eclipse.core.runtime.CoreException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I added the library org.eclipse.birt.runtime_4.8.0-20180626-4.8.0 which appears to be the required BIRT runtime. Can anyone give me a few pointers as to what else I may need?
Just in case anyone else tries to do this I thought I would list the libraries I needed to install to get standalone BIRT report generation running running. The list is as follows. Obviously this list may change over time so keep that in mind:
axis-1.4.jar
batik-anim-1.7.jar
batik-awt-util-1.7.jar
batik-bridge-1.7.jar
batik-css-1.7.jar
batik-dom-1.7.jar
batik-gvt-1.7.jar
batik-i18n-1.9.jar
batik-parser-1.7.jar
batik-script-1.7.jar
batik-svg-dom-1.7.jar
batik-transcoder-1.7.jar
batik-util-1.8.jar
batik-xml-1.7.jar
hibernate-agroal-5.4.9.Final.jar
icu4j-4.8.jar
itext-2.1.7.jar
mssql-jdbc-7.4.1.jre8.jar
org.apache.batik.css-1.6.0.jar
org.eclipse.birt.runtime_4.8.0-20180626-4.8.0.jar
org.eclipse.core.runtime_3.14.0.v20180417-0825-4.8.0.jar
org.eclipse.datatools.connectivity.oda-3.5.0.201603142002.jar
org.eclipse.datatools.connectivity.oda.consumer-3.3.0.201603142002.jar
org.eclipse.emf.common-2.16.0.jar
org.eclipse.emf.ecore-2.11.1.jar
org.eclipse.emf.ecore.xmi-2.11.1.jar
org.eclipse.equinox.common_3.10.0.v20180412-1130-4.8.0.jar
org.eclipse.equinox.registry_3.8.0.v20180426-1327-4.8.0.jar
org.eclipse.osgi_3.13.0.v20180409-1500-4.8.0.jar
org.mozilla.javascript-1.7.5.v201504281450.jar
org.w3c.dom.smil_1.0.1.v200903091627-4.8.0.jar
org.w3c.dom.svg-1.1.0.jar
sac-1.3.jar
xml-apis-1.4.01.jar
I'm using java with apache tika 1.18 to convert some files to TXT.
When I try to use the AutoDetectParser(), I'm getting the error :
[ERROR ] Error occurred during error handling, give up!
org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
[ERROR ] SRVE0777E: Exception thrown by application class 'org.apache.cxf.service.invoker.AbstractInvoker.createFault:162'
org.apache.cxf.interceptor.Fault: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162)
at [internal classes]
Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
I was dinging on internet and found this error related wrong version of commom_compress, appears this method doesn't exist in versions previous of 1.14 of commom_compress. In my case the version is 1.16.1.
After build the project, I checked the libs inside and there is only the correct version.
I'm using IBM Liberty 18.0 ... and now I'm really lost about options to solve this problem.
When I use the specific parser, like PDFParser(), everything works fine!
Any ideas?
Thanks
Source of the issue:
Spark 2.x distributions include old versions of commons-compress, while Tika library depends on version 1.18 of commons-compress library.
Solution
Use --driver-class-path argument in your spark-shell or spark-submit to point to a the right version of commons-compress library.
spark-submit
--driver-class-path ~/.m2/repository/org/apache/commons/commons-compress/1.18/commons-compress-1.18.jar
--class {you.main.class}
....
Check my detailed answer here.
I have a simple MapReduce program which I want to run it on a remote cluster. I can do this from command line by simply running
hadoop jar myjar.jar input output
but when I want to run a function in my junit TestCase class from my IDE which invokes the MR job, I get the following warnings:
WARN org.apache.hadoop.mapreduce.JobSubmitter - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
INFO org.apache.hadoop.mapred.YARNRunner - Job jar is not present. Not adding any jar to the list of resources.
although I have this line set, before submitting the MR job:
job.setJarByClass(MyJob.class);
and hence the job fails as it cannot find the appropriate classes (like MyMapKey which is the mapper key class) to operate.
Error: java.io.IOException: Initialization of all the collectors failed. Error in last collector was :java.lang.RuntimeException: java.lang.ClassNotFoundException: Class MyMapKey not found
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:414)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:698)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
any thoughts on this?
First you should add remote Hadoop cluster config files (i.e. core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml, ssl-client.xml) as resources to your Configuration object. Then follow the steps in the above link to see how you should add manually the job jar to classpath on remote cluster.
I'm getting a confusing ClassNotFoundException when I try to run ExportSnapshot from my HBase master node. hbase shell and other commands work just fine, and my cluster is fully operational.
This feels like a Classpath issue, but I don't know what I'm missing.
$ /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot ambarismoketest-snapshot -copy-to hdfs://10.0.1.90/apps/hbase/data -mappers 16
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2015-10-13 20:05:02,339 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2015-10-13 20:05:04,351 INFO [main] util.FSVisitor: No logs under directory:hdfs://cinco-de-nameservice/apps/hbase/data/.hbase-snapshot/impression_event_production_hbase-transfer-to-staging-20151013/WALs
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/Job
at org.apache.hadoop.hbase.snapshot.ExportSnapshot.runCopyJob(ExportSnapshot.java:529)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:646)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:697)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:701)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.Job
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 5 more
Problem
It turns out this is because the mapreduce2 JARs are not available in the classpath. The classpath was properly set up, but I did not have the mapreduce2 client installed on that node. HBase's ExportSnapshot apparently depends on those client JARs when exporting snapshots to another cluster because it writes to HDFS.
Fix
If you use Ambari:
Load Ambari UI
Pull up node where you were running the ExportSnapshot from and getting the above error
Under "components", click "Add"
Click "Mapreduce 2 client"
Background
There's a ticket here https://issues.apache.org/jira/browse/HBASE-9687 where the title is ClassNotFoundException is thrown when ExportSnapshot runs against hadoop cluster where HBase is not installed on the same node as resourcemanager. The title implies that installing resource manager is the fix and this may work; however, the crux is you need the hadoop mapreduce2 jars in the classpath and you can do that by simply installing the mapreduce2 client.
For us, specifically, the reason our snapshot exports were working one day and broken the next is that our HBase master switched on us b/c of another issue we had. Our backup HBase master did not have the mapreduce2 client JARs, but the original primary master did.
I am trying to run custom flume agent from terminal using linux. I am working on cloudera VM. Command running flume looks like:
flume-ng agent --conf . -f spoolDirLocal2hdfs_memoryChannel.conflume.root.logger=DEBUG,console -n Agent5
Sources with interceptor looks like:
Agent5.sources.spooldir-source.interceptors = i1
Agent5.sources.spooldir-source.interceptors.i1.type = org.flumefiles.flume.HtmlInterceptor$Buider
I've placed my jar file both into /usr/lib/hadoop/lib/ and /usr/lib/flume-ng/lib/. Also I've created plugins.d at /usr/lib/flume-ng/plugins.d/ and placed jar there. But when running flume agent I've got an error:
15/02/18 06:10:46 ERROR channel.ChannelProcessor: Builder class not found. Exception follows.
java.lang.ClassNotFoundException: org.intropro.flume.HtmlInterceptor$Buider
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
.....
Where should I place my jar file to make it find builder?
Place it into FLUME_HOME/lib and then restart flume.
If that doesn't work, make sure your interceptor actually implements the Builder interface. That might be another reason.