I have a spark cluster consists of 5 nodes and I have a spark job written in Java that read set of files from a directory and send the content to Kafka.
When I was testing the job locally, everything was working fine.
When I tried to submit the job to the cluster, the job failed with FileNoTFoundException
The files need to be processed exists in a directory mounted on all the 5 nodes so I am sure the file path appears in the exception exists.
Here is the exception appears while submitting the job
java.io.FileNotFoundException: File file:/home/me/shared/input_1.txt does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108)
at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:239)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:216)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
The directory /home/me/shared/ is mounted on all the 5 nodes.
EDIT:
Here is the command I am using to submit the job
bin$ ./spark-submit --total-executor-cores 20 --executor-memory 5G --class org.company.java.FileMigration.FileSparkMigrator --master spark://spark-master:7077 /home/me/FileMigrator-0.1.1-jar-with-dependencies.jar /home/me/shared kafka01,kafka02,kafka03,kafka04,kafka05 kafka_topic
I faced a weird behavior. I submitted the job while the directory contains only one file, the exception is thrown on the driver but the file is processed successfully. Then, I added another file, the same behavior occurred. But, once I added the third file, the exception is thrown and the job failed.
EDIT 2
After some tries, we discovered that there was a problem in the mounted directory that caused this weird behavior.
Spark defaults to HDFS by default. This looks like an NFS file, so try to access it with: file:///home/me/shared/input_1.txt
Yes, three /!
Here is what solve the problem for me. It is weird and I have no idea what the actual problem was.
Simply I asked the sysadmin to mount another directory instead of the one I was using. After that, everything worked fine.
Is seems there was an issue in the old mounted directory but I have no idea what was the actual problem.
Related
Since upgrading my os to Mac Catalina,
I'm getting the following error when trying to run the following:
MysqldResource mysqldResource = new MysqldResource(uri); // Uri - temp db dir
mysqldResource.start("embedded-mysqld-thread-" + System.currentTimeMillis(), databaseOptions);
Looks like the source of the problem is with the dependency "mysql-connector-mxj-db-files" having binary files in 32bit format, while catalina stopped supporting it.
This is the error I'm getting:
Exception in thread "embedded-mysqld-thread-1574931580040" com.mysql.management.util.WrappedException: java.io.IOException: Cannot run program "/private/var/folders/l5/4ywgvbbj1kvgvrwxzp6dz82c0000gn/T/test_db_24166997810717/bin/mysqld": error=86, Bad CPU type in executable
at com.mysql.management.util.Exceptions.toRuntime(Exceptions.java:55)
at com.mysql.management.util.Exceptions$Block.exec(Exceptions.java:89)
at com.mysql.management.util.RuntimeI$Default.exec(RuntimeI.java:180)
at com.mysql.management.util.Shell$Default.run(Shell.java:140)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Cannot run program "/private/var/folders/l5/4ywgvbbj1kvgvrwxzp6dz82c0000gn/T/test_db_24166997810717/bin/mysqld": error=86, Bad CPU type in executable
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at java.lang.Runtime.exec(Runtime.java:620)
at com.mysql.management.util.RuntimeI$Default$1.inner(RuntimeI.java:177)
at com.mysql.management.util.Exceptions$Block.exec(Exceptions.java:86)
... 3 more
Caused by: java.io.IOException: error=86, Bad CPU type in executable
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 6 more
Does anyone know of an alternative to mysql-connector-java or knows of a way to run the existing dependency on Mac OS Catalina?
For anyone who has the same issue on MacOS Catalina, you can follow the README.md in my repo (https://github.com/pengyue/mysql-connector-mxj-mac-os-catalina), which has the 64 bits executables for MacOS Catalina, and this improved connector works on Catalina.
To explain what I have done:
Download the 64 bits executables from MySQL to the folder 5-5-9
Add the platform mapping for 64 bits MacOS in platform-map.properties
Create a jar by using jar cvf mysql-connector-mxj-db-files-5.0.12.jar .
Replace the jar in your mvn settings folder ~/.m2/repository/mysql/mysql-connector-mxj-db-files/5.0.12/ with the jar you just created.
This solution works fine for my projects, Unfortunately I could not find a github for the mysql-connector-mxj, as it is deprecated and not maintainable anymore.
The other option is to use wix-embedded-mysql(https://github.com/wix/wix-embedded-mysql) instead of mysql-connector-mxj, but this probably requires some code changes in your projects.
I am trying to execute flume to get data from twitter stream but received this error while executing the flume.
[ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)] Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows.
java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StreamListener;)V
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:140)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I am a beginner to flume and working in Cloudera quickstart. While searching for solutions it was said to install maven and then build the flume-snapshot jar file from there but I don't know how I can install maven in Cloudera quickstart. Any help on how to correct this error please I have been stuck here for 1 week.
Found the solution:
The conflict is raised by twitter4j files and flume snapshot jars. So I renamed the twitter4j jars by changing their file extenstion with jarx. Another thing I did by reading from this article is to put flume snapshot in following hierarchy.
/usr/lib/flume-ng/lib/plugins.d/flumesnapshot and following same pattern in var directory.
Good day,
I tried to use POI in my application to load excel sheet.
everything is working fine on local tomcat but when I move it to server WebLogic 12.2 that run on Java 8, I got an error as follow.
I tried all versions from POI 3.8 to 3.17(latest) but none of them is working correctly on the server.
Error checking for java lib
java.lang.ExceptionInInitializerError
at org.apache.poi.openxml4j.opc.OPCPackage.init(OPCPackage.java:161)
at org.apache.poi.openxml4j.opc.OPCPackage.(OPCPackage.java:141)
at org.apache.poi.openxml4j.opc.ZipPackage.(ZipPackage.java:97)
at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:324)
at org.apache.poi.util.PackageHelper.open(PackageHelper.java:37)
Truncated. see log file for complete stacktrace
Caused By: java.lang.ClassCastException:
com.ctc.wstx.stax.WstxEventFactory cannot be cast to
javax.xml.stream.XMLEventFactory
at javax.xml.stream.XMLEventFactory.newInstance(XMLEventFactory.java:30)
at org.apache.poi.openxml4j.opc.internal.marshallers.PackagePropertiesMarshaller.(PackagePropertiesMarshaller.java:41)
at org.apache.poi.openxml4j.opc.OPCPackage.init(OPCPackage.java:161)
at org.apache.poi.openxml4j.opc.OPCPackage.(OPCPackage.java:141)
at org.apache.poi.openxml4j.opc.ZipPackage.(ZipPackage.java:97)
Truncated. see log file for complete stacktrace
commons-codec-1.11
commons-collections4-4.1
commons-logging-1.2
junit-4.12
poi-3.17
poi-excelant-3.17
poi-ooxml-3.17
poi-ooxml-schemas-3.17
poi-scratchpad-3.17
xmlbeans-2.6.0
what could be missed?
Note: I cannot use Maven or similer technologies.
thanks
We found one jar file causes the issue. the name is jsr173_api.jar. once this is deleted, all works fine.
I've been working on a Java Jersey RESTful web application and everything was going fine until I compiled once again and got the following compiler error.
I've Googled this issue and many suggested doing the File->Invalidate Caches/Restart... but this did not work.
I also tried manually deleting the files in the Mac OS X directory: /Libary/Caches/IntellijIdea14. This also did not work.
Has anyone ever experienced this before? I'm very confused why this error came and what exactly is causing it. It's certainly putting a delay to my development!
Stack Trace:
Information:Internal caches are corrupted or have outdated format, forcing project rebuild: java.io.FileNotFoundException: /Users/grantmcgovern/Dropbox/Developer/Projects/1834Software/GymAPI/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/target/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/target/classes/target/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/.idea/libraries/Maven__org_glassfish_jersey_test_framework_providers_jersey_test_framework_provider_grizzly2_2_17.xml (File name too long)
Information:4/5/15, 10:39 PM - Compilation completed with 1 error and 0 warnings in 29 sec
Error:Internal error: (java.io.FileNotFoundException) /Users/grantmcgovern/Dropbox/Developer/Projects/1834Software/GymAPI/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/target/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/target/classes/target/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/.idea/libraries/Maven__org_glassfish_jersey_test_framework_providers_jersey_test_framework_provider_grizzly2_2_17.xml (File name too long)
java.io.FileNotFoundException: /Users/grantmcgovern/Dropbox/Developer/Projects/1834Software/GymAPI/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/target/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/target/classes/target/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/out/artifacts/GymAPI_war_exploded/WEB-INF/classes/.idea/libraries/Maven__org_glassfish_jersey_test_framework_providers_jersey_test_framework_provider_grizzly2_2_17.xml (File name too long)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
at com.intellij.openapi.util.io.FileUtil.openOutputStream(FileUtil.java:508)
at com.intellij.openapi.util.io.FileUtil.performCopy(FileUtil.java:460)
at com.intellij.openapi.util.io.FileUtil.copyContent(FileUtil.java:454)
at org.jetbrains.jps.incremental.artifacts.instructions.FilterCopyHandler.copyFile(FilterCopyHandler.java:40)
at org.jetbrains.jps.incremental.artifacts.instructions.FileBasedArtifactRootDescriptor.copyFromRoot(FileBasedArtifactRootDescriptor.java:100)
at org.jetbrains.jps.incremental.artifacts.IncArtifactBuilder.build(IncArtifactBuilder.java:159)
at org.jetbrains.jps.incremental.artifacts.IncArtifactBuilder.build(IncArtifactBuilder.java:50)
at org.jetbrains.jps.incremental.IncProjectBuilder.buildTarget(IncProjectBuilder.java:855)
at org.jetbrains.jps.incremental.IncProjectBuilder.runBuildersForChunk(IncProjectBuilder.java:836)
at org.jetbrains.jps.incremental.IncProjectBuilder.buildTargetsChunk(IncProjectBuilder.java:894)
at org.jetbrains.jps.incremental.IncProjectBuilder.buildChunkIfAffected(IncProjectBuilder.java:789)
at org.jetbrains.jps.incremental.IncProjectBuilder.buildChunks(IncProjectBuilder.java:612)
at org.jetbrains.jps.incremental.IncProjectBuilder.runBuild(IncProjectBuilder.java:352)
at org.jetbrains.jps.incremental.IncProjectBuilder.build(IncProjectBuilder.java:191)
at org.jetbrains.jps.cmdline.BuildRunner.runBuild(BuildRunner.java:137)
at org.jetbrains.jps.cmdline.BuildSession.runBuild(BuildSession.java:289)
at org.jetbrains.jps.cmdline.BuildSession.run(BuildSession.java:124)
at org.jetbrains.jps.cmdline.BuildMain$MyMessageHandler$1.run(BuildMain.java:238)
at org.jetbrains.jps.service.impl.SharedThreadPoolImpl$1.run(SharedThreadPoolImpl.java:41)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Oddly enough, the following worked (since it's a Maven project):
$ mvn clean
I suppose cleaning the modules must have done something because it built just fine thereafter. I have to believe this is some sort of Intellij bug.
Thank you for taking time to look at my question.
I developing a Java WAR file intended for distribution to only one server at a time. So far, I have tested on 3 different servers with success, one Debian, one Ubuntu and the other, the Windows machine I'm developing on.
On the final deployment server, for some reason the app deploys fine, and upon each startup, runs well for about 3 minutes. Database connectivity works, everything runs fine. After 3 minutes however, I start seeing NoClassDefFoundError's consistently every time the app tries to access the DB.
I'm the following libraries with all of them packaged in the WAR file:
Mysql/j Connecter, latest version
ActiveObjects, 0.8.2
Jasorb 1.3
Jython 2.5.1 (for some scripting)
Tomcat 6.0.20
mod_python
Apache
The error:
Exception in thread "Timer-4"
java.lang.NoClassDefFoundError:
com/mysql/jdbc/SQLError
at com.mysql.jdbc.Util.handleNewInstance(Util.java:430)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:352)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:284)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at net.java.ao.DatabaseProvider.getConnectionImpl(DatabaseProvider.java:771)
at net.java.ao.DatabaseProvider.getConnection(DatabaseProvider.java:734)
at net.java.ao.EntityManager.find(EntityManager.java:666)
at net.java.ao.EntityManager.find(EntityManager.java:607)
at net.java.ao.EntityManager.find(EntityManager.java:579)
at com.sq.sa.SAPushAlertsTask.run(SAPushAlertsTask.java:32)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by:
java.lang.ClassNotFoundException:
com.mysql.jdbc.SQLError
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:138
7)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:123
3)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 13 more
Either your MySQL/J connector is somehow broken OR (more likely) you have another somewhere jar in your classpath that has only some of the classes from MySQL's JDBC driver. Either way, "com/mysql/jdbc/SQLError" class can't be found.
Your application runs fine the first few minutes, then when some error occurs the above class can't be loaded which throws the exception as posted. Check your entire classpath (CLASSPATH env. variable + tomcat's lib + webapp's lib) and see whether a) you actually do have SQLError in it and b) MySQL JDBC driver does not occur more than once.
Once the above is fixed, you'll get an actual error that's causing this and should be able to take it from there.