I'm encountering the following Exception when running a MapReduce job taking a file stored on HDFS as input:
15/03/27 17:18:12 INFO mapreduce.Job: Task Id : attempt_1427398929405_0005_m_000005_2, Status : FAILED
Error: java.lang.NullPointerException
at org.apache.hadoop.mapred.Task.getFsStatistics(Task.java:347)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:486)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:735)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
I don't have a good sense of what this means myself, other than this looks possibly related to file permissions. I've also have found the following other StackOverflow post with the same Exception/trace: NullPointerException with MR2 in windows.
To summarize the post: this Exception was caused due to the hadoop being run as a different user than that of the MR job.
I've tried the following:
chmod-ing all files and directories in HDFS to 777 (just to experiment)
running the hadoop job with sudo
but neither approach has yielded any results.
I'm running all Hadoop processes on localhost ("pseudo-distributed mode"). I started hadoop using start-yarn.sh and start-dfs.sh with my normal local user. I'm running the hadoop job with the same user. I've also set dfs.datanode.data.dir and dfs.namenode.name.dir to paths on my local machine to which I have permission to read/write with my local user. I've set dfs.permissions.enabled to false.
Am I misinterpreting this Exception? Is there anything else I should try? Thank you.
In the end, it was my own FileSplit subclass causing the problem. I was not correctly (de)serializing the the FileSplit's Path, so when send across the wire this field became null. Hadoop calls getFsStatistics on the null Path, causing the NullPointerException.
Related
I am trying to run pyspark on jupyter(via anaconda) in windows.Facing the below mentioned error while trying to create a SparkSession.
Exception: Java gateway process exited before sending its port number
Error snapshot 1
Error snapshot 2
I even tried adding JAVA_HOME,SPARK_HOME and HADOOP_HOME path into environment variable:
JAVA_HOME: C:\Java\jdk-11.0.16.1
SPARK_HOME:C:\Spark\spark-3.1.3-bin-hadoop3.2
HADOOP_HOME:C:\Spark\spark-3.1.3-bin-hadoop3.2
Even after this I am facing the same issue.
PS: My pyspark version is 3.3.1 and python version is 3.8.6.
As per spark documentation, the string for setting master should be "local[*]" or "local[N]" for only using N cores. If you leave out the master setting, it defaults to "local[*]".
After several attempts, I finally figured out the issue. It was because the windows firewall had blocked java that caused this error. Once I gave the access permission the error was rectified!
I need to use MATLAB library compiled to JAR file in my own Java application. That library uses parpool and has some parfor operators. We can use this example as a test.
On Windows it works. On Linux (Ubuntu xenial) I get the error like this but not the same:
Starting parallel pool (parpool) using the 'local_mcruserdata' profile ...
Error using parpool (line 104)
Failed to start a parallel pool. (For information in addition to the causing error, validate the profile 'local_mcruserdata' in the Cluster Profile Manager.)
Error in sample_pct (line 11)
Caused by:
Error using parallel.internal.pool.InteractiveClient>iThrowWithCause (line 666)
Failed to initialize the interactive session.
Error using parallel.internal.pool.InteractiveClient>iThrowIfBadParallelJobStatus (line 767)
The interactive communicating job failed with no message.
The error is com.mathworks.toolbox.javabuilder.MWException: Failed to start a parallel pool. (For information in addition to the causing error, validate the profile 'local_mcruserdata' in the Cluster Profile Manager.)
My MATLAB is 9.2.0.538062 (R2017a) and my JDK is 1.8.0_171 x86_64 on both systems.
If I comment line 11 (parpool function invocation) the error goes away but parfor operator does not create extra workers.
Is it a known bug and can it be fixed?
After adding setSchedulerMessageHandler(#disp); setenv('MDCE_DEBUG','true') to the example (as it was adviced in the comment) I have got the message:
matlabroot/bin/glnxa64/ctfxlauncher: error while loading shared libraries: libmwmclmcrrt.so.9.2: cannot open shared object file: No such file or directory
find matlabroot -name libmwmclmcrrt.so.9.2 gives matlabroot/runtime/glnxa64/libmwmclmcrrt.so.9.2
Adding matlabroot/runtime/glnxa64 directory to LD_LIBRARY_PATH helped!
Yesterday suddenly our solr server got stopped with following exception
org.apache.solr.common.SolrException: Error opening new searcher at
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1438) at
org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1550) at
org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1319) at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:574)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1147)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
at
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:266)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
Caused by: java.io.IOException: Map failed at
sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:889) at
org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:283) at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.
But at that time we have lot of memory left in that server. Any guess?
Assuming its a linux system you can try ulimit -v unlimited
Check the below links for more information.
https://support.lucidworks.com/hc/en-us/articles/206127457-Caused-by-java-io-IOException-Map-failed
https://stackoverflow.com/a/8893684/689625
You will get answers much faster if you include more information, like the version of your software (in this case solr version), OS Name and version, Java version etc.
The flag -fix didn't work in my later version, instead I used exorcise
java -cp lucene-core-5.5.0.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex PATH_TO_INDEX -exorcise
I hope this could help.
I got this error:
Error creating core [mm_shard1_replica1]: Error opening new searcher
and tried fiddling with ulimit and max_map_count
In the end it turned out to be a bad error message because the problem turned out to be a corrupt index.
Fixed it by shutting down solr and doing this:
cd /var/lib/solr/example/solr-webapp/webapp/WEB-INF/lib
java -cp lucene-core-4.10.4.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /path/to/index -fix
To just check wether the index is broken remove the -fix at the end of above command. After that I started up solr and it recreated its index from the rest of the solrcloud cluster.
Has anyone run into problems running the HelloWorld Twill example? My Application gets accepted but then transitions to the "FAILED" state.
Yarn application HelloWorldRunnable application_1406337868863_0013 completed with status FAILED
The YARN Web UI shows this as the error:
Application application_1406337868863_0013 failed 2 times due to AM Container for appattempt_1406337868863_0013_000002 exited with exitCode: -1000 due to: File file:/twill/HelloWorldRunnable/2ba08d9f-ca23-4363-a7be-426b93c88de2/appMaster.775a1137-6134-46e2-b270-fc466ce7fe91.jar does not exist
.Failing this attempt.. Failing the application.
Does YARN expect to find this jar on HDFS at the location above? It seems like the jar gets copied to my local FS at the location specified above but not to HDFS.
Looks like you don't have the hadoop conf directory (e.g. /etc/hadoop/conf) in the classpath so that the local file system (file:/twill/...) is used instead of HDFS.
Hey guys so I am trying to run the WordCount.java example, provided by cloudera. I ran the command below and am getting the exception that I have put below the command. So do you have any suggestions on how to proceed. I have gone through all the steps provided by cloudera.
Thanks in advance.
hadoop jar ~/Desktop/wordcount.jar org.myorg.WordCount ~/Desktop/input
~/Desktop/output
Error:
ERROR security.UserGroupInformation: PriviledgedActionException
as:root (auth:SIMPLE)
cause:org.apache.hadoop.mapred.InvalidInputException: Input path does
not exist: hdfs://localhost/home/rushabh/Desktop/input
Exception in thread "main"
org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist: hdfs://localhost/home/rushabh/Desktop/input
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:194)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:205)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:977)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:969)
at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1248)
at org.myorg.WordCount.main(WordCount.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Your input and output files should be at hdfs. Atleast input should be at hdfs.
use the following command:
hadoop jar ~/Desktop/wordcount.jar org.myorg.WordCount hdfs:/input
hdfs:/output
To copy a file from your linux to hdfs use the following command:
hadoop dfs -copyFromLocal ~/Desktop/input hdfs:/
and check your file using :
hadoop dfs -ls hdfs:/
Hope this will help.
The error message says that this file does not exist: "hdfs://localhost/home/rushabh/Desktop/input".
Check that the file does exist at the location you've told it to use.
Check the hostname is correct. You are using "localhost" which most likely resolves to a loopback IP address; e.g. 127.0.0.1. That always means "this host" ... in the context of the machine that you are running the code on.
When I tried to run wordcount MapReduce code, I was getting error as:
ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/hduser/wordcount
I was trying to execute the wordcount MapReduce java code with input and output path as /user/hduser/wordcount and /user/hduser/wordcount-output. I just added 'fs.default.name' from core-site.xml before this path and it ran perfectly.
The error clearly states that your input path is local. Please specify the input path to something on HDFS rather than on local machine. My guess
hadoop jar ~/Desktop/wordcount.jar org.myorg.WordCount ~/Desktop/input
~/Desktop/output
needs to be changed to
hadoop jar ~/Desktop/wordcount.jar org.myorg.WordCount <hdfs-input-dir>
<hdfs-output-dir>
NOTE: To run MapReduce job, the input directory should be in HDFS, not local.
Hope this helps.
So I added the input folder to HDFS using the following command
hadoop dfs -put /usr/lib/hadoop/conf input/
Check the ownership of the files in hdfs to ensure that the owner of the job (root) has read privileges on the input files. Cloudera provides an hdfs viewer that you can use to view the filespace; open a web browser to either localhost:50075 or {fqdn}:50075 and click on "Browse the filesystem" to view the Input directory and input files. Check the ownership flags; just like *nix filesystem.