Develop Reference

HADOOP_HOME and hadoop.home.dir are unset Error when trying to use sc.setCheckpointDir('checkpoint') Pyspark

I am trying to Use the ALS API on pyspark for a recommendation model but I get a java.lang.StackOverflowError Which after looking it up, I saw instructions on how to fix it by using check points.
When I try to set checkpoints directory, I get an hadoop not found error.
sc.setCheckpointDir('checkpoint')
This is the error response
Py4JJavaError: An error occurred while calling o83.setCheckpointDir.
: java.lang.RuntimeException: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:736) at org.apache.hadoop.util.Shell.getSetPermissionCommand(Shell.java:271) at org.apache.hadoop.util.Shell.getSetPermissionCommand(Shell.java:287) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:865) at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:547) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:587) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:559) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:586) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:559) at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:705) at org.apache.spark.SparkContext.$anonfun$setCheckpointDir$2(SparkContext.scala:2483) at scala.Option.map(Option.scala:230) at org.apache.spark.SparkContext.setCheckpointDir(SparkContext.scala:2480) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Unknown Source) Caused by: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:548) at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:569) at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:592) at org.apache.hadoop.util.Shell.(Shell.java:689) at org.apache.hadoop.util.StringUtils.(StringUtils.java:78) at org.apache.hadoop.conf.Configuration.getTimeDurationHelper(Configuration.java:1814) at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1791) at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183) at org.apache.hadoop.util.ShutdownHookManager$HookEntry.(ShutdownHookManager.java:207) at org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:302) at org.apache.spark.util.SparkShutdownHookManager.install(ShutdownHookManager.scala:181) at org.apache.spark.util.ShutdownHookManager$.shutdownHooks$lzycompute(ShutdownHookManager.scala:50) at org.apache.spark.util.ShutdownHookManager$.shutdownHooks(ShutdownHookManager.scala:48) at org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:153) at org.apache.spark.util.ShutdownHookManager$.(ShutdownHookManager.scala:58) at org.apache.spark.util.ShutdownHookManager$.(ShutdownHookManager.scala) at org.apache.spark.util.Utils$.createTempDir(Utils.scala:326) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:343) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:468) at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:439) at org.apache.hadoop.util.Shell.(Shell.java:516) ... 21 more
I will appreciate a response on how to fix this. I am working on a windows environment

Jenkins is not getting start on windows 10

java.lang.IllegalStateException: Expected 1 instance of hudson.PluginManager$PluginUpdateMonitor but got 0
at hudson.ExtensionList.lookupSingleton(ExtensionList.java:451)
at hudson.PluginManager$PluginUpdateMonitor.getInstance(PluginManager.java:2188)
at hudson.maven.PluginImpl.init(PluginImpl.java:58)
Caused: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
Caused: java.lang.Error
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1133)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:50)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:1166)
at jenkins.model.Jenkins.<init>(Jenkins.java:966)
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:250)
Recently install Jenkins by downloading its war file(jenkins.war), and installation I downloaded some maven plugins and then I tried to restart it again but did not find any way to restart it.
so I just open the cmd and hit the command java -jar jenkins.war. After that Jenkins stops(cmd showing the message: jenkins.model.Jenkins#cleanUp: Jenkins stopped).
Now how can I start it again, when I am hitting the localhost:8080/jenkins, it is showing the above error.

How to execute a R script file from java

I would like to call an R script from Java. I have done google searches on the topic, but almost all of the results I have seen would require me to add a dependency to some third party library. Can anyone show me a good way to accomplish the same thing without adding any dependencies to my code?
I'm setting up a new application which runs R script file from java using the following line
Runtime.getRuntime().exec("C:\\Users\\Thiru\\Desktop helloWorld.R");
but when I run the application I am getting following exception:
SEVERE: Servlet.service() for servlet [RenjinServlet] in context with
path
[/REngineCheck] threw exception
java.io.IOException: Cannot run program "C:\Users\Thiru\Desktop":
CreateProcess error=5, Access is denied
at java.lang.ProcessBuilder.start(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at org.r.Check.RenjinServlet.doGet(RenjinServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
(ApplicationFilterChain.java:291)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter (WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke
(StandardWrapperValve.java:217)
at org.apache.catalina.core.StandardContextValve.invoke
(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke
(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke
(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke
(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke
(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke
(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service
(CoyoteAdapter.java:518)
at org.apache.coyote.http11.AbstractHttp11Processor.process
(AbstractHttp11Processor.java:1091)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process
(AbstractProtocol.java:673)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
(NioEndpoint.java:1500)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run
(NioEndpoint.java:1456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run
(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: CreateProcess error=5, Access is denied
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 28 more

You should set up your path in a bracket like
("your_path\Rscript.exe your_path\filename.R")
If you set the environment, you can use Rscript
Example:
Runtime.getRuntime().exec("C:\\Program Files\\R\\R-3.6.1\\bin\\x64\\Rscript.exe C:\\Users\\Thiru\\Desktop\\helloWorld.R");

SSL java.security.NoSuchAlgorithmException

Java says that in this line, the file ssc.store was not found:
server = ((SSLServerSocketFactory) SSLServerSocketFactory.getDefault()).createServerSocket(port);
(server is a ServerSocket)
I try to use a libary containing this code, but I am not able to open this ServerSocket.
Full Error
Error opening ServerSocket
java.net.SocketException: java.security.NoSuchAlgorithmException: Error
constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at javax.net.ssl.DefaultSSLServerSocketFactory.throwException(Unknown Source)
at javax.net.ssl.DefaultSSLServerSocketFactory.createServerSocket(Unknown Source)
at com.blogspot.debukkitsblog.Net.Server.start(Server.java:327)
at com.blogspot.debukkitsblog.Net.Server.<init>(Server.java:86)
at com.blogspot.debukkitsblog.Net.Server.<init>(Server.java:51)
at de.yee.GameServer.<init>(GameServer.java:16)
at de.yee.Main.main(Main.java:9)
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Unknown Source)
at sun.security.jca.GetInstance.getInstance(Unknown Source)
at sun.security.jca.GetInstance.getInstance(Unknown Source)
at javax.net.ssl.SSLContext.getInstance(Unknown Source)
at javax.net.ssl.SSLContext.getDefault(Unknown Source)
at javax.net.ssl.SSLServerSocketFactory.getDefault(Unknown Source)
... 5 more
Caused by: java.security.PrivilegedActionException: java.io.FileNotFoundException: ssc.store (Das System kann die angegebene Datei nicht finden)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getKeyManagers(Unknown Source)
at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.<clinit>(Unknown Source)
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(Unknown Source)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
... 11 more
Caused by: java.io.FileNotFoundException: ssc.store (Das System kann die angegebene Datei nicht finden)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at sun.security.ssl.SSLContextImpl$DefaultManagersHolder$2.run(Unknown Source)
at sun.security.ssl.SSLContextImpl$DefaultManagersHolder$2.run(Unknown Source)
... 19 more
What is the problem?

You need ssc.store file in the working directory where you run your code. The store is here: https://github.com/DeBukkIt/SimpleServerClient/blob/master/ssc.store (or in the specific branch or tag you checked out).
For example, if you're running main method inside IntelliJ, the store file should be in the project root directory. Or, if you pack your application into a jar and run java -jar yourapp.jar, the ssc.store should be in the same directory as the jar file.

java.io.IOException: Could not get input splits when run the MapReduce program

I am running a MapReduce program and encountered with the below error.
14/04/22 07:44:02 INFO mapred.JobClient: Cleaning up the staging area cfs://XX.XXX.XXX.XXX/tmp/hadoop-cassandra/mapred/staging/psadmin/.staging/job_201404180932_0063
14/04/22 07:44:02 ERROR security.UserGroupInformation: PriviledgedActionException as:psadmin cause:java.io.IOException: Could not get input splits
Exception in thread "main" java.io.IOException: Could not get input splits
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:193)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at MultiOutMR.run(MultiOutMR.java:95)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at MultiOutMR.main(MultiOutMR.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:189)
... 19 more
Caused by: java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSubSplits(AbstractColumnFamilyInputFormat.java:304)
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.access$200(AbstractColumnFamilyInputFormat.java:60)
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat$SplitCallable.call(AbstractColumnFamilyInputFormat.java:226)
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat$SplitCallable.call(AbstractColumnFamilyInputFormat.java:211)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.cassandra.thrift.Cassandra$Client.recv_describe_splits_ex(Cassandra.java:1359)
at org.apache.cassandra.thrift.Cassandra$Client.describe_splits_ex(Cassandra.java:1343)
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSubSplits(AbstractColumnFamilyInputFormat.java:281)
... 7 more
Note:-
PreRequisites used:-
Datastax Enterprise(DSE 3.2.5) with Cassandra 1.2.15.1 and Hadoop 1.0.4.9
We have configured a data center with 4 nodes. The nodetool status shows as follows:
XXXXXX#XXXXXXXXX:~$ nodetool status
Datacenter: XXXXXX
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns Host ID Token Rack
UN XX.XXX.XXX.XXX 14.65 MB 25.0% XX.XXX.XXX.XXX vm01
UN XX.XXX.XXX.XXX 34.25 MB 25.0% XX.XXX.XXX.XXX vm01
UN XX.XXX.XXX.XXX 57.45 MB 25.0% XX.XXX.XXX.XXX vm01
UN XX.XXX.XXX.XXX 57.08 MB 25.0% XX.XXX.XXX.XXX vm01
Could anyone please provide help in resolving this issue? Thanks in advance.

You need provide more information about how you set up the hadoop job. It's more of configuration issue. TTransportException is more a server internal issue.