Develop Reference

Apache Livy : Could not find or load main class org.apache.livy.server.LivyServer

I am trying to start Apache Livy 0.8.0 server on my windows 10 machine for spark 3.1.2 and hadoop 3.2.1. I am taking help from here.. I have successfully built apache livy using maven (I have attached a of it) But I am not able to run the livy server. When I run it I get the following error -
> starting C:/AmazonJDK/jdk1.8.0_332/bin/java -cp /d/ApacheLivy/incubator-livy-master/incubator-livy-master/server/target/jars/*:/d/ApacheLivy/incubator-livy-master/incubator-livy-master/conf:D:/Program_files/spark/conf:D:/ApacheHadoop/hadoop-3.2.1/etc/hadoop: org.apache.livy.server.LivyServer, logging to D:/ApacheLivy/incubator-livy-master/incubator-livy-master/logs/livy--server.out
ps: unknown option -- o
Try `ps --help' for more information.
failed to launch C:/AmazonJDK/jdk1.8.0_332/bin/java -cp /d/ApacheLivy/incubator-livy-master/incubator-livy-master/server/target/jars/*:/d/ApacheLivy/incubator-livy-master/incubator-livy-master/conf:D:/Program_files/spark/conf:D:/ApacheHadoop/hadoop-3.2.1/etc/hadoop: org.apache.livy.server.LivyServer:
Error: Could not find or load main class org.apache.livy.server.LivyServer
full log in D:/ApacheLivy/incubator-livy-master/incubator-livy-master/logs/livy--server.out
I am using Git bash. If you need more information I will provide

The error got resolved when I used Windows Subsystem for Linux (WSL).

Jmeter failed to start with the Prometheus JMX agent when run not from $JMETER_HOME/bin

Trying configure Prometheus JMX agent for Jmeter but faced following issue - when I start Jmeter outside of $JMETER_HOME/bin folder - it fails with an error:
java.lang.Throwable: Could not access null/lib
at org.apache.jmeter.NewDriver.<clinit>(NewDriver.java:105)
java.lang.Throwable: Could not access null/lib/ext
at org.apache.jmeter.NewDriver.<clinit>(NewDriver.java:105)
java.lang.Throwable: Could not access null/lib/junit
at org.apache.jmeter.NewDriver.<clinit>(NewDriver.java:105)
java.lang.ClassNotFoundException: org.apache.jmeter.JMeter
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.jmeter.NewDriver.main(NewDriver.java:250)
JMeter home directory was detected as: null
Launch command:
java -Dcom.sun.management.jmxremote.port=12021 -Dcom.sun.management.jmxremote.rmi.port=12021 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dgroovy.use.classvalue=true -javaagent:/apps/injector/apache-jmeter/bin/jmx_prometheus_javaagent-0.13.0.jar=8778:/apps/injector/apache-jmeter/bin/prometheus_config.yaml -jar /apps/injector/apache-jmeter/bin/ApacheJMeter.jar -n -t /apps/injector/apache-jmeter/extras/Test.jmx
Same command works fine in case I run it from $JMETER_HOME/bin folder.
It's not seems to be a Jmeter issue itself, as I can run same command from any place, and it will not cause an error in case I remove -javaagent option.
Can somebody help me configure Prometheus JMX agent for properly work with Jmeter.

Add to java options of JMeter:
-Djmeter.home=$JMETER_HOME
As per your bugzilla ticket:
https://bz.apache.org/bugzilla/show_bug.cgi?id=64680
Credit to Felix S. member of JMeter team

Problem encountered while remote debugging java application as a service

I followed this SO post:
What are Java command line options to set to allow JVM to be remotely debugged?
and was able to remotely debug my Java app that runs as a windows service through prunsrv / procrun.
When I run the service with option suspend=n, I am able to connect remote debugger through eclipse:
myservice.exe //US//myservice --JvmOptions "-agentlib:jdwp=transport=dt_socket,address=127.0.0.1:8111,server=y,suspend=n"
but when I give option suspend=y, I get the following error:
Commons Daemon procrun failed with exit value: 5 (Failed to start
service)
any idea why suspend=y is resulting in this error. (Unfortunately detailed logs for prunsrv are not generated)

Can't connect to Spark running in Kubernetes from Java

I have installed Kuberenetes (minikube for Windows 10) and added Spark there using helm:
.\helm.exe install --name spark-test stable/spark
Then I exposed Spark master port 7077 using
.\kubectl.exe expose deployment spark-test-master --port=7070 --name=spark-master-ext --type=NodePort
For example, my UI runs on http://<MINIKUBE_IP>:31905/ and spark master is exposed to <MINIKUBE_IP>:32473. In order to check, I do:
.\minikube-windows-amd64.exe service spark-master-ext
But when I do in Java:
SparkConf conf = new SparkConf().setMaster("spark://192.168.1.168:32473").setAppName("Data Extractor");
I've got:
18/03/19 13:57:29 WARN AppClient$ClientEndpoint: Could not connect to 192.168.1.168:32473: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster#192.168.1.168:32473]
18/03/19 13:57:29 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster#192.168.1.168:32473] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://sparkMaster#192.168.1.168:32473]] Caused by: [Connection refused: no further information: /192.168.1.168:32473]
18/03/19 13:57:29 WARN AppClient$ClientEndpoint: Failed to connect to master 192.168.1.168:32473
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://sparkMaster#192.168.1.168:32473/), Path(/user/Master)]
Any ideas, how to run Java Spark jobs on Spark running in Minikube?

It looks like Helm chart for Spark is really outdated (1.5.1), so I have installed 2.3.0 locally and it runs without any issues. Case closed, sorry :)

Submitting Spark application on YARN from Eclipse IDE

I am facing an issue when I try to submit my Spark application on Yarn from eclipse. I try to submit a simple SVM program, but i gives below error. I Have macbook, and I will be so thankful if somebody give me detailed answer
16/09/17 10:04:19 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Library directory '.../MyProject/assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:368)
at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:500)
at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:834)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:167)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:149)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
at SVM.main(SVM.java:21)

Go to
Run Configurations --> Environment
in Eclipse and add the environment variable SPARK_HOME.