Error when running my jar with arguments - java

I have a spark application packaged with maven. At run-time, I have to give 3 arguments (paths of 3 files to create RDDs). So I used spark-submit command as the officiel website of spark indicates:
./bin/spark-submit \
--class <main-class> \
--master <master-url> \
--deploy-mode <deploy-mode> \
--conf <key>=<value> \
.. # other options
<application-jar> \
[application-arguments]
My submit-command looks like:
\bin\spark-submit --class myapp.Main --master local[*] file:///C:\Users\pc\Desktop\eclipse\myapp\target\myapp-0.0.1-SNAPSHOT.jar ["C:\Users\pc\Desktop\pathToFile1.csv", "C:\Users\pc\Desktop\pathToFile2.csv", "C:\Users\pc\Desktop\pathToFile3.csv"]
I moddified my Main class as follows to get paths at runtime:
String pathToFile1=args[0];
String pathToFile2=args[1];
String pathToFile3=args[2];
But I get an error message that says that the specified path does not exist. What am I doing wrong here?


#bradimus you were right i dont have to use [], i have to write it as :
\bin\spark-submit --class myapp.Main --master local[*] file:///C:\Users\pc\Desktop\eclipse\myapp\target\myapp-0.0.1-SNAPSHOT.jar C:\Users\pc\Desktop\pathToFile1.csv C:\Users\pc\Desktop\pathToFile2.csv C:\Users\pc\Desktop\pathToFile3.csv

Related

Spark classpath conflict

I am using spark version 2.1.0, I use
./bin/spark-submit --master spark://some_master_host_name:7077 \
--class com.foo.bar.SomeApp \
--deploy-mode cluster \
--conf spark.jars=libraries/avro-1.8.2.jar \
--jars libraries/avro-1.8.2.jar \
--conf spark.executor.extraClassPath=avro-1.8.2.jar \
--conf spark.driver.extraClassPath=avro-1.8.2.jar \
--conf spark.files.fetchTimeout=600s \
--conf spark.driver.extraJavaOptions="-XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/some_file_name.hprof" \
--conf spark.executor.extraJavaOptions="-XX:+PrintFlagsFinal -XX:+PrintReferenceGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy -XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/some_file_name.hprof" \
/some/where/on/disk/FOO-assembly.jar
to submit spark application to the cluster. On application side I have following debug logging enabled to inspect classpath
private static void printSystemClasspath() {
System.err.println("-------System Classpath--------------");
String[] entries = System.getProperty("java.class.path").split("\\:");
for (String entry : entries) {
System.err.println(entry + " - " + new File(entry).exists());
}
System.err.println("-------------------------------------");
}
private static void printCurrentThreadClassloaderClasspath() {
System.err.println("-------Current Thread Classpath--------------");
ClassLoader cl = Thread.currentThread().getContextClassLoader();
URL[] urls = ((URLClassLoader) cl).getURLs();
for (URL url : urls) {
System.err.println(url.getFile());
}
System.err.println("-------------------------------------");
}
Problem : I am trying to override spark's supplied avro version and supply 1.8.2 at runtime.
I have proper version of avro classes packaged inside FOO-assembly.jar but when I use spark.driver.userClassPathFirst or spark.executor.userClassPathFirst I run into some other conflicts. I want to be able to just override avro library at runtime to upgraded version.
I receive first entry in System classpath as avro-1.8.2.jar but the file isn't present on the file system (code evaluates to false)
How can I override the classpath in this situation ?

Running Java Spark program on AWS EMR

I'm having problem running Java written spark application on AWS EMR.
Locally, everything runs fine. When I submit a job to EMR, I always get "Completed" withing 20 seconds even though job should take minutes. There is no output being produced, no log messages are being printed.
I'm still confused as weather it should be ran as Spark application or CUSTOM_JAR type.
Look of my main method:
public static void main(String[] args) throws Exception {
SparkSession spark = SparkSession
.builder()
.appName("RandomName")
.getOrCreate();
//process stuff
String from_path = args[0];
String to_path = args[1];
Dataset<String> dataInput = spark.read().json(from_path).toJSON();
JavaRDD<ResultingClass> map = dataInput.toJavaRDD().map(row -> convertData(row)); //provided function didn't include here
Dataset<Row> dataFrame = spark.createDataFrame(map, ResultingClass.class);
dataFrame
.repartition(1)
.write()
.mode(SaveMode.Append)
.partitionBy("year", "month", "day", "hour")
.parquet(to_path);
spark.stop();
}
I've tried these:
1)
aws emr add-steps --cluster-id j-XXXXXXXXX --steps \
Type=Spark,Name=MyApp,Args=[--deploy-mode,cluster,--master,yarn, \
--conf,spark.yarn.submit.waitAppCompletion=false, \
--class,com.my.class.with.main.Foo,s3://mybucket/script.jar, \
s3://partitioned-input-data/*/*/*/*/*.txt, \
s3://output-bucket/table-name], \
ActionOnFailure=CONTINUE --region us-west-2 --profile default
Completes in 15 sec without error, output result or logs I've added.
2)
aws emr add-steps --cluster-id j-XXXXXXXXX --steps \
Type=CUSTOM_JAR, \
Jar=s3://mybucket/script.jar, \
MainClass=com.my.class.with.main.Foo, \
Name=MyApp, \
Args=[--deploy-mode,cluster, \
--conf,spark.yarn.submit.waitAppCompletion=true, \
s3://partitioned-input-data/*/*/*/*/*.txt, \
s3://output-bucket/table-name], \
ActionOnFailure=CONTINUE \
--region us-west-2 --profile default
Reads parameters wrongly, sees --deploy-mode as first parameter and cluster as second instead of buckets
3)
aws emr add-steps --cluster-id j-XXXXXXXXX --steps \
Type=CUSTOM_JAR, \
Jar=s3://mybucket/script.jar, \
MainClass=com.my.class.with.main.Foo, \
Name=MyApp, \
Args=[s3://partitioned-input-data/*/*/*/*/*.txt, \
s3://output-bucket/table-name], \
ActionOnFailure=CONTINUE \
--region us-west-2 --profile default
I get this: Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
When I include all dependencies (which I do not need to locally)
I get: Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
I do not want to hardcode the "yarn" into the app.
I find AWS documentation very confusing as to what is the proper way to run this.
Update:
Running command directly on the server does work. So the problem must be in the way I'm defining a cli command.
spark-submit --class com.my.class.with.main.Foo \
s3://mybucket/script.jar \
"s3://partitioned-input-data/*/*/*/*/*.txt" \
"s3://output-bucket/table-name"

The 1) was working.
The step overview on the aws console said that the task was finished within 15 seconds, but in reality it was still running on the cluster. It took him an hour to do the work and I can see the result.
I do not know why the step is misreporting the result. I'm using emr-5.9.0 with Ganglia 3.7.2, Spark 2.2.0, Zeppelin 0.7.2.

Spark Naive Bayes ML OutofMemory Error - Prediction

I'm trying to build a Machine Learning program with Spark 1.6
I have started the Spark shell with the following settings:
spark-shell --driver-class-path sqljdbc_6.0/enu/sqljdbc42.jar --driver-memory 25G --executor-memory 30G --num-executors 180 --conf spark.driver.maxResultSize=0 --conf spark.ui.port=4042 --conf spark.default.parallelism=100 --conf spark.sql.shuffle.partitions=1000`
My code works until I try to predict/use the model.
After executing this code:
scala> val predictionAndLabel = test.map(p => (model.predict(p.features), p.label))
I get this error message:
/usr/bin/spark-shell: line 41: 33686 Killed
"$FWDIR"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$#"
I hope somebody can help me because I don't have any idea how I could make this code run smoothly!
Here is the Link to the complete full track of the error.
https://app.box.com/s/w247yaoaiuogqot2zr76qjbwr9rzeb7b

object com.github.nscala_time.time.Imports not found error?

I used package com.github.nscala_time.time.Imports in code and I m running the code using spark.
Here is my stream.sh file:
#!/bin/bash
JARS_HOME=$HOME/spark-job/lib
JARS=$JARS_HOME/job-server-api_2.10-0.6.0.jar,$JARS_HOME/httpmime-4.4.1.jar,$JARS_HOME/noggit-0.6.jar,$JARS_HOME/nscala-time_2.10-2.0.0.jar
export SPARK_IP=`ifconfig | grep eth0 -1 | grep -i inet | awk '{ print $2 }' | cut -d':' -f2`
APP_JAR=$JARS_HOME/spark-jobs-tests.jar
export SPARK_LOCAL_IP=$SPARK_IP
dse spark-submit --conf "spark.cassandra.input.consistency.level=LOCAL_QUORUM" \
--total-executor-cores 2 \
--jars=$JARS \
--class "my file classpath" $APP_JAR "$1" --files $1
I have set $JARS_HOME/nscala-time_2.10-2.0.0.jar in .sh then still getting following error:
Exception in thread "main" scala.reflect.internal.MissingRequirementError: object com.github.nscala_time.time.Imports not found.
at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
at scala.reflect.internal.Mirrors$RootsBase.ensureModuleSymbol(Mirrors.scala:126)
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:161)
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:21)
How to resolve this??

How to run a Java application as Windows service using WinRun4J

I'm trying to run a Java application as a Windows service with WinRun4J.
I copied WinRun4J64c.exe in my application directory and placed the following service.ini file beside:
service.class=org.boris.winrun4j.MainService
service.id=MyAPP
service.name=MyAPP
service.description=some description
classpath.1=./lib/*
classpath.2=WinRun4J.jar
[MainService]
class=play.core.server.NettyServer
But if I start the service with: WinRun4J64c.exe --WinRun4J:RegisterService I get:
Service control dispatcher error: 1063
What is wrong?

I didn't get it working, so my workaround is to use Apache Commons Deamon. I executed the included prunsrv.exe with the following parameters:
prunsrv.exe install "MeineAnwendung" \
--Install="C:/pfad/zu/prunsrv.exe" \
--JvmOptions=-Dpidfile.path=NUL
--Jvm=auto \
--Startup=auto \
--StartMode=jvm \
--Classpath="c:/irgendwo/anwendung/lib/*;" \
--StartClass=play.core.server.NettyServer

Categories

Resources