I'm having problem running Java written spark application on AWS EMR.
Locally, everything runs fine. When I submit a job to EMR, I always get "Completed" withing 20 seconds even though job should take minutes. There is no output being produced, no log messages are being printed.
I'm still confused as weather it should be ran as Spark application or CUSTOM_JAR type.
Look of my main method:
public static void main(String[] args) throws Exception {
SparkSession spark = SparkSession
.builder()
.appName("RandomName")
.getOrCreate();
//process stuff
String from_path = args[0];
String to_path = args[1];
Dataset<String> dataInput = spark.read().json(from_path).toJSON();
JavaRDD<ResultingClass> map = dataInput.toJavaRDD().map(row -> convertData(row)); //provided function didn't include here
Dataset<Row> dataFrame = spark.createDataFrame(map, ResultingClass.class);
dataFrame
.repartition(1)
.write()
.mode(SaveMode.Append)
.partitionBy("year", "month", "day", "hour")
.parquet(to_path);
spark.stop();
}
I've tried these:
1)
aws emr add-steps --cluster-id j-XXXXXXXXX --steps \
Type=Spark,Name=MyApp,Args=[--deploy-mode,cluster,--master,yarn, \
--conf,spark.yarn.submit.waitAppCompletion=false, \
--class,com.my.class.with.main.Foo,s3://mybucket/script.jar, \
s3://partitioned-input-data/*/*/*/*/*.txt, \
s3://output-bucket/table-name], \
ActionOnFailure=CONTINUE --region us-west-2 --profile default
Completes in 15 sec without error, output result or logs I've added.
2)
aws emr add-steps --cluster-id j-XXXXXXXXX --steps \
Type=CUSTOM_JAR, \
Jar=s3://mybucket/script.jar, \
MainClass=com.my.class.with.main.Foo, \
Name=MyApp, \
Args=[--deploy-mode,cluster, \
--conf,spark.yarn.submit.waitAppCompletion=true, \
s3://partitioned-input-data/*/*/*/*/*.txt, \
s3://output-bucket/table-name], \
ActionOnFailure=CONTINUE \
--region us-west-2 --profile default
Reads parameters wrongly, sees --deploy-mode as first parameter and cluster as second instead of buckets
3)
aws emr add-steps --cluster-id j-XXXXXXXXX --steps \
Type=CUSTOM_JAR, \
Jar=s3://mybucket/script.jar, \
MainClass=com.my.class.with.main.Foo, \
Name=MyApp, \
Args=[s3://partitioned-input-data/*/*/*/*/*.txt, \
s3://output-bucket/table-name], \
ActionOnFailure=CONTINUE \
--region us-west-2 --profile default
I get this: Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
When I include all dependencies (which I do not need to locally)
I get: Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
I do not want to hardcode the "yarn" into the app.
I find AWS documentation very confusing as to what is the proper way to run this.
Update:
Running command directly on the server does work. So the problem must be in the way I'm defining a cli command.
spark-submit --class com.my.class.with.main.Foo \
s3://mybucket/script.jar \
"s3://partitioned-input-data/*/*/*/*/*.txt" \
"s3://output-bucket/table-name"
The 1) was working.
The step overview on the aws console said that the task was finished within 15 seconds, but in reality it was still running on the cluster. It took him an hour to do the work and I can see the result.
I do not know why the step is misreporting the result. I'm using emr-5.9.0 with Ganglia 3.7.2, Spark 2.2.0, Zeppelin 0.7.2.
Related
I am getting the below error even though I generated the partner.jar correctly.
I have generated the partner.jar in my windows machine using the below jars.
antlr-runtime-3.5.3.jar,
force-wsc-56.1.0.jar,
js-1.7R2.jar,
partner.wsdl,
ST4-4.3.4.jar,
tools.jar,
java -classpath tools.jar;force-wsc-56.1.0.jar;ST4-4.3.4.jar;js-1.7R2.jar;antlr-runtime-3.5.3.jar com.sforce.ws.tools.wsdlc partner.wsdl partner.jar
I have set the class path correctly before generating the partner.jar.
And I copied this partner.jar to AWS EC2 machine and try executing the below code. But its still failing with the connection error.
df = spark.read.format("com.springml.spark.salesforce") \ .option("username", "dinesh123#force.com") \
.option("password", "passwordtoken) \
.option("login", "https://dev-yh.develop.my.salesforce.com/")\ .option("soql", soql) \
.option("inferSchema", True) \
.load()
Error:
py4j.protocol.Py4JJavaError: An error occurred while calling o31.load.
: java.lang.NoClassDefFoundError: com/sforce/ws/ConnectionException
at com.springml.salesforce.wave.api.APIFactory.forceAPI(APIFactory.java:49)
at com.springml.spark.salesforce.DefaultSource.createRelation(DefaultSource.scala:102)
at com.springml.spark.salesforce.DefaultSource.createRelation(DefaultSource.scala:50)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245)
at scala.Option.getOrElse(Option.scala:189)
I have this code from Google Cloud Platform Dataflow Templates.
I wish to add more functionalities to it, namely, I wish to add support for JavaScript UDF. When I try to compile the file, using this:
mvn compile exec:java \
-Dexec.mainClass=com.google.cloud.teleport.templates.${PIPELINE_NAME} \
-Dexec.cleanupDaemonThreads=false \
-Dexec.args=" \
--project=${PROJECT_ID} \
--stagingLocation=gs://${PROJECT_ID}/dataflow/${PIPELINE_FOLDER}/staging \
--tempLocation=gs://${PROJECT_ID}/dataflow/${PIPELINE_FOLDER}/temp \
--runner=DataflowRunner \
--windowDuration=2m \
--numShards=1 \
--topic=projects/${PROJECT_ID}/topics/windowed-files \
--outputDirectory=gs://${PROJECT_ID}/temp/ \
--outputFilenamePrefix=windowed-file \
--outputFilenameSuffix=.txt"
When compiling the file, I get the following error:
An exception occured while executing the Java class. Class interface com.google.cloud.teleport.templates.PubsubToText$Options missing a property named 'topic'. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project google-cloud-teleport-java: An exception occured while executing the Java class. Class interface com.google.cloud.teleport.templates.PubsubToText$Options missing a property named 'topic'.
Even though, I've passed the --topic flag with appropriate values plugged in.
The example at the top is wrong. You have to pass --inputTopic instead of --topic. You can see this in the code where the ValueProvider is defined:
#Description("The Cloud Pub/Sub topic to read from.")
#Required
ValueProvider<String> getInputTopic();
void setInputTopic(ValueProvider<String> value);
You can also run the template from the Console UI and the job details will show that the option is indeed inputTopic:
The invocation example in the javadoc should now reflect the correct input parameter (--inputTopic) here.
I have a spark application packaged with maven. At run-time, I have to give 3 arguments (paths of 3 files to create RDDs). So I used spark-submit command as the officiel website of spark indicates:
./bin/spark-submit \
--class <main-class> \
--master <master-url> \
--deploy-mode <deploy-mode> \
--conf <key>=<value> \
.. # other options
<application-jar> \
[application-arguments]
My submit-command looks like:
\bin\spark-submit --class myapp.Main --master local[*] file:///C:\Users\pc\Desktop\eclipse\myapp\target\myapp-0.0.1-SNAPSHOT.jar ["C:\Users\pc\Desktop\pathToFile1.csv", "C:\Users\pc\Desktop\pathToFile2.csv", "C:\Users\pc\Desktop\pathToFile3.csv"]
I moddified my Main class as follows to get paths at runtime:
String pathToFile1=args[0];
String pathToFile2=args[1];
String pathToFile3=args[2];
But I get an error message that says that the specified path does not exist. What am I doing wrong here?
#bradimus you were right i dont have to use [], i have to write it as :
\bin\spark-submit --class myapp.Main --master local[*] file:///C:\Users\pc\Desktop\eclipse\myapp\target\myapp-0.0.1-SNAPSHOT.jar C:\Users\pc\Desktop\pathToFile1.csv C:\Users\pc\Desktop\pathToFile2.csv C:\Users\pc\Desktop\pathToFile3.csv
I'm trying to build a Machine Learning program with Spark 1.6
I have started the Spark shell with the following settings:
spark-shell --driver-class-path sqljdbc_6.0/enu/sqljdbc42.jar --driver-memory 25G --executor-memory 30G --num-executors 180 --conf spark.driver.maxResultSize=0 --conf spark.ui.port=4042 --conf spark.default.parallelism=100 --conf spark.sql.shuffle.partitions=1000`
My code works until I try to predict/use the model.
After executing this code:
scala> val predictionAndLabel = test.map(p => (model.predict(p.features), p.label))
I get this error message:
/usr/bin/spark-shell: line 41: 33686 Killed
"$FWDIR"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$#"
I hope somebody can help me because I don't have any idea how I could make this code run smoothly!
Here is the Link to the complete full track of the error.
https://app.box.com/s/w247yaoaiuogqot2zr76qjbwr9rzeb7b
I'm trying to run a Java application as a Windows service with WinRun4J.
I copied WinRun4J64c.exe in my application directory and placed the following service.ini file beside:
service.class=org.boris.winrun4j.MainService
service.id=MyAPP
service.name=MyAPP
service.description=some description
classpath.1=./lib/*
classpath.2=WinRun4J.jar
[MainService]
class=play.core.server.NettyServer
But if I start the service with: WinRun4J64c.exe --WinRun4J:RegisterService I get:
Service control dispatcher error: 1063
What is wrong?
I didn't get it working, so my workaround is to use Apache Commons Deamon. I executed the included prunsrv.exe with the following parameters:
prunsrv.exe install "MeineAnwendung" \
--Install="C:/pfad/zu/prunsrv.exe" \
--JvmOptions=-Dpidfile.path=NUL
--Jvm=auto \
--Startup=auto \
--StartMode=jvm \
--Classpath="c:/irgendwo/anwendung/lib/*;" \
--StartClass=play.core.server.NettyServer