Develop Reference

Override spark's libraries in spark submit

Our application's hadoop cluster has spark 1.5 installed. But due to specific requirements we have developed spark job with version 2.0.2. When I submit the job to yarn, I use the --jars command to override the spark libraries in cluster. But still it is not picking the scala library jar. It throws an error saying
ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:713)
at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:704)
Any ideas about how to override the cluster libraries during spark submit ?
The shell command I use to submit the job is below.
spark-submit \
--jars test.jar,spark-core_2.11-2.0.2.jar,spark-sql_2.11-2.0.2.jar,spark-catalyst_2.11-2.0.2.jar,scala-library-2.11.0.jar \
--class Application \
--master yarn \
--deploy-mode cluster \
--queue xxx \
xxx.jar \
<params>

That's fairly easy - Yarn doesn't care which version of Spark you are running, it will execute the jars provided by the yarn client which is packaged by spark submit. That process packages your application jar along the spark libs.
In order to deploy Spark 2.0 instead of the provided 1.5, you just need to install spark 2.0 on the host from which you start your job, e.g. in your home dir, set the YARN_CONF_DIR env vars to point to your hadoop conf and then use that spark-submit.

Running Spark jobs with Marathon

I've got a Marathon job running for the following:
./bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher
However, following that I'd like to now be able to run individual Spark jobs as separate Marathon jobs with the command:
./bin/spark-submit ....
My question is:
how can I call spark-submit from a Mesos executor of the binaries are not installed on it?
(Note: I'm aware that http://spark.apache.org/docs/latest/running-on-mesos.html#connecting-spark-to-mesos also recommends installing Spark on all the Mesos slaves, but is that the only option?)
Any guidance is much appreciated.

Just run the following command:
/opt/spark/spark-2.1.0-bin-hadoop2.7/bin/spark-submit --class org.apache.spark.examples.SparkPi --master mesos://127.0.0.1:31258 --deploy-mode cluster --supervise --executor-memory 2G --total-executor-cores 1 /opt/spark/spark-2.1.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.0.jar 1000

Ignite running in Docker (is: General Java-Docker issue)

I am trying to run ignite in a Docker container (Mac OS X, Docker 1.9.1) as committed in git:
# Start from a Java image.
FROM java:7
# Ignite version
ENV IGNITE_VERSION 1.5.0-b1
WORKDIR /opt/ignite
ADD http://www.us.apache.org/dist/ignite/1.5.0-b1/apache-ignite-fabric-1.5.0-b1-bin.zip /opt/ignite/ignite.zip
# Ignite home
ENV IGNITE_HOME /opt/ignite/apache-ignite-fabric-1.5.0-b1-bin
RUN unzip ignite.zip
RUN rm ignite.zip
# Copy sh files and set permission
ADD ./run.sh $IGNITE_HOME/
RUN chmod +x $IGNITE_HOME/run.sh
CMD $IGNITE_HOME/run.sh
After building it locally to apache/ignite and running the image with following command, the container 'hangs'
docker run --expose=4700-4800 -it -p 47500-47600:47500-47600 -p 47100-47200:47100-47200 --net=host -e "CONFIG_URI=https://raw.githubusercontent.com/apache/ignite/master/examples/config/example-default.xml" apacheignite/ignite-docker
When connecting to the container (docker exec -ti apache/ignite /bin/bash) and running the command in verbose mode via bash, it hangs on org.apache.ignite.startup.cmdline.CommandLineRandomNumberGenerator:
bash -x /opt/ignite/apache-ignite-fabric-1.5.0-b1-bin/bin/ignite.sh https://raw.githubusercontent.com/apache/ignite/master/examples/config/example-default.xml
Output where it hangs:
+ CP='/opt/ignite/apache-ignite-fabric-1.5.0-b1-bin/libs/*:/opt/ignite/apache-ignite-fabric-1.5.0-b1-bin/libs/ignite-indexing/*:/opt/ignite/apache-ignite-fabric-1.5.0-b1-bin/libs/ignite-spring/*:/opt/ignite/apache-ignite-fabric-1.5.0-b1-bin/libs/licenses/*'
++ /usr/bin/java -cp '/opt/ignite/apache-ignite-fabric-1.5.0-b1-bin/libs/*:/opt/ignite/apache-ignite-fabric-1.5.0-b1-bin/libs/ignite-indexing/*:/opt/ignite/apache-ignite-fabric-1.5.0-b1-bin/libs/ignite-spring/*:/opt/ignite/apache-ignite-fabric-1.5.0-b1-bin/libs/licenses/*' org.apache.ignite.startup.cmdline.CommandLineRandomNumberGenerator
Looking at the code of CommandLineRandomNumberGenerator, I don't see anything special, just one line to generate a UUID. Are there other things that are automatically started somehow that causes locking a threat so that the application cannot exit?

This seems to be a docker issue with java in general, see also:
https://github.com/docker/docker/issues/18180
Several solutions possible:
create a docker machine as follows and run it in here (cfr. https://github.com/docker/docker/issues/18180#issuecomment-162568282):
docker-machine create -d virtualbox --engine-storage-driver overlay overlaymachine
Add System.exit(0) explicit at the end of each main method (cfr. https://github.com/docker/docker/issues/18180#issuecomment-161129296)
Wait for next docker patched version (https://github.com/docker/docker/issues/18180#issuecomment-170656525)
I think it would be good practice (for now) to add System.exit to all main methods in Ignite since this is independent of alternative hacks on docker or linux in general (linux kernel need AUFS upgrade and many machine may lag behind before that). This way future Ignite version can safely be installed on older kernels also.

JAVA_HOME error with upgrade to Spark 1.3.0

I’m trying to upgrade a Spark project, written in Scala, from Spark 1.2.1 to 1.3.0, so I changed my build.sbt like so:
-libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.1" % "provided"
+libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % "provided"
then make an assembly jar, and submit it:
HADOOP_CONF_DIR=/etc/hadoop/conf \
spark-submit \
--driver-class-path=/etc/hbase/conf \
--conf spark.hadoop.validateOutputSpecs=false \
--conf spark.yarn.jar=hdfs:/apps/local/spark-assembly-1.3.0-hadoop2.4.0.jar \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
--deploy-mode=cluster \
--master=yarn \
--class=TestObject \
--num-executors=54 \
target/scala-2.11/myapp-assembly-1.2.jar
The job fails to submit, with the following exception in the terminal:
15/03/19 10:30:07 INFO yarn.Client:
15/03/19 10:20:03 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1420225286501_4698 failed 2 times due to AM
Container for appattempt_1420225286501_4698_000002 exited with exitCode: 127
due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Finally, I go and check the YARN app master’s web interface (since the job is there, I know it at least made it that far), and the only logs it shows are these:
Log Type: stderr
Log Length: 61
/bin/bash: {{JAVA_HOME}}/bin/java: No such file or directory
Log Type: stdout
Log Length: 0
I’m not sure how to interpret that – is {{JAVA_HOME}} a literal (including the brackets) that’s somehow making it into a script? Is this coming from the worker nodes or the driver? Anything I can do to experiment & troubleshoot?
I do have JAVA_HOME set in the hadoop config files on all the nodes of the cluster:
% grep JAVA_HOME /etc/hadoop/conf/*.sh
/etc/hadoop/conf/hadoop-env.sh:export JAVA_HOME=/usr/jdk64/jdk1.6.0_31
/etc/hadoop/conf/yarn-env.sh:export JAVA_HOME=/usr/jdk64/jdk1.6.0_31
Has this behavior changed in 1.3.0 since 1.2.1? Using 1.2.1 and making no other changes, the job completes fine.
[Note: I originally posted this on the Spark mailing list, I'll update both places if/when I find a solution.]

Have you tried setting JAVA_HOME in the etc/hadoop/yarn-env.sh file? It's possible that your JAVA_HOME environment variable not available to the YARN containers that are running your job.
It has happened to me before that certain env variables that were in the .bashrc on the nodes were not being read by the yarn workers spawned on the cluster.
There is a chance that the error is unrelated to the version upgrade but instead related to YARN environment configuration.

Okay, so I got some other people in the office to help work on this, and we figured out a solution. I'm not sure how much of this is specific to the file layouts of Hortonworks HDP 2.0.6 on CentOS, which is what we're running on our cluster.
We manually copy some directories from one of the cluster machines (or any machine that can successfully use the Hadoop client) to your local machine. Let's call that machine $GOOD.
Set up Hadoop config files:
cd /etc
sudo mkdir hbase hadoop
sudo scp -r $GOOD:/etc/hbase/conf hbase
sudo scp -r $GOOD:/etc/hadoop/conf hadoop
Set up Hadoop libraries & executables:
mkdir ~/my-hadoop
scp -r $GOOD:/usr/lib/hadoop\* ~/my-hadoop
cd /usr/lib
sudo ln –s ~/my-hadoop/* .
path+=(/usr/lib/hadoop*/bin) # Add to $PATH (this syntax is for zsh)
Set up the Spark libraries & executables:
cd ~/Downloads
wget http://apache.mirrors.lucidnetworks.net/spark/spark-1.4.1/spark-1.4.1-bin-without-hadoop.tgz
tar -zxvf spark-1.4.1-bin-without-hadoop.tgz
cd spark-1.4.1-bin-without-hadoop
path+=(`pwd`/bin)
hdfs dfs -copyFromLocal lib/spark-assembly-*.jar /apps/local/
Set some environment variables:
export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_DIST_CLASSPATH=$(hadoop --config $HADOOP_CONF_DIR classpath)
`grep 'export HADOOP_LIBEXEC_DIR' $HADOOP_CONF_DIR/yarn-env.sh`
export SPOPTS="--driver-java-options=-Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib"
export SPOPTS="$SPOPTS --conf spark.yarn.jar=hdfs:/apps/local/spark-assembly-1.4.1-hadoop2.2.0.jar"
Now the various spark shells can be run like so:
sparkR --master yarn $SPOPTS
spark-shell --master yarn $SPOPTS
pyspark --master yarn $SPOPTS
Some remarks:
The JAVA_HOME setting is the same as I've had all along - just included it here for completion. All the focus on JAVA_HOME turned out to be a red herring.
The --driver-java-options=-Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib was necessary because I was getting errors about java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path. The jnilib file is the correct choice for OS X.
The --conf spark.yarn.jar piece is just to save time, avoiding re-copying the assembly file to the cluster every time you fire up the shell or submit a job.

Well, to start off I would recommend you to move to Java 7. However, that is not what you are looking for or need help with.
For setting JAVA_HOME, I would recommend you set it in your bashrc, rather than setting in multiple files. Moreover, I would recommend you installing java with alternatives to /usr/bin.

Launching a JBoss AS Cluster on Separate Machines to run EJBCA

Right now I am running my EJBCA project on single Jboss server using following command
run.sh -c default -b test.mywebsite.net
and it works fine, but now I want to launch my EJBCA application on clustered jboss machines
for this I have read following links
1.1.2. Launching a JBoss AS Cluster
1.1.3. Web Application Clustering Quick Start
then further I read that There are two aspects to setting up web tier clustering:
Configuring an External Load Balancer.
Configuring Your Web Application for Clustering.
now my questions are
When I go with the External Load Balancer. and when I go with second
option
If I go with second option how it will work ? and what will be my -b
option while running this Jboss servers.
because in point 1.1.2 I read that
Two processes can't bind sockets to the same address and port, so we'll have to tell JBoss to use different ports for the two instances.
EJBCA Version :- ejbca_4_0_12
JBOSS Version :- jboss-6.1.0.Final
EDIT 1
right now I am running my server like this
/run.sh -c default -b test.mysite.net
so it will listen on
https://test.mysite.net:8443/ejbca/
now I have to run my application as
Machine 1
./run.sh -c node1 -g DocsPartition -u 239.255.100.100 \
-b test.mysite.net -Djboss.messaging.ServerPeerID=1 \
-Djboss.service.binding.set=ports-default
Machine 2
./run.sh -c node2 -g DocsPartition -u 239.255.100.100 \
-b test.mysite.net -Djboss.messaging.ServerPeerID=2 \
-Djboss.service.binding.set=ports-default
so that I can only access to
https://test.mysite.net:8443/ejbca/
Please can any one explain this In brife so That I can clear with this things Thanks..

When you follow the documentation you mentioned (Scenario 3: Two Nodes on a Single, Non-Multihomed, Server) you have two JBoss instances:
./run.sh -c node1 -g DocsPartition -u 239.255.100.100 \
-b 192.168.0.101 -Djboss.messaging.ServerPeerID=1 \
-Djboss.service.binding.set=ports-default
./run.sh -c node2 -g DocsPartition -u 239.255.100.100 \
-b 192.168.0.101 -Djboss.messaging.ServerPeerID=2 \
-Djboss.service.binding.set=ports-01
They will listen on:
1) http://192.168.0.101:8080
2) http://192.168.0.101:8180
So to which node you will point your browser to reach the cluster? You can't use either of the two if you want failover. You need to configure a load-balancer.
Furthermore JBoss needs to know that it should replicate the http-session. Therefore you need to configure your web application for clustering. Hence you need to configure 1) and 2) not 1) or 2).
The -b x.x.x.x points to a (the) public IP-address of your server.