Develop Reference

Adding jars to a Spark Job - spark-submit

Following is my script to kick off my spark job
#!/bin/bash
APP_DIR="/home/arvind/myApp"
JARS=$(echo $APP_DIR/lib/* | tr ' ' ',')
/home/arvind/spark3/bin/spark-submit \
--master spark://server4:7078 \
--verbose \
--jars $JARS \
--driver-class-path $APP_DIR/conf \
--class com.test.spark.MySparkApplication \
--conf spark.driver.extraJavaOptions="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Dlog4j.configuration=file:$APP_DIR/conf/log4j.properties" \
--conf spark.driver.memory="32g" \
--conf spark.cores.max="20" \
$APP_DIR/myApp.jar > $APP_DIR/logs/output.err 2> $APP_DIR/logs/output.txt &
The lib folder contains all the jar files, packaged with the application, including postgres dependencies
[arvind#server4 myApp]$ ls lib/postgresql-42.1.4.jar
lib/postgresql-42.1.4.jar
When I run the application on a spark3 cluster, using the above script, I get the following exception
265 Caused by: java.lang.ClassNotFoundException: org.postgresql.ds.PGSimpleDataSource
266 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
267 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
268 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
269 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
270 at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:77)
271 ... 25 more
But if I change the above script to submit it to a spark2.4.0 cluster, it works fine.
Could you please help me with this ?

If you add the jar to the --driver-class-path like so it should work:
#!/bin/bash
APP_DIR="/home/arvind/myApp"
JARS=$(echo $APP_DIR/lib/* | tr ' ' ',')
COLON_SEP_JARS=$(echo $APP_DIR/lib/* | tr ' ' ':')
/home/arvind/spark3/bin/spark-submit \
--master spark://server4:7078 \
--verbose \
--jars $JARS \
--driver-class-path $APP_DIR/conf:$COLON_SEP_JARS \
--class com.test.spark.MySparkApplication \
--conf spark.driver.extraJavaOptions="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Dlog4j.configuration=file:$APP_DIR/conf/log4j.properties" \
--conf spark.driver.memory="32g" \
--conf spark.cores.max="20" \
$APP_DIR/myApp.jar > $APP_DIR/logs/output.err 2> $APP_DIR/logs/output.txt &

Getting error (File does not exist) during submitting spark job via spark-submit in yarn cluster

I have set up a yarn cluster with following configuration in core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:54310</value>
</property>
</configuration>
I am submitting spark job with following command;
./spark-submit --class main.MainClass --deploy-mode cluster --master yarn sparkJob.jar [options]
Now I am getting an exception after failing the job,
Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/user/root/.sparkStaging/application_1489672189113_0002/__spark_conf__.zip
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1122)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$6.apply(ApplicationMaster.scala:158)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$6.apply(ApplicationMaster.scala:155)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.deploy.yarn.ApplicationMaster.<init>(ApplicationMaster.scala:155)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:748)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:71)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:70)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:747)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
I am very new with yarn. Can you give any hint about this exception. If any more resource required please mention.
I am starting the SparkSession in my java code as follow
ss=SparkSession.builder()
.appName("appname")
.getOrCreate();
Anyone please, what am I doing wrong?

MySQL JDBC driver not found in Tomcat 7 on Ubuntu 14.04 LTS

I am deploying an application in a Tomcat 7 on My Ubuntu 14.04 LTS Profile.
I just added the file mysql-connector-java-5.1.25.jar to /var/lib/tomcat7/common/lib and /usr/share/tomcat7/lib, even that when I start the application I have this error:
Dec 19, 2015 9:29:14 AM org.apache.naming.NamingContext lookup
WARNING: Unexpected exception resolving reference
java.sql.SQLException: com.mysql.jdbc.Driver
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:254)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:486)
at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:144)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:116)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:103)
at org.apache.tomcat.jdbc.pool.DataSourceFactory.createDataSource(DataSourceFactory.java:554)
at org.apache.tomcat.jdbc.pool.DataSourceFactory.getObjectInstance(DataSourceFactory.java:242)
at org.apache.naming.factory.ResourceFactory.getObjectInstance(ResourceFactory.java:141)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:321)
at org.apache.naming.NamingContext.lookup(NamingContext.java:842)
at org.apache.naming.NamingContext.lookup(NamingContext.java:153)
at org.apache.naming.NamingContext.lookup(NamingContext.java:830)
at org.apache.naming.NamingContext.lookup(NamingContext.java:167)
at fr.telecom.config.PersistenceConfig.dataSource(PersistenceConfig.java:109)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.CGLIB$dataSource$2(<generated>)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c$$FastClassBySpringCGLIB$$1602d00c.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:318)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.dataSource(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.dataSource(<generated>)
at fr.telecom.config.PersistenceConfig.entityManagerFactory(PersistenceConfig.java:158)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.CGLIB$entityManagerFactory$1(<generated>)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c$$FastClassBySpringCGLIB$$1602d00c.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:318)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.entityManagerFactory(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1051)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:828)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:537)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:446)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:328)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4971)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1073)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:553)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1648)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:301)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at org.apache.catalina.manager.ManagerServlet.check(ManagerServlet.java:1445)
at org.apache.catalina.manager.HTMLManagerServlet.upload(HTMLManagerServlet.java:284)
at org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:213)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:246)
... 106 more
this is the tomcat start script
#!/bin/sh
#
# /etc/init.d/tomcat7 -- startup script for the Tomcat 6 servlet engine
#
# Written by Miquel van Smoorenburg <miquels#cistron.nl>.
# Modified for Debian GNU/Linux by Ian Murdock <imurdock#gnu.ai.mit.edu>.
# Modified for Tomcat by Stefan Gybas <sgybas#debian.org>.
# Modified for Tomcat6 by Thierry Carrez <thierry.carrez#ubuntu.com>.
# Modified for Tomcat7 by Ernesto Hernandez-Novich <emhn#itverx.com.ve>.
# Additional improvements by Jason Brittain <jason.brittain#mulesoft.com>.
#
### BEGIN INIT INFO
# Provides: tomcat7
# Required-Start: $local_fs $remote_fs $network
# Required-Stop: $local_fs $remote_fs $network
# Should-Start: $named
# Should-Stop: $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start Tomcat.
# Description: Start the Tomcat servlet engine.
### END INIT INFO
set -e
PATH=/bin:/usr/bin:/sbin:/usr/sbin
NAME=tomcat7
DESC="Tomcat servlet engine"
DEFAULT=/etc/default/$NAME
JVM_TMP=/tmp/tomcat7-$NAME-tmp
if [ `id -u` -ne 0 ]; then
echo "You need root privileges to run this script"
exit 1
fi
# Make sure tomcat is started with system locale
if [ -r /etc/default/locale ]; then
. /etc/default/locale
export LANG
fi
. /lib/lsb/init-functions
if [ -r /etc/default/rcS ]; then
. /etc/default/rcS
fi
# The following variables can be overwritten in $DEFAULT
# Run Tomcat 7 as this user ID and group ID
TOMCAT7_USER=tomcat7
TOMCAT7_GROUP=tomcat7
# this is a work-around until there is a suitable runtime replacement
# for dpkg-architecture for arch:all packages
# this function sets the variable OPENJDKS
find_openjdks()
{
for jvmdir in /usr/lib/jvm/java-7-openjdk-*
do
if [ -d "${jvmdir}" -a "${jvmdir}" != "/usr/lib/jvm/java-7-openjdk-common" ]
then
OPENJDKS=$jvmdir
fi
done
for jvmdir in /usr/lib/jvm/java-6-openjdk-*
do
if [ -d "${jvmdir}" -a "${jvmdir}" != "/usr/lib/jvm/java-6-openjdk-common" ]
then
OPENJDKS="${OPENJDKS} ${jvmdir}"
fi
done
}
OPENJDKS=""
find_openjdks
# The first existing directory is used for JAVA_HOME (if JAVA_HOME is not
# defined in $DEFAULT)
JDK_DIRS="/usr/lib/jvm/default-java ${OPENJDKS} /usr/lib/jvm/java-6-openjdk /usr/lib/jvm/java-6-sun /usr/lib/jvm/java-7-oracle"
# Look for the right JVM to use
for jdir in $JDK_DIRS; do
if [ -r "$jdir/bin/java" -a -z "${JAVA_HOME}" ]; then
JAVA_HOME="$jdir"
fi
done
export JAVA_HOME
# Directory where the Tomcat 6 binary distribution resides
CATALINA_HOME=/usr/share/$NAME
# Directory for per-instance configuration files and webapps
CATALINA_BASE=/var/lib/$NAME
# Use the Java security manager? (yes/no)
TOMCAT7_SECURITY=no
# Default Java options
# Set java.awt.headless=true if JAVA_OPTS is not set so the
# Xalan XSL transformer can work without X11 display on JDK 1.4+
# It also looks like the default heap size of 64M is not enough for most cases
# so the maximum heap size is set to 128M
if [ -z "$JAVA_OPTS" ]; then
JAVA_OPTS="-Djava.awt.headless=true -Xmx128M"
fi
# End of variables that can be overwritten in $DEFAULT
# overwrite settings from default file
if [ -f "$DEFAULT" ]; then
. "$DEFAULT"
fi
if [ ! -f "$CATALINA_HOME/bin/bootstrap.jar" ]; then
log_failure_msg "$NAME is not installed"
exit 1
fi
POLICY_CACHE="$CATALINA_BASE/work/catalina.policy"
if [ -z "$CATALINA_TMPDIR" ]; then
CATALINA_TMPDIR="$JVM_TMP"
fi
# Set the JSP compiler if set in the tomcat7.default file
if [ -n "$JSP_COMPILER" ]; then
JAVA_OPTS="$JAVA_OPTS -Dbuild.compiler=\"$JSP_COMPILER\""
fi
SECURITY=""
if [ "$TOMCAT7_SECURITY" = "yes" ]; then
SECURITY="-security"
fi
# Define other required variables
CATALINA_PID="/var/run/$NAME.pid"
CATALINA_SH="$CATALINA_HOME/bin/catalina.sh"
# Look for Java Secure Sockets Extension (JSSE) JARs
if [ -z "${JSSE_HOME}" -a -r "${JAVA_HOME}/jre/lib/jsse.jar" ]; then
JSSE_HOME="${JAVA_HOME}/jre/"
fi
catalina_sh() {
# Escape any double quotes in the value of JAVA_OPTS
JAVA_OPTS="$(echo $JAVA_OPTS | sed 's/\"/\\\"/g')"
AUTHBIND_COMMAND=""
if [ "$AUTHBIND" = "yes" -a "$1" = "start" ]; then
AUTHBIND_COMMAND="/usr/bin/authbind --deep /bin/bash -c "
fi
# Define the command to run Tomcat's catalina.sh as a daemon
# set -a tells sh to export assigned variables to spawned shells.
TOMCAT_SH="set -a; JAVA_HOME=\"$JAVA_HOME\"; source \"$DEFAULT\"; \
CATALINA_HOME=\"$CATALINA_HOME\"; \
CATALINA_BASE=\"$CATALINA_BASE\"; \
JAVA_OPTS=\"$JAVA_OPTS\"; \
CATALINA_PID=\"$CATALINA_PID\"; \
CATALINA_TMPDIR=\"$CATALINA_TMPDIR\"; \
LANG=\"$LANG\"; JSSE_HOME=\"$JSSE_HOME\"; \
cd \"$CATALINA_BASE\"; \
\"$CATALINA_SH\" $#"
if [ "$AUTHBIND" = "yes" -a "$1" = "start" ]; then
TOMCAT_SH="'$TOMCAT_SH'"
fi
# Run the catalina.sh script as a daemon
set +e
touch "$CATALINA_PID" "$CATALINA_BASE"/logs/catalina.out
chown $TOMCAT7_USER "$CATALINA_PID" "$CATALINA_BASE"/logs/catalina.out
start-stop-daemon --start -b -u "$TOMCAT7_USER" -g "$TOMCAT7_GROUP" \
-c "$TOMCAT7_USER" -d "$CATALINA_TMPDIR" -p "$CATALINA_PID" \
-x /bin/bash -- -c "$AUTHBIND_COMMAND $TOMCAT_SH"
status="$?"
set +a -e
return $status
}
case "$1" in
start)
if [ -z "$JAVA_HOME" ]; then
log_failure_msg "no JDK or JRE found - please set JAVA_HOME"
exit 1
fi
if [ ! -d "$CATALINA_BASE/conf" ]; then
log_failure_msg "invalid CATALINA_BASE: $CATALINA_BASE"
exit 1
fi
log_daemon_msg "Starting $DESC" "$NAME"
if start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT7_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null; then
# Regenerate POLICY_CACHE file
umask 022
echo "// AUTO-GENERATED FILE from /etc/tomcat7/policy.d/" \
> "$POLICY_CACHE"
echo "" >> "$POLICY_CACHE"
cat $CATALINA_BASE/conf/policy.d/*.policy \
>> "$POLICY_CACHE"
# Remove / recreate JVM_TMP directory
rm -rf "$JVM_TMP"
mkdir -p "$JVM_TMP" || {
log_failure_msg "could not create JVM temporary directory"
exit 1
}
chown $TOMCAT7_USER "$JVM_TMP"
catalina_sh start $SECURITY
sleep 5
if start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT7_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null; then
if [ -f "$CATALINA_PID" ]; then
rm -f "$CATALINA_PID"
fi
log_end_msg 1
else
log_end_msg 0
fi
else
log_progress_msg "(already running)"
log_end_msg 0
fi
;;
stop)
log_daemon_msg "Stopping $DESC" "$NAME"
set +e
if [ -f "$CATALINA_PID" ]; then
start-stop-daemon --stop --pidfile "$CATALINA_PID" \
--user "$TOMCAT7_USER" \
--retry=TERM/20/KILL/5 >/dev/null
if [ $? -eq 1 ]; then
log_progress_msg "$DESC is not running but pid file exists, cleaning up"
elif [ $? -eq 3 ]; then
PID="`cat $CATALINA_PID`"
log_failure_msg "Failed to stop $NAME (pid $PID)"
exit 1
fi
rm -f "$CATALINA_PID"
rm -rf "$JVM_TMP"
else
log_progress_msg "(not running)"
fi
log_end_msg 0
set -e
;;
status)
set +e
start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT7_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null 2>&1
if [ "$?" = "0" ]; then
if [ -f "$CATALINA_PID" ]; then
log_success_msg "$DESC is not running, but pid file exists."
exit 1
else
log_success_msg "$DESC is not running."
exit 3
fi
else
log_success_msg "$DESC is running with pid `cat $CATALINA_PID`"
fi
set -e
;;
restart|force-reload)
if [ -f "$CATALINA_PID" ]; then
$0 stop
sleep 1
fi
$0 start
;;
try-restart)
if start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT7_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null; then
$0 start
fi
;;
*)
log_success_msg "Usage: $0 {start|stop|restart|try-restart|force-reload|status}"
exit 1
;;
esac
exit 0

The library was well placed.
It was a lack of permissions:
chmod 777 mysql-connector-java-5.1.25.jar

Very likely /var/lib/tomcat7/common/lib is not on the classpath. Check your Tomcat startup script. Have you try adding it to /var/lib/tomcat7/ assuming this is the location that tomcat load its base lib.

Spark: Could not load native gpl library

I had the following error when trying to run a very simple spark job (which uses logistic regression with SGD in mllib):
ERROR GPLNativeCodeLoader: Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1028)
at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32)
at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1659)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1624)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:155)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:187)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:181)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:93)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
14/08/06 20:32:11 ERROR LzoCodec: Cannot load native-lzo without native-hadoop
This is the command I used to submit the job:
~/spark/spark-1.0.0-bin-hadoop2/bin/spark-submit \
--class com.jk.sparktest.Test \
--master yarn-cluster \
--num-executors 40 \
~/sparktest-0.0.1-SNAPSHOT-jar-with-dependencies.jar
The actual java command is :
/usr/java/latest/bin/java -cp /apache/hadoop/share/hadoop/common/hadoop-common-2.2.0.2.0.6.0-61.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar::/home/jilei/spark/spark-1.0.0-bin-hadoop2/conf:/home/jilei/spark/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar:/home/jilei/spark/spark-1.0.0-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar:/home/jilei/spark/spark-1.0.0-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/home/jilei/spark/spark-1.0.0-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar:/apache/hadoop/conf:/apache/hadoop/conf \
-XX:MaxPermSize=128m \
-Djava.library.path=
-Xms512m -Xmx512m org.apache.spark.deploy.SparkSubmit \
--class com.jk.sparktest.Test \
--master yarn-cluster \
--num-executors 40 \
~/sparktest-0.0.1-SNAPSHOT-jar-with-dependencies.jar
Seems the -Djava.library.path is not set. I also tried the java command above and supplied the native lib directory to the java.library.path, but still got the same errors. This is the content of the native lib directory:
ls /apache/hadoop/lib/native/
libgplcompression.a libgplcompression.so libgplcompression.so.0.0.0 libhadooppipes.a
libhadoop.so.1.0.0 libhdfs.a libhdfs.so.0.0.0 libsnappy.so.1
libgplcompression.la libgplcompression.so.0 libhadoop.a libhadoop.so
libhadooputils.a libhdfs.so libsnappy.so libsnappy.so.1.1.4
Any idea on what's wrong? Thanks.

I don't know why what you tried with java.library.path didn't work. I solved a problem similar to what you described by adding the argument:
--driver-library-path /usr/lib/hadoop-lzo/lib/native/
where /usr/lib/hadoop-lzo/lib/native/ contained libgplcompression.so.

Jruby error (wrong name: ruby/jit/FILE_D40BB14F6D953C9F0017A388A07364EC0E5B3F6E)

I am trying to compile a file using jruby with the following commands but I have not had any luck.
jruby -S jrubyc test5.rb
java -cp .:/Users/Ted/.rvm/rubies/jruby-1.6.2/lib test5
I get the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: test5 (wrong name: ruby/jit/FILE_D40BB14F6D953C9F0017A388A07364EC0E5B3F6E)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:698)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:315)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:330)
at java.lang.ClassLoader.loadClass(ClassLoader.java:250)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:398)
Ted

You need to package your class in a jar file.
Also, your -cp argument needs to specify an actual jar file (or zip) - it cannot refer to a directory as in your example.