Develop Reference

Pyspark submit py-files as zip: java.io.FileNotFoundException

I am submitting a Pyspark job with all modules packaged in a zip file like so:
$SPARK_HOME/bin/spark-submit \
--master local[*] \
--deploy-mode client \
--name spark-python \
--conf spark.driver.memory=4g \
--files "https://raw.githubusercontent.com/config.yml" \
--py-files "https://github.com/jobs.zip?raw=true" \
"https://raw.githubusercontent.com/main.py"
But the job throws the following exception:
Traceback (most recent call last):
File "/tmp/spark-637f7cfd-ff09-4784-9aa6-110779426bd4/main_kubernetes_new.py", line 62, in <module>
main()
File "/tmp/spark-637f7cfd-ff09-4784-9aa6-110779426bd4/main_kubernetes_new.py", line 27, in main
spark = SparkSession.builder.getOrCreate()
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 228, in getOrCreate
File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 384, in getOrCreate
File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 146, in __init__
File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 209, in _do_init
File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 321, in _initialize_context
File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1568, in __call__
File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.io.FileNotFoundException: https://github.com/LorenzWackenhut/py-files-public/blob/main/jobs.zip%3Fraw=true
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:765)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:541)
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1633)
at org.apache.spark.SparkContext.$anonfun$new$13(SparkContext.scala:508)
at org.apache.spark.SparkContext.$anonfun$new$13$adapted(SparkContext.scala:508)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:508)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Is the way I supply the .zip file the problem? Or is there something else I overlooked?
Thank you in advance!

Mounting Maven Repository to Docker

I am trying to build a Java application and make a package using docker. This builds needs a maven repository which I don't want to include in the image, since it's very large. I wanted to try using volumes and mount my local maven repository to the maven repository in the image. I used apt-get install -y maven in order to have maven available, but I can't find the directory .m2 in the image $HOME.
I used ls -la $HOME, ls -la and ls -la /root to find the maven home, but there is no .m2 directory there.
EDIT 1:
I have these lines in Dockerfile:
FROM ubuntu
MAINTAINER Zeinab Abbasimazar
# Install and configure required packages
RUN apt-get update; \
apt-get install -y --no-install-recommends apt-utils; \
apt-get install -y dialog; \
apt-get install -y wget unzip curl maven; \
mkdir $HOME/.m2/; \
ls -la /usr/share/maven/conf/; \
echo \
"<settings xmlns='http://maven.apache.org/SETTINGS/1.0.0\' \
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' \
xsi:schemaLocation='http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd'> \
<localRepository>/root/.m2/repository</localRepository> \
<interactiveMode>true</interactiveMode> \
<usePluginRegistry>false</usePluginRegistry> \
<offline>false</offline> \
</settings>" \
> /usr/share/maven/conf/settings.xml
VOLUME ["/home/zeinab/.m2/", "/root/.m2/"]
# Build
RUN mvn -X clean install -pl components -P profile
Which puts local repository configurations in image's maven configuration file, mounts my local maven repository to a directory in the image and finally performs the build. As I can see in the maven build log that it's using the local repository path I expected:
[DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml
[DEBUG] Reading user settings from /root/.m2/settings.xml
[DEBUG] Using local repository at /root/.m2/repository
But still can't detect dependencies.

I finally found the solution for mounting my local maven repository in docker. I changed my solution; I am mounting it in the run phase instead of build phase. This is my Dockerfile:
FROM ubuntu
MAINTAINER Zeinab Abbasimazar
ADD gwr $HOME
RUN apt-get update; \
apt-get install -y --no-install-recommends apt-utils; \
apt-get install -y wget unzip curl maven git; \
echo \
"<settings xmlns='http://maven.apache.org/SETTINGS/1.0.0\' \
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' \
xsi:schemaLocation='http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd'> \
<localRepository>/root/.m2/repository</localRepository> \
<interactiveMode>true</interactiveMode> \
<usePluginRegistry>false</usePluginRegistry> \
<offline>false</offline> \
</settings>" \
> /usr/share/maven/conf/settings.xml; \
mkdir /root/.m2/; \
echo \
"<settings xmlns='http://maven.apache.org/SETTINGS/1.0.0\' \
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' \
xsi:schemaLocation='http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd'> \
<localRepository>/root/.m2/repository</localRepository> \
<interactiveMode>true</interactiveMode> \
<usePluginRegistry>false</usePluginRegistry> \
<offline>false</offline> \
</settings>" \
> /root/.m2/settings.xml
WORKDIR .
CMD mvn -X clean install -pl components -P profile
At first, I build the image using above Dockerfile:
sudo docker build -t imageName:imageTag .
Then, I run a container as below:
sudo docker run -d -v /home/zeinab/.m2/:/root/.m2/ --name containerName imageName:imageTag

You don't find the ~/.m2 directory because it is created only when needed, i.e. when you store libraries in the local repository or when you add a config file.
You can create the ~/.m2 directory yourself and create your own settings.xml inside. There you can define the emplacement of the local repository:
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
https://maven.apache.org/xsd/settings-1.0.0.xsd">
...
<localRepository>/path/to/local/repo/</localRepository>
...
</settings>
Read the documentation for more details.

Java build error in Android apps packages

I am trying to build apps packages of LineageOS. I am getting the following error. This seems to be a JAVA error. I am using oracle jdk 8.
I am already using /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/ as my ANDROID_JAVA_HOME variable.
[ 6% 196/3238] host Java: doclava (/Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes)
FAILED: /bin/bash -c "(rm -f /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes-full-debug.jar ) && (rm -rf /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes ) && (mkdir -p /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/ ) && (mkdir -p /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes ) && (for f in /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/tagsouplib_intermediates/javalib.jar; do if [ ! -f \$f ]; then echo Missing file \$f; exit 1; fi; unzip -qo \$f -d /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes; done ) && (rm -rf /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/META-INF ) && (rm -f /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list ) && (touch /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list ) && (echo -n 'external/doclava/src/com/google/doclava/AnnotationInstanceInfo.java external/doclava/src/com/google/doclava/AnnotationValueInfo.java external/doclava/src/com/google/doclava/AtLinksNavTree.java external/doclava/src/com/google/doclava/AttrTagInfo.java external/doclava/src/com/google/doclava/AttributeInfo.java external/doclava/src/com/google/doclava/ClassInfo.java external/doclava/src/com/google/doclava/ClearPage.java external/doclava/src/com/google/doclava/CodeTagInfo.java external/doclava/src/com/google/doclava/Comment.java external/doclava/src/com/google/doclava/ContainerInfo.java external/doclava/src/com/google/doclava/Converter.java external/doclava/src/com/google/doclava/DocFile.java external/doclava/src/com/google/doclava/DocInfo.java external/doclava/src/com/google/doclava/Doclava.java external/doclava/src/com/google/doclava/Doclava2.java external/doclava/src/com/google/doclava/DoclavaDiff.java external/doclava/src/com/google/doclava/Errors.java external/doclava/src/com/google/doclava/FederatedSite.java external/doclava/src/com/google/doclava/FederationTagger.java external/doclava/src/com/google/doclava/FieldInfo.java external/doclava/src/com/google/doclava/Hierarchy.java external/doclava/src/com/google/doclava/InfoBuilder.java external/doclava/src/com/google/doclava/InheritedTags.java external/doclava/src/com/google/doclava/JarUtils.java external/doclava/src/com/google/doclava/KeywordEntry.java external/doclava/src/com/google/doclava/LinkReference.java external/doclava/src/com/google/doclava/LiteralTagInfo.java external/doclava/src/com/google/doclava/MemberInfo.java external/doclava/src/com/google/doclava/MethodInfo.java external/doclava/src/com/google/doclava/NavTree.java external/doclava/src/com/google/doclava/PackageInfo.java external/doclava/src/com/google/doclava/PageMetadata.java external/doclava/src/com/google/doclava/ParamTagInfo.java external/doclava/src/com/google/doclava/ParameterInfo.java external/doclava/src/com/google/doclava/ParsedTagInfo.java external/doclava/src/com/google/doclava/Proofread.java external/doclava/src/com/google/doclava/Resolution.java external/doclava/src/com/google/doclava/Resolvable.java external/doclava/src/com/google/doclava/SampleCode.java external/doclava/src/com/google/doclava/SampleTagInfo.java external/doclava/src/com/google/doclava/Scoped.java external/doclava/src/com/google/doclava/SeeTagInfo.java external/doclava/src/com/google/doclava/SinceTagger.java external/doclava/src/com/google/doclava/Sorter.java external/doclava/src/com/google/doclava/SourcePositionInfo.java external/doclava/src/com/google/doclava/Stubs.java external/doclava/src/com/google/doclava/TagInfo.java external/doclava/src/com/google/doclava/TextTagInfo.java external/doclava/src/com/google/doclava/ThrowsTagInfo.java external/doclava/src/com/google/doclava/TodoFile.java external/doclava/src/com/google/doclava/TypeInfo.java external/doclava/src/com/google/doclava/apicheck/AbstractMethodInfo.java external/doclava/src/com/google/doclava/apicheck/ApiCheck.java external/doclava/src/com/google/doclava/apicheck/ApiFile.java external/doclava/src/com/google/doclava/apicheck/ApiInfo.java external/doclava/src/com/google/doclava/apicheck/ApiParseException.java external/doclava/src/com/google/doclava/apicheck/XmlApiFile.java external/doclava/src/com/google/doclava/parser/JavaLexer.java external/doclava/src/com/google/doclava/parser/JavaParser.java ' >> /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list ) && (if [ -d \"/Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/src\" ]; then find /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/src -name '*.java' -and -not -name '.*' >> /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list; fi ) && (tr ' ' '\\n' < /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list | build/tools/normalize_path.py | sort -u > /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list-uniq ) && (if [ -s /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list-uniq ] ; then javac -J-Xmx1024M -Xmaxerrs 9999999 -encoding UTF-8 -classpath /Volumes/SSD/android/system/out/host/darwin-x86/framework/antlr-runtime.jar:/Volumes/SSD/android/system/out/host/darwin-x86/framework/guavalib.jar:/Volumes/SSD/android/system/out/host/darwin-x86/framework/jsilver.jar:/Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/tagsouplib_intermediates/javalib.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/lib/tools.jar -extdirs \"\" -d /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes -g -source 1.8 -target 1.8 \\#/Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list-uniq 2>/Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/stderr && ( [ -s /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/stderr ] && echo \"\`cat /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/stderr\`\" 1>&2; rm -f /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/stderr ) || ( [ -s /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/stderr ] && echo \"\`cat /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/stderr\`\" 1>&2; rm -rf /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes; exit 41 ) fi ) && (rm -f /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list ) && (rm -f /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes/java-source-list-uniq ) && (jar -cf /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes-full-debug.jar -C /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes . ) && (rm -f /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes-full-debug.jar.jar-arg-list ) && (touch /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes-full-debug.jar.jar-arg-list ) && (echo -n '-C \"external/doclava/res\" \"assets/customizations/assets/customizations.css\" -C \"external/doclava/res\" \"assets/customizations/assets/customizations.js\" -C \"external/doclava/res\" \"assets/customizations/customizations.cs\" -C \"external/doclava/res\" \"assets/templates/assets/doclava-developer-core.css\" -C \"external/doclava/res\" \"assets/templates/assets/doclava-developer-docs.css\" -C \"external/doclava/res\" \"assets/templates/assets/doclava-developer-docs.js\" -C \"external/doclava/res\" \"assets/templates/assets/doclava-developer-reference.js\" -C \"external/doclava/res\" \"assets/templates/assets/images/bg_fade.jpg\" -C \"external/doclava/res\" \"assets/templates/assets/images/bg_logo.png\" -C \"external/doclava/res\" \"assets/templates/assets/images/body-gradient-tab.png\" -C \"external/doclava/res\" \"assets/templates/assets/images/body-gradient.png\" -C \"external/doclava/res\" \"assets/templates/assets/images/grad-rule-qv.png\" -C \"external/doclava/res\" \"assets/templates/assets/images/hr_gray_main.jpg\" -C \"external/doclava/res\" \"assets/templates/assets/images/hr_gray_side.jpg\" -C \"external/doclava/res\" \"assets/templates/assets/images/left_off.jpg\" -C \"external/doclava/res\" \"assets/templates/assets/images/left_on.jpg\" -C \"external/doclava/res\" \"assets/templates/assets/images/preliminary.png\" -C \"external/doclava/res\" \"assets/templates/assets/images/resizable-e.gif\" -C \"external/doclava/res\" \"assets/templates/assets/images/resizable-e2.gif\" -C \"external/doclava/res\" \"assets/templates/assets/images/resizable-eg.gif\" -C \"external/doclava/res\" \"assets/templates/assets/images/resizable-s.gif\" -C \"external/doclava/res\" \"assets/templates/assets/images/resizable-s2.gif\" -C \"external/doclava/res\" \"assets/templates/assets/images/resizable-sg.gif\" -C \"external/doclava/res\" \"assets/templates/assets/images/right_off.jpg\" -C \"external/doclava/res\" \"assets/templates/assets/images/right_on.jpg\" -C \"external/doclava/res\" \"assets/templates/assets/images/sidenav-rule.png\" -C \"external/doclava/res\" \"assets/templates/assets/images/spacer.gif\" -C \"external/doclava/res\" \"assets/templates/assets/images/triangle-closed-small.png\" -C \"external/doclava/res\" \"assets/templates/assets/images/triangle-closed.png\" -C \"external/doclava/res\" \"assets/templates/assets/images/triangle-opened-small.png\" -C \"external/doclava/res\" \"assets/templates/assets/images/triangle-opened.png\" -C \"external/doclava/res\" \"assets/templates/assets/jquery-history.js\" -C \"external/doclava/res\" \"assets/templates/assets/jquery-resizable.min.js\" -C \"external/doclava/res\" \"assets/templates/assets/prettify.js\" -C \"external/doclava/res\" \"assets/templates/assets/search_autocomplete.js\" -C \"external/doclava/res\" \"assets/templates/assets/style.css\" -C \"external/doclava/res\" \"assets/templates/assets/triangle-none.gif\" -C \"external/doclava/res\" \"assets/templates/at_links_navtree.cs\" -C \"external/doclava/res\" \"assets/templates/class.cs\" -C \"external/doclava/res\" \"assets/templates/classes.cs\" -C \"external/doclava/res\" \"assets/templates/components.cs\" -C \"external/doclava/res\" \"assets/templates/components/api_filter.cs\" -C \"external/doclava/res\" \"assets/templates/components/left_nav.cs\" -C \"external/doclava/res\" \"assets/templates/components/masthead.cs\" -C \"external/doclava/res\" \"assets/templates/components/search_box.cs\" -C \"external/doclava/res\" \"assets/templates/data.hdf\" -C \"external/doclava/res\" \"assets/templates/designpage.cs\" -C \"external/doclava/res\" \"assets/templates/diff.cs\" -C \"external/doclava/res\" \"assets/templates/docpage.cs\" -C \"external/doclava/res\" \"assets/templates/doctype.cs\" -C \"external/doclava/res\" \"assets/templates/footer.cs\" -C \"external/doclava/res\" \"assets/templates/head_tag.cs\" -C \"external/doclava/res\" \"assets/templates/header.cs\" -C \"external/doclava/res\" \"assets/templates/hierarchy.cs\" -C \"external/doclava/res\" \"assets/templates/index.cs\" -C \"external/doclava/res\" \"assets/templates/jd_lists.cs\" -C \"external/doclava/res\" \"assets/templates/jd_lists_unified.cs\" -C \"external/doclava/res\" \"assets/templates/keywords.cs\" -C \"external/doclava/res\" \"assets/templates/lists.cs\" -C \"external/doclava/res\" \"assets/templates/macros.cs\" -C \"external/doclava/res\" \"assets/templates/navtree_data.cs\" -C \"external/doclava/res\" \"assets/templates/nosidenavpage.cs\" -C \"external/doclava/res\" \"assets/templates/package-list.cs\" -C \"external/doclava/res\" \"assets/templates/package.cs\" -C \"external/doclava/res\" \"assets/templates/packages.cs\" -C \"external/doclava/res\" \"assets/templates/sample.cs\" -C \"external/doclava/res\" \"assets/templates/sampleindex.cs\" -C \"external/doclava/res\" \"assets/templates/timestamp.cs\" -C \"external/doclava/res\" \"assets/templates/todo.cs\" -C \"external/doclava/res\" \"assets/templates/trailer.cs\" -C \"external/doclava/res\" \"assets/templates/yaml_navtree.cs\" ' >> /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes-full-debug.jar.jar-arg-list ) && (jar uf /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes-full-debug.jar #/Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes-full-debug.jar.jar-arg-list ) && (rm -f /Volumes/SSD/android/system/out/host/common/obj/JAVA_LIBRARIES/doclava_intermediates/classes-full-debug.jar.jar-arg-list )"
external/doclava/src/com/google/doclava/Converter.java:407: error: cannot find symbol
m.isAbstract(), m.isSynchronized(), m.isNative(), m.isDefault(), true,
^
symbol: method isDefault()
location: variable m of type AnnotationTypeElementDoc
external/doclava/src/com/google/doclava/Converter.java:429: error: cannot find symbol
m.isAbstract(), m.isSynchronized(), m.isNative(), m.isDefault(), false,
^
symbol: method isDefault()
location: variable m of type MethodDoc
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
2 errors
ninja: build stopped: subcommand failed.
make: *** [ninja_wrapper] Error 1
Help Please

MySQL JDBC driver not found in Tomcat 7 on Ubuntu 14.04 LTS

I am deploying an application in a Tomcat 7 on My Ubuntu 14.04 LTS Profile.
I just added the file mysql-connector-java-5.1.25.jar to /var/lib/tomcat7/common/lib and /usr/share/tomcat7/lib, even that when I start the application I have this error:
Dec 19, 2015 9:29:14 AM org.apache.naming.NamingContext lookup
WARNING: Unexpected exception resolving reference
java.sql.SQLException: com.mysql.jdbc.Driver
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:254)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:486)
at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:144)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:116)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:103)
at org.apache.tomcat.jdbc.pool.DataSourceFactory.createDataSource(DataSourceFactory.java:554)
at org.apache.tomcat.jdbc.pool.DataSourceFactory.getObjectInstance(DataSourceFactory.java:242)
at org.apache.naming.factory.ResourceFactory.getObjectInstance(ResourceFactory.java:141)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:321)
at org.apache.naming.NamingContext.lookup(NamingContext.java:842)
at org.apache.naming.NamingContext.lookup(NamingContext.java:153)
at org.apache.naming.NamingContext.lookup(NamingContext.java:830)
at org.apache.naming.NamingContext.lookup(NamingContext.java:167)
at fr.telecom.config.PersistenceConfig.dataSource(PersistenceConfig.java:109)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.CGLIB$dataSource$2(<generated>)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c$$FastClassBySpringCGLIB$$1602d00c.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:318)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.dataSource(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.dataSource(<generated>)
at fr.telecom.config.PersistenceConfig.entityManagerFactory(PersistenceConfig.java:158)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.CGLIB$entityManagerFactory$1(<generated>)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c$$FastClassBySpringCGLIB$$1602d00c.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:318)
at fr.telecom.config.PersistenceConfig$$EnhancerBySpringCGLIB$$424c908c.entityManagerFactory(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1051)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:828)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:537)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:446)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:328)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4971)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1073)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:553)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1648)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:301)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at org.apache.catalina.manager.ManagerServlet.check(ManagerServlet.java:1445)
at org.apache.catalina.manager.HTMLManagerServlet.upload(HTMLManagerServlet.java:284)
at org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:213)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:246)
... 106 more
this is the tomcat start script
#!/bin/sh
#
# /etc/init.d/tomcat7 -- startup script for the Tomcat 6 servlet engine
#
# Written by Miquel van Smoorenburg <miquels#cistron.nl>.
# Modified for Debian GNU/Linux by Ian Murdock <imurdock#gnu.ai.mit.edu>.
# Modified for Tomcat by Stefan Gybas <sgybas#debian.org>.
# Modified for Tomcat6 by Thierry Carrez <thierry.carrez#ubuntu.com>.
# Modified for Tomcat7 by Ernesto Hernandez-Novich <emhn#itverx.com.ve>.
# Additional improvements by Jason Brittain <jason.brittain#mulesoft.com>.
#
### BEGIN INIT INFO
# Provides: tomcat7
# Required-Start: $local_fs $remote_fs $network
# Required-Stop: $local_fs $remote_fs $network
# Should-Start: $named
# Should-Stop: $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start Tomcat.
# Description: Start the Tomcat servlet engine.
### END INIT INFO
set -e
PATH=/bin:/usr/bin:/sbin:/usr/sbin
NAME=tomcat7
DESC="Tomcat servlet engine"
DEFAULT=/etc/default/$NAME
JVM_TMP=/tmp/tomcat7-$NAME-tmp
if [ `id -u` -ne 0 ]; then
echo "You need root privileges to run this script"
exit 1
fi
# Make sure tomcat is started with system locale
if [ -r /etc/default/locale ]; then
. /etc/default/locale
export LANG
fi
. /lib/lsb/init-functions
if [ -r /etc/default/rcS ]; then
. /etc/default/rcS
fi
# The following variables can be overwritten in $DEFAULT
# Run Tomcat 7 as this user ID and group ID
TOMCAT7_USER=tomcat7
TOMCAT7_GROUP=tomcat7
# this is a work-around until there is a suitable runtime replacement
# for dpkg-architecture for arch:all packages
# this function sets the variable OPENJDKS
find_openjdks()
{
for jvmdir in /usr/lib/jvm/java-7-openjdk-*
do
if [ -d "${jvmdir}" -a "${jvmdir}" != "/usr/lib/jvm/java-7-openjdk-common" ]
then
OPENJDKS=$jvmdir
fi
done
for jvmdir in /usr/lib/jvm/java-6-openjdk-*
do
if [ -d "${jvmdir}" -a "${jvmdir}" != "/usr/lib/jvm/java-6-openjdk-common" ]
then
OPENJDKS="${OPENJDKS} ${jvmdir}"
fi
done
}
OPENJDKS=""
find_openjdks
# The first existing directory is used for JAVA_HOME (if JAVA_HOME is not
# defined in $DEFAULT)
JDK_DIRS="/usr/lib/jvm/default-java ${OPENJDKS} /usr/lib/jvm/java-6-openjdk /usr/lib/jvm/java-6-sun /usr/lib/jvm/java-7-oracle"
# Look for the right JVM to use
for jdir in $JDK_DIRS; do
if [ -r "$jdir/bin/java" -a -z "${JAVA_HOME}" ]; then
JAVA_HOME="$jdir"
fi
done
export JAVA_HOME
# Directory where the Tomcat 6 binary distribution resides
CATALINA_HOME=/usr/share/$NAME
# Directory for per-instance configuration files and webapps
CATALINA_BASE=/var/lib/$NAME
# Use the Java security manager? (yes/no)
TOMCAT7_SECURITY=no
# Default Java options
# Set java.awt.headless=true if JAVA_OPTS is not set so the
# Xalan XSL transformer can work without X11 display on JDK 1.4+
# It also looks like the default heap size of 64M is not enough for most cases
# so the maximum heap size is set to 128M
if [ -z "$JAVA_OPTS" ]; then
JAVA_OPTS="-Djava.awt.headless=true -Xmx128M"
fi
# End of variables that can be overwritten in $DEFAULT
# overwrite settings from default file
if [ -f "$DEFAULT" ]; then
. "$DEFAULT"
fi
if [ ! -f "$CATALINA_HOME/bin/bootstrap.jar" ]; then
log_failure_msg "$NAME is not installed"
exit 1
fi
POLICY_CACHE="$CATALINA_BASE/work/catalina.policy"
if [ -z "$CATALINA_TMPDIR" ]; then
CATALINA_TMPDIR="$JVM_TMP"
fi
# Set the JSP compiler if set in the tomcat7.default file
if [ -n "$JSP_COMPILER" ]; then
JAVA_OPTS="$JAVA_OPTS -Dbuild.compiler=\"$JSP_COMPILER\""
fi
SECURITY=""
if [ "$TOMCAT7_SECURITY" = "yes" ]; then
SECURITY="-security"
fi
# Define other required variables
CATALINA_PID="/var/run/$NAME.pid"
CATALINA_SH="$CATALINA_HOME/bin/catalina.sh"
# Look for Java Secure Sockets Extension (JSSE) JARs
if [ -z "${JSSE_HOME}" -a -r "${JAVA_HOME}/jre/lib/jsse.jar" ]; then
JSSE_HOME="${JAVA_HOME}/jre/"
fi
catalina_sh() {
# Escape any double quotes in the value of JAVA_OPTS
JAVA_OPTS="$(echo $JAVA_OPTS | sed 's/\"/\\\"/g')"
AUTHBIND_COMMAND=""
if [ "$AUTHBIND" = "yes" -a "$1" = "start" ]; then
AUTHBIND_COMMAND="/usr/bin/authbind --deep /bin/bash -c "
fi
# Define the command to run Tomcat's catalina.sh as a daemon
# set -a tells sh to export assigned variables to spawned shells.
TOMCAT_SH="set -a; JAVA_HOME=\"$JAVA_HOME\"; source \"$DEFAULT\"; \
CATALINA_HOME=\"$CATALINA_HOME\"; \
CATALINA_BASE=\"$CATALINA_BASE\"; \
JAVA_OPTS=\"$JAVA_OPTS\"; \
CATALINA_PID=\"$CATALINA_PID\"; \
CATALINA_TMPDIR=\"$CATALINA_TMPDIR\"; \
LANG=\"$LANG\"; JSSE_HOME=\"$JSSE_HOME\"; \
cd \"$CATALINA_BASE\"; \
\"$CATALINA_SH\" $#"
if [ "$AUTHBIND" = "yes" -a "$1" = "start" ]; then
TOMCAT_SH="'$TOMCAT_SH'"
fi
# Run the catalina.sh script as a daemon
set +e
touch "$CATALINA_PID" "$CATALINA_BASE"/logs/catalina.out
chown $TOMCAT7_USER "$CATALINA_PID" "$CATALINA_BASE"/logs/catalina.out
start-stop-daemon --start -b -u "$TOMCAT7_USER" -g "$TOMCAT7_GROUP" \
-c "$TOMCAT7_USER" -d "$CATALINA_TMPDIR" -p "$CATALINA_PID" \
-x /bin/bash -- -c "$AUTHBIND_COMMAND $TOMCAT_SH"
status="$?"
set +a -e
return $status
}
case "$1" in
start)
if [ -z "$JAVA_HOME" ]; then
log_failure_msg "no JDK or JRE found - please set JAVA_HOME"
exit 1
fi
if [ ! -d "$CATALINA_BASE/conf" ]; then
log_failure_msg "invalid CATALINA_BASE: $CATALINA_BASE"
exit 1
fi
log_daemon_msg "Starting $DESC" "$NAME"
if start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT7_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null; then
# Regenerate POLICY_CACHE file
umask 022
echo "// AUTO-GENERATED FILE from /etc/tomcat7/policy.d/" \
> "$POLICY_CACHE"
echo "" >> "$POLICY_CACHE"
cat $CATALINA_BASE/conf/policy.d/*.policy \
>> "$POLICY_CACHE"
# Remove / recreate JVM_TMP directory
rm -rf "$JVM_TMP"
mkdir -p "$JVM_TMP" || {
log_failure_msg "could not create JVM temporary directory"
exit 1
}
chown $TOMCAT7_USER "$JVM_TMP"
catalina_sh start $SECURITY
sleep 5
if start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT7_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null; then
if [ -f "$CATALINA_PID" ]; then
rm -f "$CATALINA_PID"
fi
log_end_msg 1
else
log_end_msg 0
fi
else
log_progress_msg "(already running)"
log_end_msg 0
fi
;;
stop)
log_daemon_msg "Stopping $DESC" "$NAME"
set +e
if [ -f "$CATALINA_PID" ]; then
start-stop-daemon --stop --pidfile "$CATALINA_PID" \
--user "$TOMCAT7_USER" \
--retry=TERM/20/KILL/5 >/dev/null
if [ $? -eq 1 ]; then
log_progress_msg "$DESC is not running but pid file exists, cleaning up"
elif [ $? -eq 3 ]; then
PID="`cat $CATALINA_PID`"
log_failure_msg "Failed to stop $NAME (pid $PID)"
exit 1
fi
rm -f "$CATALINA_PID"
rm -rf "$JVM_TMP"
else
log_progress_msg "(not running)"
fi
log_end_msg 0
set -e
;;
status)
set +e
start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT7_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null 2>&1
if [ "$?" = "0" ]; then
if [ -f "$CATALINA_PID" ]; then
log_success_msg "$DESC is not running, but pid file exists."
exit 1
else
log_success_msg "$DESC is not running."
exit 3
fi
else
log_success_msg "$DESC is running with pid `cat $CATALINA_PID`"
fi
set -e
;;
restart|force-reload)
if [ -f "$CATALINA_PID" ]; then
$0 stop
sleep 1
fi
$0 start
;;
try-restart)
if start-stop-daemon --test --start --pidfile "$CATALINA_PID" \
--user $TOMCAT7_USER --exec "$JAVA_HOME/bin/java" \
>/dev/null; then
$0 start
fi
;;
*)
log_success_msg "Usage: $0 {start|stop|restart|try-restart|force-reload|status}"
exit 1
;;
esac
exit 0

The library was well placed.
It was a lack of permissions:
chmod 777 mysql-connector-java-5.1.25.jar

Very likely /var/lib/tomcat7/common/lib is not on the classpath. Check your Tomcat startup script. Have you try adding it to /var/lib/tomcat7/ assuming this is the location that tomcat load its base lib.

Spark: Could not load native gpl library

I had the following error when trying to run a very simple spark job (which uses logistic regression with SGD in mllib):
ERROR GPLNativeCodeLoader: Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1028)
at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32)
at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1659)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1624)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:155)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:187)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:181)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:93)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
14/08/06 20:32:11 ERROR LzoCodec: Cannot load native-lzo without native-hadoop
This is the command I used to submit the job:
~/spark/spark-1.0.0-bin-hadoop2/bin/spark-submit \
--class com.jk.sparktest.Test \
--master yarn-cluster \
--num-executors 40 \
~/sparktest-0.0.1-SNAPSHOT-jar-with-dependencies.jar
The actual java command is :
/usr/java/latest/bin/java -cp /apache/hadoop/share/hadoop/common/hadoop-common-2.2.0.2.0.6.0-61.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar::/home/jilei/spark/spark-1.0.0-bin-hadoop2/conf:/home/jilei/spark/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar:/home/jilei/spark/spark-1.0.0-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar:/home/jilei/spark/spark-1.0.0-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/home/jilei/spark/spark-1.0.0-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar:/apache/hadoop/conf:/apache/hadoop/conf \
-XX:MaxPermSize=128m \
-Djava.library.path=
-Xms512m -Xmx512m org.apache.spark.deploy.SparkSubmit \
--class com.jk.sparktest.Test \
--master yarn-cluster \
--num-executors 40 \
~/sparktest-0.0.1-SNAPSHOT-jar-with-dependencies.jar
Seems the -Djava.library.path is not set. I also tried the java command above and supplied the native lib directory to the java.library.path, but still got the same errors. This is the content of the native lib directory:
ls /apache/hadoop/lib/native/
libgplcompression.a libgplcompression.so libgplcompression.so.0.0.0 libhadooppipes.a
libhadoop.so.1.0.0 libhdfs.a libhdfs.so.0.0.0 libsnappy.so.1
libgplcompression.la libgplcompression.so.0 libhadoop.a libhadoop.so
libhadooputils.a libhdfs.so libsnappy.so libsnappy.so.1.1.4
Any idea on what's wrong? Thanks.

I don't know why what you tried with java.library.path didn't work. I solved a problem similar to what you described by adding the argument:
--driver-library-path /usr/lib/hadoop-lzo/lib/native/
where /usr/lib/hadoop-lzo/lib/native/ contained libgplcompression.so.