To optimise Docker layers, I am trying to split our 30M Spring Boot fat jar into the 2M app.jar and 28M libs.jar.
I can use exploded mode, but I prefer using 2 jars, because it simplifies a few things, such as deployments, scripts etc. In particular the fat jar is more easily and intuitively executed with java -jar, as opposed to the more cumbersome java org.springframework.boot.loader....Launcher.
My problem is that the moment I separate the libs out, I can't get the Launcher to find them. In either jar or exploded mode (with two dirs) - I keep getting
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/odoro/common/api/ServiceType
at com.odoro.sync.service.Application.main(Application.java:14)
... 6 more
Caused by: java.lang.ClassNotFoundException: com.odoro.common.api.ServiceType
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.springframework.boot.loader.LaunchedURLClassLoader.doLoadClass(LaunchedURLClassLoader.java:178)
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:142)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
I get this in all the following cases:
# java -jar app.jar -cp ../lib.jar
# java -cp .:../lib org.springframework.boot.loader.JarLauncher
# java -Dloader.path=../lib org.springframework.boot.loader.PropertiesLauncher
Any idea how I can get this to work?
It seems that the Spring Boot thin launcher is what you're looking for.
Related
For various experimentations, I take care of a java project in github.
After the Maven build, the program runs with a script bat.
Now I opened a branch because I would use the library args4j to parsing the arguments.
The build works fine, the jars exist in the directory lib, but when I run I have this stacktrace of Exception
Exception in thread "main" java.lang.NoClassDefFoundError:
org/kohsuke/args4j/CmdLineException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
at java.lang.Class.getMethod0(Class.java:2866)
at java.lang.Class.getMethod(Class.java:1676)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException:
org.kohsuke.args4j.CmdLineException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 6 more
in bat I configured the classpath so that the args4j jar in in lib: this are the instructions of bat script
SET JAVA_DIR=C:\Program Files\Java\jdk1.7.0_80\bin\
>CUT
"%JAVA_DIR%\java" -jar ".\lib\buildCSS-1.0.jar" -cp ".\lib\" -conf "./conf/environment.properties"
I don't understand the deal of java.lang.NoClassDefFoundError. The jar are present and linked by -cp option
Do you have any idea (and solution), please?
You cannot combine -jar and -cp arguments on the command line. If the java command sees -jar it treats everything after the jarfile name as application arguments, AND it ignores any earlier classpath arguments.
You have two choices:
Use -cp, include the main JAR in the classpath, and put the full class name for the main class on the command line.
Use -jar, and add a "Class-Path" attribute to the main JAR's manifest file listing all of the dependencies.
References:
The java command page - explains -jar versus -cp
The JAR file specification - explains the "Class-Path" attribute
Note: since you are building the JAR file using Maven, there are other options; for example
Use the "Shade" plugin to create an executable "uber-jar" containing all of the dependencies in a single JAR.
I am executing the java archive on Spark using spark-submit in Ubuntu. The command is given below. This JAR file was build using Maven Package. The dependencies are specified in pom.xml file.
]$ spark-submit --class HighScore.Driver --master local[*] JarfilePath/Levelwise_PCFS-0.0.1-SNAPSHOT.jar InputFilePath/K9_Site1.csv 1000.
I am getting following error even when packageName.className (HighScore.Driver) is specified in the command.
Here is the error message.
Exception in thread "main" java.lang.NoClassDefFoundError: com/github/lwhite1/tablesaw/api/Table
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:727)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.github.lwhite1.tablesaw.api.Table
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
com/github/lwhite1/tablesaw/api/Table dependency was also specified in pom.xml file. But still it throws the exception.
Can some one help me in rectifying this error.
Remember that the classloader (specifically java.net.URLClassLoader) will look for classes in package a.b.c in folder a/b/c/ in each entry in your classpath.
NoClassDefFoundError can also indicate that you're missing a transitive dependency of a .jar file that you've compiled against and you're trying to use.
For example, if you had a class com.example.Foo, after compiling you would have a class file Foo.class.
Say for example your working directory is .../project/. That class file must be placed in .../project/com/example, and you would set your classpath to .../project/.
Please refer this post for more details. It would be helpful.
I am trying to run a program on the command line that uses the JavaMail API. This program also reads in a message from a text file. However, I'm getting this error:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: javax/mail/Address
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: javax.mail.Address
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
Originally I used java -cp /path/to/mail.jar mailTest < message.txt
I configured the classpath for the jar successfully (as far as I know). Can anyone give me some pointers? :/
Your classpath is definitely wrong. The classpath needs to include both the javax.mail.jar file and the classes for your application. Exactly where is the JavaMail jar file, exactly where are your application classes, and exactly what java command line did you use? If your application classes are in the current directory, you need something like
java -cp /path/to/javax.mail.jar:. mailTest
Actually if you look at the error message:
Caused by: java.lang.ClassNotFoundException: javax.mail.Address
That's probably because your jar depends on another jar (probably mail-x.y.z.jar) and you did not include it on your classpath when trying to run your jar.
Either include the missing jar(s) on the command line, see this post, or add them to your manifest file, see the doc
I am trying to run a jar file that is using pi4j.
sudo java -classpath /opt/pi4j/lib/pi4j-core.jar -jar Test.jar test.Main
Problem is I keep getting this error
Exception in thread "main" java.lang.NoClassDefFoundError: com/pi4j/io/gpio/GpioFactory
at test.Main.main(Main.java:11)
Caused by: java.lang.ClassNotFoundException: com.pi4j.io.gpio.GpioFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
I don't see why the link isn't being made properly.
I have never done it in command line. But this worked for me:
in IntelliJ, I added the pi4J jar files to my project as modules. I then made an artifact and got IntelliJ to create a jar file for me (with all the dependencies including pi4J). Then I transferred the jar file to the pi and it ran fine.
I have maven based mixed scala/java application that can submit spar jobs. My application jar "myapp.jar" has some nested jars inside lib folder. one of which is "common.jar". I have defined class-path attribute in Manifest file like Class-Path: lib/common.jar. Spark executor throws java.lang.NoClassDefFoundError:com/myapp/common/myclass error when submitting application in yarn-client mode. Class(com/myapp/common/myclass.class) and jar(common.jar) is there and nested inside my main myapp.jar. Fat jar is created using spring-boot-maven plugin which nest other jars inside lib folder of parent jar. I prefer not to create shaded flat jar as that would create other issues. Anyway spark executor jvm can load nested jars here?
EDIT spark (jvm classloader) can find all the classes those are flat inside myapp.jar itself. i.e. com/myapp/abc.class, com/myapp/xyz.class etc.
EDIT2 spark executor classloader can also find some classes from nested jar but it throws NoClassDefFoundError some other classes in same nested jar!
here's the error:
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, host4.local): java.lang.NoClassDefFoundError: com/myapp/common/myclass
at com.myapp.UserProfileRDD$.parse(UserProfileRDDInit.scala:111)
at com.myapp.UserProfileRDDInit$$anonfun$generateUserProfileRDD$1.apply(UserProfileRDDInit.scala:87)
at com.myapp.UserProfileRDDInit$$anonfun$generateUserProfileRDD$1.applyUserProfileRDDInit.scala:87)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:249)
at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:172)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:79)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:242)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException:
com.myapp.common.myclass
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 14 more
I do submit myapp.jar with sparkConf.setJar(String[] {"myapp.jar"}) and also tried setting it on spark.yarn.executor.extraClassPath
EDIT 3
As a workaround, I extracted myapp.jar and set sparkConf.setJar(String[] {"myapp.jar","lib/common.jar"}) manually and error went away but obviously I have to do that for all the nested jar which is not desirable.
You can use --jars options, to give comma separated list of jars while starting the Spark Application.
Something like
spark-submit --jars lib/abc.jar,lib/xyz.jar --class <CLASSNAME> myapp.jar