FileNotFound Exception when trying to spark submit - java

I have a spark jar which I am trying to submit to my local spark instance. The jar is packages such that all config files are present in the /resources folder as per the maven structure.
Below is the exception:
java.io.FileNotFoundException: file:/Users/prime/Desktop/TibcoMsgConsumer-1.0-SNAPSHOT.jar!/config.properties (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
One observable thing in the above exception is the ! mark present at the end of the jar: TibcoMsgConsumer-1.0-SNAPSHOT.jar!/
If I remove the spark relevant parts (Java Spark context) from this jar and make it a normal Java Jar, I do not face any issue. What might be wrong here?
I decompiled the class and here is how I load the file:
ClassLoader classLoader = getClass().getClassLoader();
File file = new File(this.classLoader.getResource("config.properties").getFile());
private Properties properties = this.propLoader.initProp(this.file);
Here is how I run the job:
/spark-submit --conf spark.shuffle.consolidateFiles=true --verbose --class "ril.bigdata.com.Main" --master local[*] ~/Desktop/TibcoMsgConsumer-1.0-SNAPSHOT.jar
What am I missing?

Whatever comes after ! is inside the jar. It is not looking for config.properties inside resources, it's searching in the root of your JAR.
File file = new File(this.classLoader.getResource("resources/config.properties").getFile());
This should make it work.
Also, JavaDoc for getResource

Related

Gradle and src/main/resources

I have a gradle project which is working fine within eclipse. In the java code I have
Properties p = MyClass.class.getClassLoader.getResourceAsStream("myProps.properties");
File f = new File(p.getProperty("myFile"));
My structure is:
src/main/java - has java files
src/main/resources - has myProps.properties and myFile.txt
myProps.properties has:
myFile=src/main/resources/myFile.txt
If I do a gradle build (using a default shadowJar task) my resources have now been moved from src/main/resources to the root of the jar file.
shadowJar {
manifest{
// The only thing I modify is the attributes
}
}
When I execute the jar by running
"java -jar myJar.jar" I get FileNotFound exception referring to the myFile.txt resource file.
What is the correct way to fix this issue so that the code works from both the Eclipse and executable jar environments?
I needed to read the file as a stream (the same way I read in the properties file). I removed 'src/main/resources' from the path in the properties file and it worked

Java classLoader within a jar file

I work with the follwing project:
Test-Project
/lib/classA.jar
/src/main/java/org/test/classB.java
/src/main/resources/log.txt
In classA.jar, I work with the ClassLoader.getSystemClassLoader() to get the path of the "log.txt" file. In the project of classA.jar, there is no problem with that.
But when I use the jar in the Test-Project I get the follwing path to the "log.txt" file:
/lib/classA.jar!/log.txt
Is there a way to get the classloader from Test-Project into to the jarfile?
For accessing to a resource file you should use Class#getResourceAsStream for reading this file.
InputStream stream = Class.class.getResourceAsStream("/log.txt");
Note: a jar file is an archive, which is meant to be unchanged, if that is really a log file you shouldn't store it inside a jar file.

Java load properties file correctly

I have the following project structure:
ProjectName/src/java/com/main/Main.java
And I have a properties file in the following folder:
ProjectName/config/settings.properties
Now I tried to load the properties file in the Main.java with:
InputStream input = Main.class.getResourceAsStream("/config/settings.properties");
Properties prop = System.getProperties();
prop.load(input);
But this does not work. How is it the right way to do it?
Edit: I got the following error:
Exception in thread "main" java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Unknown Source)
at java.util.Properties.load0(Unknown Source)
at java.util.Properties.load(Unknown Source)
Edit2: Now it works with eclipse. I did the following (adapted from getResourceAsStream() is returning null. Properties file is not loading)
1) This directory [config] must be put into the "build path". Right-click the directory in the Package Explorer or Project Explorer view, select "Build Path", then "Use as Source Folder". Note: This build path will be the class path for the project, when you run it.
2) As the config directory now is part of your class path and contains your properties file, you can simply load it with InputStream input = Server.class.getClassLoader().getResourceAsStream("settings.properties");
This works well for eclipse but not yet for a jar. I think I also have to add the build path somehow to the ant script.
How can I do that?
Edit3: If I take the settings.properties out of the config folder in the jar, then it works. Why? I want it in the config folder in the jar too.
Use following :
InputStream input = Main.class.getResourceAsStream("../../../../config/settings.properties");
Properties prop = System.getProperties();
prop.load(input);
By adding ../ I am getting 1 step backward in your filesystem. In your code you were using /config....., which means C:/config..... (if it is in C drive).
The following will also work in your case. (but not in zipped file)
InputStream input = new FileInputStream("config/settings.properties");
Properties prop = System.getProperties();
prop.load(input);
getResourceAsStream find files relatively compiled (class) files on in jar file. Example:
Your compiled files are in bin directory than you should put properties file to bin/config/settings.properties for give access to it.
You are using maven. So follow the maven structure and keep your resources like prop, xml etc files in resource folder and then call.
So folder structure would be like this
src/main/resource|
|
config|
|
settings.properties
InputStream input = Main.class.getClass().getResourceAsStream("/config/settings.properties");
Three things
You need add the settings.properties to a jar (can be the same jar as your Main)
Your classpath should include this jar
You need to use the context Class Loader to load the properties file in you Main.
Classloader cl = Thread.currentThread().getContextClassLoader();
cl.getResourceAsStream("settings.properties");

Can't find resource file after exporting to a runnable JAR

I have a project structure that looks like this:
Tester
\-- src
|-- hello.java
\-- vocabulary.csv
I load the csv file using getResource:
url = this.getClass().getResource("vocabulary.csv");
System.out.println(url.getPath());
new FileReader(url.getPath());
The program works completly fine until I export the project to a runnable jar file. Even though the URL is still valid (he prints the path instead of null) the console shows this:
jar:file:/home/malte/Desktop/vocs.jar!/main/vocabulary.csv
Exception in thread "main" java.io.FileNotFoundException: file:/home/malte/Desktop/vocs.jar!/main/vocabulary.csv (No such file or directory)
Can you explain me why this happens and how I can solv the issue?
P.S. I am using Xtend instead of pure Java
You need to use
InputStream is = this.getClass().getResourceAsStream();
once you package your source as a jar. You have just one file : your jar. Your vocabulary.csv is no longer a standalone file on filesystem anymore.
You can read more here.

Reference jars inside a jar

I have a jar whose content looks as shown below,
Below is my manifest file
Manifest-Version: 1.0
Ant-Version: Apache Ant 1.8.3
Created-By: 1.7.0_06-b24 (Oracle Corporation)
Main-Class: org.sai.com.DerbyDemo
Class-Path: derby.jar derbyclient.jar derbynet.jar derbytools.jar
When i try to run the jar, it has thrown a ClassNotFoundExcception meaning it isn't referencing the jars inside the outer jar.
In the Class-Path attribute, how can I reference jars (derby.jar, etc) inside the actual jar?
You will need a custom class loader for this, have a look at One Jar.
One-JAR lets you package a Java application together with its dependency Jars into a single executable Jar file.
It has an ant task which can simplify the building of it as well.
REFERENCE (from background)
Most developers reasonably assume that putting a dependency Jar file into their own Jar file, and adding a Class-Path attribute to the META-INF/MANIFEST will do the trick:
jarname.jar
| /META-INF
| | MANIFEST.MF
| | Main-Class: com.mydomain.mypackage.Main
| | Class-Path: commons-logging.jar
| /com/mydomain/mypackage
| | Main.class
| commons-logging.jar
Unfortunately this is does not work. The Java Launcher$AppClassLoader does not know how to load classes from a Jar inside a Jar with this kind of Class-Path. Trying to use jar:file:jarname.jar!/commons-logging.jar also leads down a dead-end. This approach will only work if you install (i.e. scatter) the supporting Jar files into the directory where the jarname.jar file is installed.
You can't. From the official tutorial:
By using the Class-Path header in the manifest, you can avoid having
to specify a long -classpath flag when invoking Java to run the your
application.
Note: The Class-Path header points to classes or JAR files on the
local network, not JAR files within the JAR file or classes accessible
over internet protocols. To load classes in JAR files within a JAR
file into the class path, you must write custom code to load those
classes. For example, if MyJar.jar contains another JAR file called
MyUtils.jar, you cannot use the Class-Path header in MyJar.jar's
manifest to load classes in MyUtils.jar into the class path.
In Eclipse you have option to export executable jar.
You have an option to package all project related jars into generated jar and in this way eclipse add custom class loader which will refer to you integrated jars within new jar.
Default implementations of the classloader cannot load from a jar-within-a-jar: in order to do so, the entire 'sub-jar' would have to be loaded into memory, which defeats the random-access benefits of the jar format (reference pending - I'll make an edit once I find the documentation supporting this).
I recommend using a program such as JarSplice to bundle everything for you into one clean executable jar.
Edit: Couldn't find the source reference, but here's an un-resolved RFE off the Sun website describing this exact 'problem': http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4648386
Also, you could 'test' that your program works by placing the library jar files in a \lib sub-directory of your classes directory, then running from the command line. In other words, with the following directory structure:
classes/org/sai/com/DerbyDemo.class
classes/org/sai/com/OtherClassFiles.class
classes/lib/derby.jar
classes/lib/derbyclient.jar
From the command line, navigate to the above-mentioned 'classes' directory, and type:
java -cp .:lib/* org.sai.com.DerbyDemo
if you do not want to create a custom class loader. You can read the jar file stream. And transfer it to a File object. Then you can get the url of the File. Send it to the URLClassLoader, you can load the jar file as you want.
sample:
InputStream resourceAsStream = this.getClass().getClassLoader().getResourceAsStream("example"+ ".jar");
final File tempFile = File.createTempFile("temp", ".jar");
tempFile.deleteOnExit(); // you can delete the temp file or not
try (FileOutputStream out = new FileOutputStream(tempFile)) {
IOUtils.copy(resourceAsStream, out);
}
IOUtils.closeQuietly(resourceAsStream);
URL url = tempFile.toURI().toURL();
URLClassLoader urlClassLoader = new URLClassLoader(new URL[]{url});
urlClassLoader.loadClass()
...
Add the jar files to your library(if using netbeans) and modify your manifest's file classpath as follows:
Class-Path: lib/derby.jar lib/derbyclient.jar lib/derbynet.jar lib/derbytools.jar
a similar answer exists here
in eclipse, right click project, select RunAs -> Run Configuration and save your run configuration, this will be used when you next export as Runnable JARs

Categories

Resources