I am trying to run Spark locally in my Intellij and passing in an external jar with sparkConf:
sparkConf.set("spark.jars", "C:\\path\\to\\my\\jar.jar");
I see in the console when running that the jar has been added:
INFO org.apache.spark.SparkContext:57 - Added JAR C:\\path\\to\\my\\jar.jar at spark://some spark path
I also see the jar in Spark UI underneath Classpath Entries:
some-spark-path Added By User
However I am running into two issues.
When I tried to call upon a resource file in the external jar via code below, I get a null for the Inputstream, path would be like "/file/i/need.json":
try(InputStream is= SomeClass.class.getResourceAsStream(path))
When I am also passing a className (this class exists in the external jar) as string to a method which then creates an instance, I get ClassNotFoundException. ClassName would be like "class.i.need":
Class<?> classes = Class.forName(className);
Any advice would be helpful. Thanks!
Edit: Running on Spark 3.0.0.
I have also tried with "spark.driver.extraClassPath" to no avail.
Related
I'm running a server in a docker image openjdk:8-jdk-slim.
Inside this server, using spring boot, I receive from the frontend a protobuf file, compile it and add the generated class to the classpath in runtime. I then proceed to use java reflection to get the added class, with the method Class.forName().
Outside of docker, when running in my pc, it works like a charm. However, when I run in docker when I try to access using the method Class.forName(), with the same class name I use when not in docker, it doesn't find the class.
Is there something that I am missing because it is docker?
Edit for more information:
I'm running a maven build inside docker. What I have in this specific operation is an endpoint that receives a byte array, which corresponds to a protobuf file. I "construct" a file with this byte array, save it in a specific folder. I proceed to compile it with the protoc compiler, which generates a .java file, which I save in a specific folder. This folder corresponds to a package, which we will call "xxx.yyy.zzz".
When I try to add the class to the classpath, using this code -
File newClass = new File(relativePath);
URL url = newClass.toURI().toURL();
URLClassLoader classLoader = (URLClassLoader)ClassLoader.getSystemClassLoader();
Method newMethod = URLClassLoader.class.getDeclaredMethod("addURL", URL.class);
newMethod.setAccessible(true);
newMethod.invoke(classLoader, url);
I provide the relativePath to the .class file but it isn't added to the classpath. I know that the .class file is generated in the correct way and is saved in the specific folder.
I uploaded a Spring application to Heroku but the application crashed with the following error:
java.io.FileNotFoundException: class path resource [com/myname/myapp/config/dao-context.xml
The file is definitely there, and it is in GIT, and the app runs successfully locally.
Any ideas what is happening here?
I suspect that when you are running locally, it is picking up the file on the classpath as a regular file on the filesystem (i.e. not inside of a JAR).
On Heroku, it is probably inside of a JAR file, which means it is not a regular file, and must be read as an input stream, which might look like this:
ClassLoader cl = this.getClass().getClassLoader();
InputStream inputStream = cl.getResourceAsStream("com/myname/myapp/config/dao-context.xml");
You can probably reproduce the problem locally by running the same command that's in your Procfile.
If this is not the case, then make sure the file exists on Heroku by running this command:
$ heroku run ls com/myname/myapp/config/dao-context.xml
For future visitors to this question, I overcame the problem by converting my DAO XML config file to the Java Config method, therefore Spring no longer required that XML file. This didn't directly solve the issue of being unable to find the XML file, but the added benefit is that I am now using the more modern and less verbose Java Config method.
Iam using spring boot org.springframework.boot.loader.JarLauncher to run my self executable spring boot app jar.
Now, I have my self executable jar packed like this (For brevity Iam just adding the files only that are needed to show the problem):
Main.jar
---META-INF
---lib
---a.jar
---com
---comp
---Abc.class
---folder1
---1.txt
---2.txt
---b.jar
and so on.
In my Abc.class Iam trying to load the resource 1.txt which is working fine when I run it in eclipse; but the story starts when I run using command line as self executable jar. I was not able to read that resource and ends up with null pointer error.
After reading few threads, I learnt that my IDEs does some magic on Class Loaders and so it was working fine, which will not be the case when I run it as executable jar
This is how I was Loading the files and the all the possible options I have tried to no luck.
Option 1:
Abc.class.getResourceAsStream("\folder1\1.txt")
Option 2:
Abc.class.getClassLoader().getResourceAsStream("folder1\1.txt")
Option 3: Reading thread, I have tried considered the current thread class loader context as below too, But to no luck again
Thread.currentThread().getContextClassLoader().getResourceAsStream("folder1\1.txt")
Can any one advise. How should I write my code to be able to read my resource that is in the jar and by the class that is in the same jar ?
PS: Spring boot is not adding class-path entry to MANIFEST.MF and if I have to do something around that, how do I do that ? In-fact I tried -cp when running my jar setting it to current directory, but that have not worked either
In Spring, the best way to access a Resource is via Resource APIs. For a Classpath resource what you should use a ClassPathResource and it would look something like this:
Resource resource = new ClassPathResource("/my/resource.json", this.getClass());
After obtaining a Resource you can either get a File reference via getFile() or get an InputStream straight off by calling getInputStream().
Spring provides quite a few different implementations of the Resource interface, take a look at the list of known implementations in the docs
Use Spring's class ClassPathResource to load file from classpath.
For example you can read file into String like this from classpath:
String fileContent = FileUtils.readFileToString(
new ClassPathResource("folder1" + File.separator + "1.txt").getFile());
It doesn't matter what kind of build tool or project structure you are using as long as that file is on classpath.
In Eclipse Luna I have a Dynamic Web Project (with default build settings) on Apache Tomcat 8.0. In the project, I have a JSP loader.jsp that calls a method in a Java class FileLoader that returns a String value for the location of a non-config txt file (lets call it key.txt).
My issue is that I am getting a null value for the file location when FileLoader tries to get key.txt and return its location.
Here is an idea of what I've tried and failed with:
In FileLoader constructor I pass it String "key.txt" as a value and it has a method called get getKeyPath() that returns the file path. I use the following in getKeyPath() to get the file path:
String path = this.getClass().getClassLoader().getResource("key.txt").getFile();
The path variable is returned to the calling object. Here is how I call getKeyPath() in loader.jsp:
String keyPath = new FileLoader("key.txt").getKeyPath();
My issue is that a NullPointerException is thrown in FileLoader when getKeyPath() tries to set the path value. I am lost because this happens no matter where I put the physical "key.txt" in my project directory or at file paths that should be recognized by my project. In my project I have tried the above code with key.txt at the following paths (assume the root folder for my project in Eclipse is called PRO):
PRO/build/classes
PRO/build
PRO/WebContent/WEB-INF/lib
PRO/WebContent(where loader.jsp is located)
PRO/Java Resources/src (Tomcat Install)
Dir>/lib /bin
I got the same NullPointerException for all attempts. Once I resorted to the Tomcat directories I realized I needed help.
Is there something else I need to do so "key.txt" can be loaded in the way I want? Am I doing this completely wrong? I can post screenshots if that would make answering this easier.
As an aside, due to application requirements, I'd prefer that loader.jsp not load key.txt directly. Of course, I'll have to do that if what I'm trying is not possible.
Please note that I am trying to do this in an Eclipse Dynamic Web Project so there is no "bin" folder like a standard Java project. Finally, I'd prefer not to make any build config changes in Eclipse but of course I will if I have to.
Separate out the initialization and calling of the method into two separate lines.
Problem is not that file is not loaded but rather that at the time .getKeyPath(); is called object initisation is not done and that the reason for NullPointer
So change to this :
FileLoader FileLoader fileLoader = new FileLoader("key.txt");
String keyPath = fileLoader.getKeyPath();
I'm not able to give this question an apt title so apology for that.
I am making a modularised application. I load various jar files at runtime and invoke particular method of a particular class (of the jar file) at run time.
The jar file has some supported file. Now my jar file uses another application , lets say abc which is located in the same directory where i have kept the jar file. When i run the jar file then
new File(".").getAbsolutePath()
gives the correct path (this is where abc is located) and program runs fine. But when i load this jar file dynamically and invoke method using reflection above code gives the path of the parent program and abc is not found at that path.
Now my question is how do i find the path in which my jar file exists when i'm running my jar file's code using reflection.
Please let me know if you need more explanation.
Try something like this:
public static void main(String[] args) {
System.out.println(StringUtils.class.getResource("StringUtils.class"));
}
(Note: StringUtils is present on my classpath as a Maven dependency at the time) This gives me:
jar:file:/home/******/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar!/org/apache/commons/lang3/StringUtils.class
Since the class is in a JAR file, it also gives me the location of the class file within the JAR.