For context, I'm trying to configure my Spark session to use fair scheduling. For that reason I have a file, fairscheduler.xml, at the 'root' of my resources folder. This file is correctly packaging into my build folder in the location I'd expect.
To further prove that's working, I have this line:
val mypath = getClass.getResource("/fairscheduler.xml")
That is working and returns exactly the path I'd expect: /<some path>/<my jar>/fairscheduler.xml
The following is throwing a FileNotFound exception:
sessionBuilder
...
.config("spark.scheduler.mode", "FAIR")
.config("spark.scheduler.allocation.file", mypath.toString) <- THIS LINE
.config("spark.scheduler.pool", "fair_pool")
...
.getOrCreate
Just for sanity, I have logged out mypath.toString.
Why is it that the spark session builder can't recognize a resource file that otherwise seems valid? Does it expect "spark configuration" files to exist somewhere specific? Or am I completely missing some dumb little thing here?
This is not a Apache Spark limitation but an expected behaviour of how resource files are handled in Java JAR files. Question How to get a path to a resource in a Java JAR file provides more details and a workaround of persisting the content in a temp file.
Related
I uploaded a Spring application to Heroku but the application crashed with the following error:
java.io.FileNotFoundException: class path resource [com/myname/myapp/config/dao-context.xml
The file is definitely there, and it is in GIT, and the app runs successfully locally.
Any ideas what is happening here?
I suspect that when you are running locally, it is picking up the file on the classpath as a regular file on the filesystem (i.e. not inside of a JAR).
On Heroku, it is probably inside of a JAR file, which means it is not a regular file, and must be read as an input stream, which might look like this:
ClassLoader cl = this.getClass().getClassLoader();
InputStream inputStream = cl.getResourceAsStream("com/myname/myapp/config/dao-context.xml");
You can probably reproduce the problem locally by running the same command that's in your Procfile.
If this is not the case, then make sure the file exists on Heroku by running this command:
$ heroku run ls com/myname/myapp/config/dao-context.xml
For future visitors to this question, I overcame the problem by converting my DAO XML config file to the Java Config method, therefore Spring no longer required that XML file. This didn't directly solve the issue of being unable to find the XML file, but the added benefit is that I am now using the more modern and less verbose Java Config method.
Iam using spring boot org.springframework.boot.loader.JarLauncher to run my self executable spring boot app jar.
Now, I have my self executable jar packed like this (For brevity Iam just adding the files only that are needed to show the problem):
Main.jar
---META-INF
---lib
---a.jar
---com
---comp
---Abc.class
---folder1
---1.txt
---2.txt
---b.jar
and so on.
In my Abc.class Iam trying to load the resource 1.txt which is working fine when I run it in eclipse; but the story starts when I run using command line as self executable jar. I was not able to read that resource and ends up with null pointer error.
After reading few threads, I learnt that my IDEs does some magic on Class Loaders and so it was working fine, which will not be the case when I run it as executable jar
This is how I was Loading the files and the all the possible options I have tried to no luck.
Option 1:
Abc.class.getResourceAsStream("\folder1\1.txt")
Option 2:
Abc.class.getClassLoader().getResourceAsStream("folder1\1.txt")
Option 3: Reading thread, I have tried considered the current thread class loader context as below too, But to no luck again
Thread.currentThread().getContextClassLoader().getResourceAsStream("folder1\1.txt")
Can any one advise. How should I write my code to be able to read my resource that is in the jar and by the class that is in the same jar ?
PS: Spring boot is not adding class-path entry to MANIFEST.MF and if I have to do something around that, how do I do that ? In-fact I tried -cp when running my jar setting it to current directory, but that have not worked either
In Spring, the best way to access a Resource is via Resource APIs. For a Classpath resource what you should use a ClassPathResource and it would look something like this:
Resource resource = new ClassPathResource("/my/resource.json", this.getClass());
After obtaining a Resource you can either get a File reference via getFile() or get an InputStream straight off by calling getInputStream().
Spring provides quite a few different implementations of the Resource interface, take a look at the list of known implementations in the docs
Use Spring's class ClassPathResource to load file from classpath.
For example you can read file into String like this from classpath:
String fileContent = FileUtils.readFileToString(
new ClassPathResource("folder1" + File.separator + "1.txt").getFile());
It doesn't matter what kind of build tool or project structure you are using as long as that file is on classpath.
I'm having trouble with a legacy Web Application that I'm migrating to Maven3.
I need to obtain a file from the Classpath that in the directory structure is located in:
/src/main/resources/com/thinkglish/geoip/GeoIP.dat
When I create the .war file with the Maven build, I can confirm that this .dat file is located (as it should be) in:
WEB-INF/classes/com/thinkglish/geoip/GeoIP.dat
I'm trying two different approaches to get the resource from one of my classes, which implements javax.servlet.Filter:
ClassPathResource resource = new ClassPathResource("com/thinkglish/geoip/GeoIp.dat");
and
URL resource = getClass().getResource("/com/thinkglish/geoip/GeoIp.dat");
If I start the application using Maven's Jetty plugin, that works fine in both ways. However, when I deploy the application in a Tomcat and start the server, the resource cannot be located.
In the first case I get a java.io.FileNotFoundException: class path resource [com/thinkglish/geoip/GeoIp.dat] cannot be resolved to URL because it does not exist and in the second case the resource is null.
A curious thing about all this is that if I use one method or the other trying to obtain another resource from the Classpath (e.g. com/thinkglish/struts/i18n/MessageResources.properties or com/thinkglish/filter/LanguageFilter.class) it works without any problems.
Do you have any guess about this? Is it possible that the .dat extension has anything to do with this?
Edited - More data!
I added a new .properties mock file to the exact same directory in which the .dat file lives:
/src/main/resources/com/thinkglish/geoip/mock.properties
I tried to obtain it in Tomcat6 and it worked!
ClassPathResource resource = new ClassPathResource("com/thinkglish/geoip/mock.properties");
I'm starting to think that I need to do something else configuration-wise to make Tomcat6 accept the .dat file as a Classpath resource.
Thanks in advance!
I might be barking up completely the wrong tree here... but have you checked the capitalisation of GeoIP.dat / GeoIp.dat? Is Tomcat running on a case-sensitive OS?
Following should work:
String classpathLocation = "com/thinkglish/geoip/GeoIp.dat";
URL classpathResource = Thread.currentThread().getContextClassLoader().getResource(classpathLocation);
// Or:
InputStream input = Thread.currentThread().getContextClassLoader().getResourceAsStream(classpathLocation);
I am trying to use ESAPI.jar for providing security to my web application.Basically I have just started using ESAPI.jar.
But problem is I am not able to run even a simple program using ESAPI.
The small code snippet is:
String clean = ESAPI.encoder().canonicalize("someString");
Randomizer r=ESAPI.randomizer();
System.out.println(r);
System.out.println(clean);
I get this error:
Attempting to load ESAPI.properties via file I/O.
Attempting to load ESAPI.properties as resource file via file I/O.
Not found in 'org.owasp.esapi.resources' directory or file not readable: D:\Eclipse-Workspace\Test\ESAPI.properties
Not found in SystemResource Directory/resourceDirectory: .esapi\ESAPI.properties
Not found in 'user.home' (C:\Documents and Settings\user.user) directory: C:\Documents and Settings\user.user\esapi\ESAPI.properties
Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoundException
Attempting to load ESAPI.properties via the classpath.
ESAPI.properties could not be loaded by any means. Fail. Exception was: java.lang.IllegalArgumentException: Failed to load ESAPI.properties as a classloader resource.
Exception in thread "main" org.owasp.esapi.errors.ConfigurationException: java.lang.reflect.InvocationTargetException SecurityConfiguration class (org.owasp.esapi.reference.DefaultSecurityConfiguration) CTOR threw exception.
at org.owasp.esapi.util.ObjFactory.make(ObjFactory.java:129)
at org.owasp.esapi.ESAPI.securityConfiguration(ESAPI.java:184)
at org.owasp.esapi.ESAPI.encoder(ESAPI.java:99)
at org.rancore.testJasp.TestEsapi.main(TestEsapi.java:59)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.owasp.esapi.util.ObjFactory.make(ObjFactory.java:86)
... 3 more
Caused by: org.owasp.esapi.errors.ConfigurationException: ESAPI.properties could not be loaded by any means. Fail.
at org.owasp.esapi.reference.DefaultSecurityConfiguration.loadConfiguration(DefaultSecurityConfiguration.java:439)
at org.owasp.esapi.reference.DefaultSecurityConfiguration.<init>(DefaultSecurityConfiguration.java:227)
at org.owasp.esapi.reference.DefaultSecurityConfiguration.getInstance(DefaultSecurityConfiguration.java:75)
... 8 more
Caused by: java.lang.IllegalArgumentException: Failed to load ESAPI.properties as a classloader resource.
at org.owasp.esapi.reference.DefaultSecurityConfiguration.loadConfigurationFromClasspath(DefaultSecurityConfiguration.java:667)
at org.owasp.esapi.reference.DefaultSecurityConfiguration.loadConfiguration(DefaultSecurityConfiguration.java:436)
... 10 more
I have tried copying the 3 ESAPI properties files in my source folder and also configuring them on build path but still I have not succeeded. I have tried many permutations and combinations to no avail.
Please guide me.
The content of property file is:
# User Messages
Error.creating.randomizer=Error creating randomizer
This.is.test.message=This {0} is {1} a test {2} message
# Validation Messages
# Log Messages
The ESAPI.properties file should have more than 3 lines in it. See for example:
https://web.archive.org/web/20150904064147/http://code.google.com:80/p/owasp-esapi-java/source/browse/trunk/configuration/esapi/ESAPI.properties
In my experience the ESAPI.properties file either needs to be in the same folder as the esapi.jar or needs to be compiled into the jar in a resources directory.
/resources/ESAPI.properties
I believe that either one should work. If ESAPI does not find the file it one location it looks in others.
The code for that is here around line 620:
https://web.archive.org/web/20161005210258/http://code.google.com/p/owasp-esapi-java/source/browse/trunk/src/main/java/org/owasp/esapi/reference/DefaultSecurityConfiguration.java
I just struggled through this one myself.
I created a folder called esapi in my C:/users/myname/ directory and loaded up the ESAPI.properties, validation.properties, and the ESAPI-AccessControlPolicy.xml which got me past all the not finding files errors. ESAPI looks in several places for the files. I'm running Windows 7 64 bit by the way.
Then I had to update some of the jars. I should have done this from the beginning, but I didn't know it. I was getting this error:
AccessController class (org.owasp.esapi.reference.DefaultAccessController) CTOR threw exception.
for every jar that needed to be a newer version, like commons collections or log4j. At the bottom of the stack trace it would reference the offending jar.
When I added the newer jars from the esapi lib directory everything just worked!
I got this message at the end in my console:
ESAPI.accessController found: org.owasp.esapi.reference.DefaultAccessController#1cb8deef
Note: There is an ESAPI_en_US.properties file, but it's the one with only a few lines in it. Just use the file in: \esapi-2.1.0-dist\src\test\resources\esapi\ESAPI.properties. That is the full complete file.
After having this problem and looking at the installation documentation (esapi-x.x.x-dist\documentation\esapi4java-core-x-x-install-guide.pdf) I found a very useful section which detailed that the properties file can be anywhere, provided a vm flag (-Dorg.owasp.esapi.resources=path") points to a particular directory.
For example, if I stick the file in a "resources" folder at the root of my project directory, then the flag would be:
-Dorg.owasp.esapi.resources="path\to\project\root\resources"
The reference SecurityConfiguration manages all the settings used by the ESAPI in a single place. In this reference implementation, resources can be put in several locations, which are searched in the following order:
1) Inside a directory set with a call to SecurityConfiguration.setResourceDirectory( "C:\temp\resources" ).
2) Inside the System.getProperty( "org.owasp.esapi.resources" ) directory. You can set this on the java command line as follows (for example):
java -Dorg.owasp.esapi.resources="C:\temp\resources"
You may have to add this to the start-up script that starts your web server. For example, for Tomcat, in the "catalina" script that starts Tomcat, you can set the JAVA_OPTS variable to the -D string above.
3) Inside the System.getProperty( "user.home" ) + "/.esapi" directory (supported for backward compatibility) or inside the System.getProperty( "user.home" ) + "/esapi" directory.
4) The first ".esapi" or "esapi" directory on the classpath. (The former for backward compatibility.)
Can you put your file (with this name) in:
D:\Eclipse-Workspace\Test\ESAPI.properties
And show us the contents and exception again.
Extract the esapi jar
create a folder named resources under org.owasp.esapi
copy ESAPI.properties to the org.owasp.esapi.resources
Build and deploy
extract the jar
add properties file under resources folder.
initially getting same error, after updating properties file it worked for me
Thanks for providing the information
-Dorg.owasp.esapi.resources="path\to\project\root\resources"
This is a good source of information and has resolved my issue
I had the same problem too. I resolved it using a little bit of James Drinkard solution. What I basically did is created a new folder with name ESAPI and added ESAPI.properties file, Validation.properties, and the ESAPI-AccessControlPolicy.xml. And archived it into a jar file and added to the lib folder under WebContet/WEB-INF and build it to the path and it worked.
*To archive it into a jar file I used windows command move ESAPI ESAPI.jar
Look at the various scripts under 'src/examples/scripts' and they will show you one easy way to control where the ESAPI.properties files is found. (This is for ESAPI 2.0 or later.)
You will find a copy of the ESAPI.properties file will be under 'configuration/esapi'.
The project that I am currently working on utilizes an old application contained within a .jar file. One of the responsibilities of this application is that it updates the database when changes to the configuration files are made. Every time I try to run this file (which is a simple Ant Task extension), I get an exception thrown early in the process. I decompiled the Java file responsible, and found that the exception being thrown happens here. I do not know what the issue is, as "hibernate.cfg.xml" is contained within the same .jar file as the .class throwing the exception.
ClassLoader loader = Thread.currentThread().getContextClassLoader();
InputStream in = loader.getResourceAsStream("hibernate.cfg.xml");
if (in == null) {
throw new RuntimeException("Couldn't find built in hibernate config");
}
If anyone has any ideas, even pointing me in the right direction, I would be grateful.
Of course, any solution will have to be external, as the client already has this build of the program in production use.
Are you loading it from the root dir and you need "/hibernate.cfg.xml" instead of just hibernate.cfg.xml?
getResourceAsStream() expects the file to be in the same package as the Class that was the origin of the ClassLoader. Is your file in the right package?
Doh. Didn't read the question fully. Please ignore this.
try
InputStream in = YourClass.class..getResourceAsStream("hibernate.cfg.xml");
this will work if the class is in the same jar as the cfg file.
edit:
if you can't rebuild the application, you will need to add the jar to the bootstrap classloader. this is very ugly.
you can do it by running the jvm with (play with the exact arguments, you may need to add rt.jar from your jre to it as well).
-Xbootclasspath your.jar
your problem is that the code is using the classloader that loaded the Thread class, which is most likely the bootstrap classloader. and that you are now in a more complex environment (app server?) that loads your jar using a different classloader. so the bootstrap classloader can't find your resource.
It is possible that the classloader cannot open files contained in the jar file. Try one of two things 1) try extracting the jar file and running it from the file system, or 2) if the jar manifest has a ClassPath entry, try extracting the hibernate.cfg.xml into a directory in the classpath and see if it runs.
Apparently the hibernate.cfg.xml file isn't located in the source root of the JAR file, but it is instead placed inside a package. You'll need to specify the package path in the resource name as well.