Develop Reference

Loading files from classpath and .jar [duplicate]

This question already has answers here:
How to read text file from classpath in Java?
(17 answers)
How to load/reference a file as a File instance from the classpath
(4 answers)
Get a list of resources from classpath directory
(15 answers)
How to locate all resources in classpath with a specified name?
(3 answers)
How to list the files inside a JAR file?
(17 answers)
Closed 4 years ago.
I make a Spring Boot web application, and I have to read static (.json) files on startup.
No problem, it works when I start the app in Idea. But if I run mvn package, and start I start the jar, the given path (BASE_DIR) does not exists anymore, so the files cannot be found.
How can I solve to read the files starting with Idea and the .jar as well?
package skyxplore.dataaccess.gamedata.base;
import java.io.File;
import java.io.IOException;
import java.util.HashMap;
import org.apache.commons.io.FilenameUtils;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import skyxplore.dataaccess.gamedata.entity.abstractentity.GeneralDescription;
#Slf4j
public abstract class AbstractGameDataService<V> extends HashMap<String, V> {
public static final String BASE_DIR = "src/main/resources/data/gamedata/";
private static final ObjectMapper objectMapper = new ObjectMapper();
private static final JsonFileFilter jsonFilter = new JsonFileFilter();
private final String source;
public AbstractGameDataService(String source) {
this.source = BASE_DIR + source;
}
protected void loadFiles(Class<V> clazz) {
File root = new File(source);
if (!root.exists()) {
throw new IllegalStateException("Source directory does not exists. Path: " + root.getAbsolutePath());
}
if (!root.isDirectory()) {
throw new IllegalArgumentException("Source must be a directory. Path: " + root.getAbsolutePath());
}
File[] files = root.listFiles(jsonFilter);
for (File file : files) {
try {
String key = FilenameUtils.removeExtension(file.getName());
V content = objectMapper.readValue(file, clazz);
if (content instanceof GeneralDescription) {
GeneralDescription d = (GeneralDescription) content;
log.info("Loaded element. Key: {}, Value: {}", key, content);
put(d.getId(), content);
} else {
throw new RuntimeException(source + " cannot be loaded. Unknown data type.");
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
public abstract void init();
}

After you package your applications, resource files are no longer accessible from file system - thus you cannot list/open them using File. You need to read them as resources from JAR using class.readResourceAsStream for example. When you run it from IntelliJ files are not read from JAR but from project directory.
Here you have Oracles Java tutorial on how to use resources
Another solution (as you have some additional logic for reading files) you could put files aside JAR not as resource and read it from there then.

BASE_DIR = "src/main/resources/data/gamedata/"
Once you do a mvn package, src/main/resources folder becomes the root of the jar. You can use a class-loader to load this file. Something like:
AbstractGameDataService.class.getResourceAsStream("/data/gamedata/<your json file>")
Other possibility since you are using spring-boot, you can leverage spring scanning itself. Using spring constructs load file using : classpath:/data/gamedata/<your json file>

Add custom folders to classpath in bazel java tests

I'm trying to migrate a large codebase from maven to bazel and I've found that some of the tests write to target/classes and target/test-classes and the production code reads it as resources on the classpath. This is because maven surefire/failsafe run by default from the module directory and add target/classes and target/test-classes to the classpath.
For me to migrate this large codebase the only reasonable solution is to create target, target/classes and target/test-classes folders and add the last two to the classpath of the tests.
Any ideas on how this can be achieved?
Thanks

Another line of approach. Instead of generating a test suite, create a custom javaagent and a custom class loader. Use jvm_flags to setup and configure it.
The javaagent has a premain method. This sounds like a natural place to do things that happen before the regular main method, even if they don't have anything to do with class instrumentation, debugging, coverage gathering, or any other usual uses of javaagents.
The custom javaagent reads system property extra.dirs and creates directories specified there. It then reads property extra.link.path and creates the symbolic links as specified there, so I can place resources where the tests expect them, without having to copy them.
Classloader is needed so that we can amend the classpath at runtime without hacks. Great advantage is that this solution works on Java 10.
The custom classloader reads system property extra.class.path and (in effect) prepends it before what is in java.class.path.
Doing things this way means that standard bazel rules can be used.
BUILD
runtime_classgen_dirs = ":".join([
"target/classes",
"target/test-classes",
])
java_test(
...,
jvm_flags = [
# agent
"-javaagent:$(location //tools:test-agent_deploy.jar)",
"-Dextra.dirs=" + runtime_classgen_dirs,
# classloader
"-Djava.system.class.loader=ResourceJavaAgent",
"-Dextra.class.path=" + runtime_classgen_dirs,
],
,,,,
deps = [
# not runtime_deps, cause https://github.com/bazelbuild/bazel/issues/1566
"//tools:test-agent_deploy.jartest-agent_deploy.jar"
],
...,
)
tools/BUILD
java_binary(
name = "test-agent",
testonly = True,
srcs = ["ResourceJavaAgent.java"],
deploy_manifest_lines = ["Premain-Class: ResourceJavaAgent"],
main_class = "ResourceJavaAgent",
visibility = ["//visibility:public"],
)
tools/ResourceJavaAgent.java
import java.io.File;
import java.io.IOException;
import java.lang.instrument.Instrumentation;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;
// https://stackoverflow.com/questions/60764/how-should-i-load-jars-dynamically-at-runtime
public class ResourceJavaAgent extends URLClassLoader {
private final ClassLoader parent;
public ResourceJavaAgent(ClassLoader parent) throws MalformedURLException {
super(buildClassPath(), null);
this.parent = parent; // I need the parent as backup for SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
System.out.println("initializing url classloader");
}
private static URL[] buildClassPath() throws MalformedURLException {
final String JAVA_CLASS_PATH = "java.class.path";
final String EXTRA_CLASS_PATH = "extra.class.path";
List<String> paths = new LinkedList<>();
paths.addAll(Arrays.asList(System.getProperty(EXTRA_CLASS_PATH, "").split(File.pathSeparator)));
paths.addAll(Arrays.asList(System.getProperty(JAVA_CLASS_PATH, "").split(File.pathSeparator)));
URL[] urls = new URL[paths.size()];
for (int i = 0; i < paths.size(); i++) {
urls[i] = Paths.get(paths.get(i)).toUri().toURL(); // important only for resource url, really: this url must be absolute, to pass getClass().getResource("/users.properties").toURI()) with uri that isOpaque == false.
// System.out.println(urls[i]);
}
// this is for spawnVM functionality in tests
System.setProperty(JAVA_CLASS_PATH, System.getProperty(EXTRA_CLASS_PATH, "") + File.pathSeparator + System.getProperty(JAVA_CLASS_PATH));
return urls;
}
#Override
public Class<?> loadClass(String s) throws ClassNotFoundException {
try {
return super.loadClass(s);
} catch (ClassNotFoundException e) {
return parent.loadClass(s); // we search parent second, not first, as the default URLClassLoader would
}
}
private static void createRequestedDirs() {
for (String path : System.getProperty("extra.dirs", "").split(File.pathSeparator)) {
new File(path).mkdirs();
}
}
private static void createRequestedLinks() {
String linkPaths = System.getProperty("extra.link.path", null);
if (linkPaths == null) {
return;
}
for (String linkPath : linkPaths.split(",")) {
String[] fromTo = linkPath.split(":");
Path from = Paths.get(fromTo[0]);
Path to = Paths.get(fromTo[1]);
try {
Files.createSymbolicLink(from.toAbsolutePath(), to.toAbsolutePath());
} catch (IOException e) {
throw new IllegalArgumentException("Unable to create link " + linkPath, e);
}
}
}
public static void premain(String args, Instrumentation instrumentation) throws Exception {
createRequestedDirs();
createRequestedLinks();
}
}

If you could tell the tests where to write these files (in case target/classes and target/test-classes are hardcoded), and then turn the test run into a genrule, then you can specify the genrule's outputs as data for the production binary's *_binary rule.

I solved the first part, creating the directories. I still don't know how to add the latter two to classpath.
Starting from https://gerrit.googlesource.com/bazlets/+/master/tools/junit.bzl, I modified it to read
_OUTPUT = """import org.junit.runners.Suite;
import org.junit.runner.RunWith;
import org.junit.BeforeClass;
import java.io.File;
#RunWith(Suite.class)
#Suite.SuiteClasses({%s})
public class %s {
#BeforeClass
public static void setUp() throws Exception {
new File("./target").mkdir();
}
}
"""
_PREFIXES = ("org", "com", "edu")
# ...
I added the #BeforeClass setUp method.
I stored this as junit.bzl into third_party directory in my project.
Then in a BUILD file,
load("//third_party:junit.bzl", "junit_tests")
junit_tests(
name = "my_bundled_test",
srcs = glob(["src/test/java/**/*.java"]),
data = glob(["src/test/resources/**"]),
resources = glob(["src/test/resources/**"]),
tags = [
# ...
],
runtime_deps = [
# ...
],
],
deps = [
# ...
],
)
Now the test itself is wrapped with a setUp method which will create a directory for me. I am not deleting them afterwards, which is probably a sound idea to do.
The reason I need test resources in a directory (as opposed to in a jar file, which bazel gives by default) is that my test passes the URI to new FileInputStream(new File(uri)). If the file resides in a JAR, the URI will be file:/path/to/my.jar!/my.file and the rest of the test cannot work with such URI.

Equivalent of this.getClass().getClassLoader().getResource(".").toURI() from within a Jar

I'm trying to achieve a way to obtain the base path of the current classloader when runnning from within a jar.
I mean programatically, I already know it should have the shape of "jarPath+jarFile.jar!/"
Unlike file system's call, getResource(".") or .getResource("/") do not work from inside the jar.
Ideally it should be an abstract solution for any file provider, so something like:
Path BASE_PATH = Paths.get(...getResource("").toURI())
which could return the correct root path for both jars and file system so I can use relative urls to my resources without having to do any conditional statements and url manual string parsing/build.

You should be able to find out the path of the jar and or target folder containing you class or any resource by using this code:
package com.stackoverflow.test;
import java.net.URL;
import java.nio.file.Paths;
public class ClassPathUtils {
public static String getBasePath(String jarPath) {
String path = getJarPathFromClass(jarPath);
if (path == null) {
return null;
}
if (path.startsWith("jar:")) {
path = path.substring("jar:".length());
}
if (path.startsWith("file:")) {
path = path.substring("file:".length());
}
if (path.endsWith(jarPath)) {
path = path.substring(0, path.length()-jarPath.length());
}
return path;
}
public static String getBasePath(Class clazz) {
return getBasePath(classNameDotClass(clazz));
}
private static String classNameDotClass(Class clazz) {
return clazz.getName().replaceAll("\\.", "/") + ".class";
}
private static String getJarPathFromClass(String resource) {
final URL url = ClassPathUtils.class.getClassLoader().getResource(resource);
return url == null ? null : url.toString();
}
public static void main(String[] args) {
//System.out.println(Paths.get(ClassPathUtils.getBasePath("."))); // doesn't work in a jar
System.out.println(Paths.get(ClassPathUtils.getBasePath(ClassPathUtils.class)));
System.out.println(Paths.get(ClassPathUtils.getBasePath("fonts/atcitadelscript.ttf"))); // any classpath resource
System.out.println(Paths.get(ClassPathUtils.getBasePath(String.class))); // actually finds rt.jar
}
}
If you run this code from your IDE, or from maven, it will give you the paths to target/classes for your own resources, or the path to a jar for other resources (E.g. String.class).
If you call it from a jar, it will always tell you the path of the jar file.
run from IDE:
/home/alexander/projects/stackoverflow/stuff/target/classes
/home/alexander/projects/stackoverflow/stuff/target/classes
/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar!`
run from JAR:
/home/alexander/projects/stackoverflow/stuff/target/test-stuff-0.1-SNAPSHOT.jar!
/home/alexander/projects/stackoverflow/stuff/target/test-stuff-0.1-SNAPSHOT.jar!
/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar!
Is that what you're looking for?

Reading From Config File Outside Jar Java

Currently I am trying to read my config file from root of project directory, in order to make this actual configuration I want to move this to external location and then read from there.
Adding a complete path in following code throws out error :
package CopyEJ;
import java.util.Properties;
public class Config
{
Properties configFile;
public Config()
{
configFile = new java.util.Properties();
try {
// configFile.load(this.getClass().getClassLoader().getResourceAsStream("CopyEJ/config.properties"));
Error Statement ** configFile.load(this.getClass().getClassLoader().getResourceAsStream("C://EJ_Service//config.properties"));
}catch(Exception eta){
eta.printStackTrace();
}
}
public String getProperty(String key)
{
String value = this.configFile.getProperty(key);
return value;
}
}
Here's the error:
java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:365)
at java.util.Properties.load(Properties.java:293)
at CopyEJ.Config.<init>(Config.java:13)
at CopyEJ.CopyEJ.main(CopyEJ.java:22)
Exception in thread "main" java.lang.NullPointerException
at java.io.File.<init>(File.java:194)
at CopyEJ.CopyEJ.main(CopyEJ.java:48)
How can I fix this ?

The purpose of method getResourceAsStream is to open stream on some file, which exists inside your jar. If you know exact location of particular file, just open new FileInputStream.
I.e. your code should look like:
try (FileInputStream fis = new FileInputStream("C://EJ_Service//config.properties")) {
configFile.load(fis);
} catch(Exception eta){
eta.printStackTrace();
}

This line requires your config.properties to be in the java CLASSPATH
this.getClass().getClassLoader().getResourceAsStream("C://EJ_Service//config.properties")
When it is not, config.properties won't be accessible.
You can try some other alternative and use the configFile.load() function to read from.
One example would be:
InputStream inputStream = new FileInputStream(new File("C:/EJ_Service/config.properties"));
configFile.load(inputStream);

Jar file name form java code

I would like to determine the jar file name from my java code. I found many solutions in the google, but nothing works. Just to see what I tried here is a stackoverflow forum where a bunch of solutions is posted: stackoverflow
I have Mac OS X 10.6.5.
When I type java -version I get this result:
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
Thank you for your help.
Edit:
I edit my post to answer for the comment.
Some of the solutions gives me "null" value when I want to System.out.println the path and also fails when I want to create an instance of a File.
Other solutions when I ask for the path they don't give something like file:/....., instead they give something like rsch:/ or something like, this I don't know exactly, but it is a 4 character simple word.
Edit 2:
I run an executable jar from the console. And I would like to have this jar file name in the classes which are in the executed jar file.
Edit 3:
The 4 character word is: rsrc:./
Code how I got this:
File file = null;
try {
System.out.println(MyClass.class.getProtectionDomain().getCodeSource().getLocation().toURI());
} catch (URISyntaxException e) {
e.printStackTrace();
}
Edit 4:
I also tried this code:
package core;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;
public class MyClass {
public String getText(String key) {
String path = "" + MyClass.class.getResource("../any.properties");
File file = new File((path).substring(5, path.length()));
Properties props = readProps(file);
return props.getProperty(key);
}
private Properties readProps(File file) {
Properties props = new Properties();
InputStream in = null;
try {
in = new FileInputStream(file);
props.load(in);
in.close();
} catch (FileNotFoundException e1) {
e1.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return props;
}
public static void main(String[] args) {
System.out.println(new MyClass().getText("anything"));
}
}
With this result:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1937)
at core.PropHandler.getText(MyClass.java:14)
at core.PropHandler.main(MyClass.java:39)
... 5 more
This code perfectly runs in the eclipse, but when I create the runnable jar file I think you can see the problem.

Is this what you want? http://www.uofr.net/~greg/java/get-resource-listing.html
jcomeau#intrepid:/tmp$ cat test.java; javac test.java; jar cvf test.jar test.class; java -cp test.jar test
public class test {
public static void main(String[] args) {
System.out.println(test.class.getResource("test.class"));
}
}
adding: META-INF/ (in=0) (out=0) (stored 0%)
adding: META-INF/MANIFEST.MF (in=56) (out=56) (stored 0%)
adding: test.class (in=845) (out=515) (deflated 39%)
Total:
------
(in = 885) (out = 883) (deflated 0%)
jar:file:/tmp/test.jar!/test.class
For access to resources that works regardless of the presence of a jar file, I always use classname.class.getResourceAsStream(). But the linked document shows how to use JarFile() for the same purpose.

Ok, finally this is the code which resolved my problem:
String sConfigFile = "any.properties";
InputStream in = MyClass.class.getClassLoader().getResourceAsStream(sConfigFile);
if (in == null) {
System.out.println("ugly error handling :D");
}
Properties props = new java.util.Properties();
try {
props.load(in);
} catch (IOException e) {
e.printStackTrace();
}
With this way it founds my property file.