Develop Reference

Read files from sub directories of class path resource folder in spring boot

I would like to read files from sub directories of resource folder.
I am facing issues with jar execution.
This is my directory structure.
src/main/resources
|_ Conf
|_ conf1
|_ config.txt
|_ conf2
|_ config.txt
Here, I am trying to read config.txt files from all sub directories of Conf folder. I do not know what sub directories Conf will have. I know the classpath till Conf. So, I will give classpath till Conf and trying to get sub directories and files.
I tried to achieve this using ClassPathResource. This works fine if it is file. I am facing issues when it comes to directory. I am using getFile api to get the directory path to walk through that directory for sub directories which is causing issue in jar execution.
Here is my code:
Below code is to read sub directories in Conf folder.
List<Map<String,String>> list = new ArrayList<Map<String,String>>();
ClassPathResource classPathResource = new ClassPathResource("Conf");
File dir = classPathResource.getFile();
Files.walk(Paths.get(dir.toString()))
.filter(Files::isDirectory)
// This is to exempt current dir.
.filter((Path p)->!p.toString().equals(dir.toString()))
.forEach(f-> {list.add(readDirectory(f.toString()));});
Reading each sub directory.
public Map<String, String> readDirectory(String dir) {
Map<String, String> map = new HashMap<String, String>();
String confDir = dir.substring(dir.lastIndexOf(File.separator)+1);
try {
Files.list(Paths.get(dir))
.filter(f->f.toString().matches(".*conf\\.txt"))
.forEach(file ->approvedTermsMap.put
(confDir,readFile(file.toFile())));
} catch (IOException e) {
e.printStackTrace();
}
return map;
}
Reading file:
public String readFile(File confFile) {
StringBuffer terms = new StringBuffer();
try (BufferedReader reader = new BufferedReader(new
FileReader(confFile)))
{
reader.lines().forEach(term->
terms.append(term + "|"));
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return terms.toString();
}
Here, I should not use classPathResource.getFile() to get the absolute path because it tries to find file in file system which will not avilable in case of jar. So, I need alternate way to get absolute path of resource directory. I have to pass it to File.walk api to find sub directories and files.

As mentioned in the question, first I want to get confX directories then read conf.txt files.
Finally, I could solve my issue as below.
ClassLoader cl = this.getClass().getClassLoader();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);
try {
Resource resources[] = resolver.getResources("classpath:Conf/*/");
} catch (IOException e) {
e.printStackTrace();
}
This will give all sub directories of Conf directory. Here / at the end in classpath:Conf/*/ is very important. If we do not give / it will work normally but will not work in jar.
From the above code block resources[] array will contains directory location like this class path resource [Conf/conf1/] and so on. I need sub directory name to read corresponding file. Here is the code for it.
Arrays.asList(resources).stream()
.forEach(resource ->{
Pattern dirPattern = Pattern.compile(".*?\\[(.*/(.*?))/\\]$");
if (resource.toString().matches(".*?\\[.*?\\]$")) {
Matcher matcher = dirPattern.matcher(resource.toString());
if (matcher.find()) {
String dir = matcher.group(1);
readFile(dir);
}
}
});
public void readFile(String dir)
{
ClassPathResource classPathResource = new ClassPathResource(dir+ "/conf.txt");
try (BufferedReader fileReader = new BufferedReader(
new InputStreamReader(classPathResource2.getInputStream()))) {
fileReader.lines().forEach(data -> System.out.println(data));
}catch (IOException e) {
e.printStackTrace();
}
}
I need to map each txt file with its corresponding directory. That is why I approached this way. If you just need to get files and read you can do it like below. This will list everything under Conf directory.
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);
try {
Resource resources[] = resolver.getResources("classpath:Conf/**");
} catch (IOException e) {
e.printStackTrace();
}

Try the following code. It can scan up the required files up to n levels which can be specified using maxDepth varaible in following code
// Finding a file upto x level in File Directory using NIO Files.find
Path start = Paths.get("/Users/***/Documents/server_pull");
int maxDepth = 5;
try(Stream<Path> stream = Files.find(start,
maxDepth,
(path, attr) -> String.valueOf(path).endsWith(".txt"))){
String fileName = stream
.sorted()
.map(String::valueOf)
.filter((path) -> {
//System.out.println("In Filter : "+path);
return String.valueOf(path).endsWith("config.txt");
})
.collect(Collectors.joining());
System.out.println("fileName : "+fileName);
}catch(Exception e){
e.printStackTrace();
}
Another way by using Files.walk methods as follows:
// Finding a file upto x level in File Directory using NIO Files.walk
Path startWalk = Paths.get("/Users/***/Documents/server_pull");
int depth = 5;
try( Stream<Path> stream1 = Files.walk(startWalk,
depth)){
String walkedFile = stream1
.map(String::valueOf)
.filter(path -> {
return String.valueOf(path).endsWith("config.txt");
})
.sorted()
.collect(Collectors.joining());
System.out.println("walkedFile = "+walkedFile);
}catch(Exception e){
e.printStackTrace();
}

Unzip nested jar files java

I am trying to unzip all the jar files and the jar files which is nested in jar file.
For example, let's say there's a Test.jar and inside of the Test.jar, there is Test1.jar,,
What I tried to do is that making a temp directory and unzip them, when it was jar file, recursive call.
Here below is my code and the log I got. I have no idea on that.
I am pretty sure that the input was directory. I have no idea on resolving this error. Also, I am pretty sure that the error is from here (Collection<File> files = FileUtils.listFiles(root, null, recursive);)
Curr directory:/Users/younghoonkwon/jar-analyzer
unzipping directory/Users/younghoonkwon/jar-analyzer/test1.jar#
Curr directory:/Users/younghoonkwon/jar-analyzer/test1.jar#
java.lang.IllegalArgumentException: Parameter 'directory' is not a directory
at org.apache.commons.io.FileUtils.validateListFilesParameters(FileUtils.java:545)
at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:521)
at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:691)
at org.vulnerability.checker.JarParser.unzipJars(JarParser.java:31)
at org.vulnerability.checker.JarParser.unzipJars(JarParser.java:38)
at org.vulnerability.checker.VulnerabilityChecker.main(VulnerabilityChecker.java:26)
[/Users/younghoonkwon/jar-analyzer/test1.jar]
My code:
public void unzipJars(String toFind, String currDirectory) {
File root = new File(currDirectory);
try {
boolean recursive = true;
System.out.println("Curr directory:"+root);
Collection<File> files = FileUtils.listFiles(root, null, recursive);
for (Iterator<File> iterator = files.iterator(); iterator.hasNext();) {
File file = (File) iterator.next();
if (file.getName().endsWith(toFind)) {
if(toFind.endsWith("jar")) {
unzip(file.getAbsolutePath() + "#",file.getAbsolutePath());
System.out.println("unzipping directory"+ file.getAbsolutePath()+"#");
unzipJars("jar", file.getAbsolutePath()+"#");
this.jarList.add(file.getAbsolutePath());
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
static void unzip(String destDirPath, String zipFilePath) throws IOException {
Runtime.getRuntime().exec("unzip "+ zipFilePath + " -d" + destDirPath);
}

The algorithm seems ok to me. The error seems to be caused by that the unzipped file is not a directory (but a file) or it does not exist. Your unzip() method does not throw any exeption if unzipping the .jar fails because of output file already existing.
Have you been running the code earlier which may have caused that the .jars or directories contain unwanted output files with the same name?
Before the call to FileUtils.listFiles(), check if the root File object is actually a directory and it exists (or if it's a file but not a directory) by File.isDirectory() or File.isFile().

The following method decompress() unzips a JAR file and all the JAR files within it (recursively).
/**
* Size of the buffer to read/write data.
*/
private static final int BUFFER_SIZE = 16384;
/**
* Decompress all JAR files located in a given directory.
*
* #param outputDirectory Path to the directory where the decompressed JAR files are located.
*/
public static void decompress(final String outputDirectory) {
File files = new File(outputDirectory);
for (File f : Objects.requireNonNull(files.listFiles())) {
if (f.getName().endsWith(".jar")) {
try {
JarUtils.decompressDependencyFiles(f.getAbsolutePath());
// delete the original dependency jar file
org.apache.commons.io.FileUtils.forceDelete(f);
} catch (IOException e) {
log.warn("Problem decompressing jar file: " + f.getAbsolutePath());
}
}
}
}
/**
* Decompress all JAR files (recursively).
*
* #param zipFile The file to be decompressed.
*/
private static void decompressDependencyFiles(String zipFile) throws IOException {
File file = new File(zipFile);
try (ZipFile zip = new ZipFile(file)) {
String newPath = zipFile.substring(0, zipFile.length() - 4);
new File(newPath).mkdir();
Enumeration<? extends ZipEntry> zipFileEntries = zip.entries();
// Process each entry
while (zipFileEntries.hasMoreElements()) {
// grab a zip file entry
ZipEntry entry = zipFileEntries.nextElement();
String currentEntry = entry.getName();
File destFile = new File(newPath, currentEntry);
File destinationParent = destFile.getParentFile();
// create the parent directory structure if needed
destinationParent.mkdirs();
if (!entry.isDirectory()) {
BufferedInputStream is = new BufferedInputStream(zip.getInputStream(entry));
int currentByte;
// establish buffer for writing file
byte[] data = new byte[BUFFER_SIZE];
// write the current file to disk
FileOutputStream fos = new FileOutputStream(destFile);
try (BufferedOutputStream dest = new BufferedOutputStream(fos, BUFFER_SIZE)) {
// read and write until last byte is encountered
while ((currentByte = is.read(data, 0, BUFFER_SIZE)) != -1) {
dest.write(data, 0, currentByte);
}
dest.flush();
is.close();
}
}
if (currentEntry.endsWith(".jar")) {
// found a zip file, try to open
decompressDependencyFiles(destFile.getAbsolutePath());
FileUtils.forceDelete(new File(destFile.getAbsolutePath()));
}
}
}
}

Java nio move folder with content throws exception [duplicate]

How do you move a file from one location to another? When I run my program any file created in that location automatically moves to the specified location. How do I know which file is moved?

myFile.renameTo(new File("/the/new/place/newName.file"));
File#renameTo does that (it can not only rename, but also move between directories, at least on the same file system).
Renames the file denoted by this abstract pathname.
Many aspects of the behavior of this method are inherently platform-dependent: The rename operation might not be able to move a file from one filesystem to another, it might not be atomic, and it might not succeed if a file with the destination abstract pathname already exists. The return value should always be checked to make sure that the rename operation was successful.
If you need a more comprehensive solution (such as wanting to move the file between disks), look at Apache Commons FileUtils#moveFile

With Java 7 or newer you can use Files.move(from, to, CopyOption... options).
E.g.
Files.move(Paths.get("/foo.txt"), Paths.get("bar.txt"), StandardCopyOption.REPLACE_EXISTING);
See the Files documentation for more details

Java 6
public boolean moveFile(String sourcePath, String targetPath) {
File fileToMove = new File(sourcePath);
return fileToMove.renameTo(new File(targetPath));
}
Java 7 (Using NIO)
public boolean moveFile(String sourcePath, String targetPath) {
boolean fileMoved = true;
try {
Files.move(Paths.get(sourcePath), Paths.get(targetPath), StandardCopyOption.REPLACE_EXISTING);
} catch (Exception e) {
fileMoved = false;
e.printStackTrace();
}
return fileMoved;
}

File.renameTo from Java IO can be used to move a file in Java. Also see this SO question.

To move a file you could also use Jakarta Commons IOs FileUtils.moveFile
On error it throws an IOException, so when no exception is thrown you know that that the file was moved.

Just add the source and destination folder paths.
It will move all the files and folder from source folder to
destination folder.
File destinationFolder = new File("");
File sourceFolder = new File("");
if (!destinationFolder.exists())
{
destinationFolder.mkdirs();
}
// Check weather source exists and it is folder.
if (sourceFolder.exists() && sourceFolder.isDirectory())
{
// Get list of the files and iterate over them
File[] listOfFiles = sourceFolder.listFiles();
if (listOfFiles != null)
{
for (File child : listOfFiles )
{
// Move files to destination folder
child.renameTo(new File(destinationFolder + "\\" + child.getName()));
}
// Add if you want to delete the source folder
sourceFolder.delete();
}
}
else
{
System.out.println(sourceFolder + " Folder does not exists");
}

Files.move(source, target, REPLACE_EXISTING);
You can use the Files object
Read more about Files

You could execute an external tool for that task (like copy in windows environments) but, to keep the code portable, the general approach is to:
read the source file into memory
write the content to a file at the new location
delete the source file
File#renameTo will work as long as source and target location are on the same volume. Personally I'd avoid using it to move files to different folders.

Try this :-
boolean success = file.renameTo(new File(Destdir, file.getName()));

Wrote this method to do this very thing on my own project only with the replace file if existing logic in it.
// we use the older file i/o operations for this rather than the newer jdk7+ Files.move() operation
private boolean moveFileToDirectory(File sourceFile, String targetPath) {
File tDir = new File(targetPath);
if (tDir.exists()) {
String newFilePath = targetPath+File.separator+sourceFile.getName();
File movedFile = new File(newFilePath);
if (movedFile.exists())
movedFile.delete();
return sourceFile.renameTo(new File(newFilePath));
} else {
LOG.warn("unable to move file "+sourceFile.getName()+" to directory "+targetPath+" -> target directory does not exist");
return false;
}
}

Please try this.
private boolean filemovetoanotherfolder(String sourcefolder, String destinationfolder, String filename) {
boolean ismove = false;
InputStream inStream = null;
OutputStream outStream = null;
try {
File afile = new File(sourcefolder + filename);
File bfile = new File(destinationfolder + filename);
inStream = new FileInputStream(afile);
outStream = new FileOutputStream(bfile);
byte[] buffer = new byte[1024 * 4];
int length;
// copy the file content in bytes
while ((length = inStream.read(buffer)) > 0) {
outStream.write(buffer, 0, length);
}
// delete the original file
afile.delete();
ismove = true;
System.out.println("File is copied successful!");
} catch (IOException e) {
e.printStackTrace();
}finally{
inStream.close();
outStream.close();
}
return ismove;
}

Moving file from one directory to another and delete from source directory using java

I am trying to move files from one directory to another delete that file from source directory after moving.
for (File file : files) {
if (file != null) {
boolean status = moveFile(file, filePath, name, docGroupId);
if (status) {
//some operations....
}
}
}
public static boolean moveFile(final File file, final String filePath, final String groupName, Integer docGroupId) {
// TODO Auto-generated method stub
String selectedDirectory = filePath + File.separator + groupName;
InputStream in = null;
OutputStream out = null;
try {
if (!file.isDirectory()) {
File dir = new File(selectedDirectory);
if (!dir.exists()) {
dir.mkdirs();
}
String newFilString = dir.getAbsolutePath() +
File.separator + file.getName();
File newFile = new File(newFilString);
in = new FileInputStream(file);
out = new FileOutputStream(newFile);
byte[] moveBuff = new byte[1024];
int butesRead;
while ((butesRead = in.read(moveBuff)) > 0) {
out.write(moveBuff, 0, butesRead);
}
}
in.close();
out.close();
if(file.delete())
return true;
} catch (Exception e) {
return false;
}
}
The program works on Linux-Ubuntu and all files are moved to another directory and deleted from source directory, but in Windows system all files are moved but failed to delete one or two files from source directory. Please note that while debugging the program is working fine.

Consider using Files.delete instead of File.delete. The javadoc says:
Note that the Files class defines the delete method to throw an IOException when a file cannot be deleted. This is useful for error reporting and to diagnose why a file cannot be deleted.
This should provide the information necessary to diagnose the problem.

So, if problem comes with delete, possible explanations:
you do file.delete() on every files and directories. How do you know the directory is empty ? If not, it will fail, then what happen to next instructions ?
file deletion is OS-dependant. On Windows, you can have many security issues, depending on which user, which rights, which location. You should check with a file-delete-alone program;
last: files can be locked by other programs (even explorer), it is also OS-dependant.

You don't need any of this if the source and target are in the same file system. Just use File.renameTo().

How can I access a folder inside of a resource folder from inside my jar File?

I have a resources folder/package in the root of my project, I "don't" want to load a certain File. If I wanted to load a certain File, I would use class.getResourceAsStream and I would be fine!! What I actually want to do is to load a "Folder" within the resources folder, loop on the Files inside that Folder and get a Stream to each file and read in the content... Assume that the File names are not determined before runtime... What should I do? Is there a way to get a list of the files inside a Folder in your jar File?
Notice that the Jar file with the resources is the same jar file from which the code is being run...

Finally, I found the solution:
final String path = "sample/folder";
final File jarFile = new File(getClass().getProtectionDomain().getCodeSource().getLocation().getPath());
if(jarFile.isFile()) { // Run with JAR file
final JarFile jar = new JarFile(jarFile);
final Enumeration<JarEntry> entries = jar.entries(); //gives ALL entries in jar
while(entries.hasMoreElements()) {
final String name = entries.nextElement().getName();
if (name.startsWith(path + "/")) { //filter according to the path
System.out.println(name);
}
}
jar.close();
} else { // Run with IDE
final URL url = Launcher.class.getResource("/" + path);
if (url != null) {
try {
final File apps = new File(url.toURI());
for (File app : apps.listFiles()) {
System.out.println(app);
}
} catch (URISyntaxException ex) {
// never happens
}
}
}
The second block just work when you run the application on IDE (not with jar file), You can remove it if you don't like that.

Try the following.
Make the resource path "<PathRelativeToThisClassFile>/<ResourceDirectory>" E.g. if your class path is com.abc.package.MyClass and your resoure files are within src/com/abc/package/resources/:
URL url = MyClass.class.getResource("resources/");
if (url == null) {
// error - missing folder
} else {
File dir = new File(url.toURI());
for (File nextFile : dir.listFiles()) {
// Do something with nextFile
}
}
You can also use
URL url = MyClass.class.getResource("/com/abc/package/resources/");

The following code returns the wanted "folder" as Path regardless of if it is inside a jar or not.
private Path getFolderPath() throws URISyntaxException, IOException {
URI uri = getClass().getClassLoader().getResource("folder").toURI();
if ("jar".equals(uri.getScheme())) {
FileSystem fileSystem = FileSystems.newFileSystem(uri, Collections.emptyMap(), null);
return fileSystem.getPath("path/to/folder/inside/jar");
} else {
return Paths.get(uri);
}
}
Requires java 7+.

I know this is many years ago . But just for other people come across this topic.
What you could do is to use getResourceAsStream() method with the directory path, and the input Stream will have all the files name from that dir. After that you can concat the dir path with each file name and call getResourceAsStream for each file in a loop.

I had the same problem at hands while i was attempting to load some hadoop configurations from resources packed in the jar... on both the IDE and on jar (release version).
I found java.nio.file.DirectoryStream to work the best to iterate over directory contents over both local filesystem and jar.
String fooFolder = "/foo/folder";
....
ClassLoader classLoader = foofClass.class.getClassLoader();
try {
uri = classLoader.getResource(fooFolder).toURI();
} catch (URISyntaxException e) {
throw new FooException(e.getMessage());
} catch (NullPointerException e){
throw new FooException(e.getMessage());
}
if(uri == null){
throw new FooException("something is wrong directory or files missing");
}
/** i want to know if i am inside the jar or working on the IDE*/
if(uri.getScheme().contains("jar")){
/** jar case */
try{
URL jar = FooClass.class.getProtectionDomain().getCodeSource().getLocation();
//jar.toString() begins with file:
//i want to trim it out...
Path jarFile = Paths.get(jar.toString().substring("file:".length()));
FileSystem fs = FileSystems.newFileSystem(jarFile, null);
DirectoryStream<Path> directoryStream = Files.newDirectoryStream(fs.getPath(fooFolder));
for(Path p: directoryStream){
InputStream is = FooClass.class.getResourceAsStream(p.toString()) ;
performFooOverInputStream(is);
/** your logic here **/
}
}catch(IOException e) {
throw new FooException(e.getMessage());
}
}
else{
/** IDE case */
Path path = Paths.get(uri);
try {
DirectoryStream<Path> directoryStream = Files.newDirectoryStream(path);
for(Path p : directoryStream){
InputStream is = new FileInputStream(p.toFile());
performFooOverInputStream(is);
}
} catch (IOException _e) {
throw new FooException(_e.getMessage());
}
}

Another solution, you can do it using ResourceLoader like this:
import org.springframework.core.io.Resource;
import org.apache.commons.io.FileUtils;
#Autowire
private ResourceLoader resourceLoader;
...
Resource resource = resourceLoader.getResource("classpath:/path/to/you/dir");
File file = resource.getFile();
Iterator<File> fi = FileUtils.iterateFiles(file, null, true);
while(fi.hasNext()) {
load(fi.next())
}

If you are using Spring you can use org.springframework.core.io.support.PathMatchingResourcePatternResolver and deal with Resource objects rather than files. This works when running inside and outside of a Jar file.
PathMatchingResourcePatternResolver r = new PathMatchingResourcePatternResolver();
Resource[] resources = r.getResources("/myfolder/*");
Then you can access the data using getInputStream and the filename from getFilename.
Note that it will still fail if you try to use the getFile while running from a Jar.

As the other answers point out, once the resources are inside a jar file, things get really ugly. In our case, this solution:
https://stackoverflow.com/a/13227570/516188
works very well in the tests (since when the tests are run the code is not packed in a jar file), but doesn't work when the app actually runs normally. So what I've done is... I hardcode the list of the files in the app, but I have a test which reads the actual list from disk (can do it since that works in tests) and fails if the actual list doesn't match with the list the app returns.
That way I have simple code in my app (no tricks), and I'm sure I didn't forget to add a new entry in the list thanks to the test.

Below code gets .yaml files from a custom resource directory.
ClassLoader classLoader = this.getClass().getClassLoader();
URI uri = classLoader.getResource(directoryPath).toURI();
if("jar".equalsIgnoreCase(uri.getScheme())){
Pattern pattern = Pattern.compile("^.+" +"/classes/" + directoryPath + "/.+.yaml$");
log.debug("pattern {} ", pattern.pattern());
ApplicationHome home = new ApplicationHome(SomeApplication.class);
JarFile file = new JarFile(home.getSource());
Enumeration<JarEntry> jarEntries = file.entries() ;
while(jarEntries.hasMoreElements()){
JarEntry entry = jarEntries.nextElement();
Matcher matcher = pattern.matcher(entry.getName());
if(matcher.find()){
InputStream in =
file.getInputStream(entry);
//work on the stream
}
}
}else{
//When Spring boot application executed through Non-Jar strategy like through IDE or as a War.
String path = uri.getPath();
File[] files = new File(path).listFiles();
for(File file: files){
if(file != null){
try {
InputStream is = new FileInputStream(file);
//work on stream
} catch (Exception e) {
log.error("Exception while parsing file yaml file {} : {} " , file.getAbsolutePath(), e.getMessage());
}
}else{
log.warn("File Object is null while parsing yaml file");
}
}
}

Took me 2-3 days to get this working, in order to have the same url that work for both Jar or in local, the url (or path) needs to be a relative path from the repository root.
..meaning, the location of your file or folder from your src folder.
could be "/main/resources/your-folder/" or "/client/notes/somefile.md"
Whatever it is, in order for your JAR file to find it, the url must be a relative path from the repository root.
it must be "src/main/resources/your-folder/" or "src/client/notes/somefile.md"
Now you get the drill, and luckily for Intellij Idea users, you can get the correct path with a right-click on the folder or file -> copy Path/Reference.. -> Path From Repository Root (this is it)
Last, paste it and do your thing.

Simple ... use OSGi. In OSGi you can iterate over your Bundle's entries with findEntries and findPaths.

Inside my jar file I had a folder called Upload, this folder had three other text files inside it and I needed to have an exactly the same folder and files outside of the jar file, I used the code below:
URL inputUrl = getClass().getResource("/upload/blabla1.txt");
File dest1 = new File("upload/blabla1.txt");
FileUtils.copyURLToFile(inputUrl, dest1);
URL inputUrl2 = getClass().getResource("/upload/blabla2.txt");
File dest2 = new File("upload/blabla2.txt");
FileUtils.copyURLToFile(inputUrl2, dest2);
URL inputUrl3 = getClass().getResource("/upload/blabla3.txt");
File dest3 = new File("upload/Bblabla3.txt");
FileUtils.copyURLToFile(inputUrl3, dest3);