Develop Reference

Read files from sub directories of class path resource folder in spring boot

I would like to read files from sub directories of resource folder.
I am facing issues with jar execution.
This is my directory structure.
src/main/resources
|_ Conf
|_ conf1
|_ config.txt
|_ conf2
|_ config.txt
Here, I am trying to read config.txt files from all sub directories of Conf folder. I do not know what sub directories Conf will have. I know the classpath till Conf. So, I will give classpath till Conf and trying to get sub directories and files.
I tried to achieve this using ClassPathResource. This works fine if it is file. I am facing issues when it comes to directory. I am using getFile api to get the directory path to walk through that directory for sub directories which is causing issue in jar execution.
Here is my code:
Below code is to read sub directories in Conf folder.
List<Map<String,String>> list = new ArrayList<Map<String,String>>();
ClassPathResource classPathResource = new ClassPathResource("Conf");
File dir = classPathResource.getFile();
Files.walk(Paths.get(dir.toString()))
.filter(Files::isDirectory)
// This is to exempt current dir.
.filter((Path p)->!p.toString().equals(dir.toString()))
.forEach(f-> {list.add(readDirectory(f.toString()));});
Reading each sub directory.
public Map<String, String> readDirectory(String dir) {
Map<String, String> map = new HashMap<String, String>();
String confDir = dir.substring(dir.lastIndexOf(File.separator)+1);
try {
Files.list(Paths.get(dir))
.filter(f->f.toString().matches(".*conf\\.txt"))
.forEach(file ->approvedTermsMap.put
(confDir,readFile(file.toFile())));
} catch (IOException e) {
e.printStackTrace();
}
return map;
}
Reading file:
public String readFile(File confFile) {
StringBuffer terms = new StringBuffer();
try (BufferedReader reader = new BufferedReader(new
FileReader(confFile)))
{
reader.lines().forEach(term->
terms.append(term + "|"));
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return terms.toString();
}
Here, I should not use classPathResource.getFile() to get the absolute path because it tries to find file in file system which will not avilable in case of jar. So, I need alternate way to get absolute path of resource directory. I have to pass it to File.walk api to find sub directories and files.

As mentioned in the question, first I want to get confX directories then read conf.txt files.
Finally, I could solve my issue as below.
ClassLoader cl = this.getClass().getClassLoader();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);
try {
Resource resources[] = resolver.getResources("classpath:Conf/*/");
} catch (IOException e) {
e.printStackTrace();
}
This will give all sub directories of Conf directory. Here / at the end in classpath:Conf/*/ is very important. If we do not give / it will work normally but will not work in jar.
From the above code block resources[] array will contains directory location like this class path resource [Conf/conf1/] and so on. I need sub directory name to read corresponding file. Here is the code for it.
Arrays.asList(resources).stream()
.forEach(resource ->{
Pattern dirPattern = Pattern.compile(".*?\\[(.*/(.*?))/\\]$");
if (resource.toString().matches(".*?\\[.*?\\]$")) {
Matcher matcher = dirPattern.matcher(resource.toString());
if (matcher.find()) {
String dir = matcher.group(1);
readFile(dir);
}
}
});
public void readFile(String dir)
{
ClassPathResource classPathResource = new ClassPathResource(dir+ "/conf.txt");
try (BufferedReader fileReader = new BufferedReader(
new InputStreamReader(classPathResource2.getInputStream()))) {
fileReader.lines().forEach(data -> System.out.println(data));
}catch (IOException e) {
e.printStackTrace();
}
}
I need to map each txt file with its corresponding directory. That is why I approached this way. If you just need to get files and read you can do it like below. This will list everything under Conf directory.
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);
try {
Resource resources[] = resolver.getResources("classpath:Conf/**");
} catch (IOException e) {
e.printStackTrace();
}

Try the following code. It can scan up the required files up to n levels which can be specified using maxDepth varaible in following code
// Finding a file upto x level in File Directory using NIO Files.find
Path start = Paths.get("/Users/***/Documents/server_pull");
int maxDepth = 5;
try(Stream<Path> stream = Files.find(start,
maxDepth,
(path, attr) -> String.valueOf(path).endsWith(".txt"))){
String fileName = stream
.sorted()
.map(String::valueOf)
.filter((path) -> {
//System.out.println("In Filter : "+path);
return String.valueOf(path).endsWith("config.txt");
})
.collect(Collectors.joining());
System.out.println("fileName : "+fileName);
}catch(Exception e){
e.printStackTrace();
}
Another way by using Files.walk methods as follows:
// Finding a file upto x level in File Directory using NIO Files.walk
Path startWalk = Paths.get("/Users/***/Documents/server_pull");
int depth = 5;
try( Stream<Path> stream1 = Files.walk(startWalk,
depth)){
String walkedFile = stream1
.map(String::valueOf)
.filter(path -> {
return String.valueOf(path).endsWith("config.txt");
})
.sorted()
.collect(Collectors.joining());
System.out.println("walkedFile = "+walkedFile);
}catch(Exception e){
e.printStackTrace();
}

Getting the list of filenames from resource folder when running in jar

I have some Json files in the folder "resource/json/templates". I want to read these Json files. So far, the snippet of code below is allowing me to do so when running the program in IDE but it fails when I run it in the jar.
JSONParser parser = new JSONParser();
ClassLoader loader = getClass().getClassLoader();
URL url = loader.getResource(templateDirectory);
String path = url.getPath();
File[] files = new File(path).listFiles();
PipelineTemplateRepo pipelineTemplateRepo = new PipelineTemplateRepoImpl();
File templateFile;
JSONObject templateJson;
PipelineTemplateVo templateFromFile;
PipelineTemplateVo templateFromDB;
String templateName;
for (int i = 0; i < files.length; i++) {
if (files[i].isFile()) {
templateFile = files[i];
templateJson = (JSONObject) parser.parse(new FileReader(templateFile));
//Other logic
}
}
}
catch (Exception e) {
e.printStackTrace();
}
Any help would be greatly appreciated.
Thanks a lot.

Assuming that in the class path, in the jar the directory starts with /json (/resource is a root directory), it could be as such:
URL url = getClass().getResource("/json");
Path path = Paths.get(url.toURI());
Files.walk(path, 5).forEach(p -> System.out.printf("- %s%n", p.toString()));
This uses a jar:file://... URL, and opens a virtual file system on it.
Inspect that the jar indeed uses that path.
Reading can be done as desired.
BufferedReader in = Files.newBufferedReader(p, StandardCharsets.UTF_8);

Firstly, remember that Jars are Zip files so you can't get an individual File out of it without unzipping it. Zip files don't exactly have directories, so it's not as simple as getting the children of a directory.
This was a bit of a difficult one but I too was curious, and after researching I have come up with the following.
Firstly, you could try putting the resources into a flat Zip file (resource/json/templates.zip) nested in the Jar, then loading all the resources from that zip file since you know all the zip entries will be the resources you want. This should work even in the IDE.
String path = "resource/json/templates.zip";
ZipInputStream zis = new ZipInputStream(getClass().getResourceAsStream(path));
for (ZipEntry ze = zis.getNextEntry(); ze != null; ze = zis.getNextEntry()) {
// 'zis' is the input stream and will yield an 'EOF' before the next entry
templateJson = (JSONObject) parser.parse(zis);
}
Alternatively, you could get the running Jar, iterate through its entries, and collect the ones that are children of resource/json/templates/ then get the streams from those entries. NOTE: This will only work when running the Jar, add a check to run something else while running in the IDE.
public void runOrSomething() throws IOException, URISyntaxException {
// ... other logic ...
final String path = "resource/json/templates/";
Predicate<JarEntry> pred = (j) -> !j.isDirectory() && j.getName().startsWith(path);
try (JarFile jar = new Test().getThisJar()) {
List<JarEntry> resources = getEntriesUnderPath(jar, pred);
for (JarEntry entry : resources) {
System.out.println(entry.getName());
try (InputStream is = jar.getInputStream(entry)) {
// JarEntry streams are closed when their JarFile is closed,
// so you must use them before closing 'jar'
templateJson = (JSONObject) parser.parse(is);
// ... other logic ...
}
}
}
}
// gets ALL the children, not just direct
// path should usually end in backslash
public static List<JarEntry> getEntriesUnderPath(JarFile jar, Predicate<JarEntry> pred)
{
List<JarEntry> list = new LinkedList<>();
Enumeration<JarEntry> entries = jar.entries();
// has to iterate through all the Jar entries
while (entries.hasMoreElements()) {
JarEntry entry = entries.nextElement();
if (pred.test(entry))
list.add(entry);
}
return list;
}
public JarFile getThisJar() throws IOException, URISyntaxException {
URL url = getClass().getProtectionDomain().getCodeSource().getLocation();
return new JarFile(new File(url.toURI()));
}
I hope this helps.

Java stream - too many open files

I have a program that goes to a few directories from time to time and and do some kind of processing to the files in those directories.
The problem is from time to time (each two or three days) the program is reaching the OS open files limit.
It is a spring-boot application running in a RHEL 7.
the method that get the files is this:
public File[] getFiles(String dir, int numberOfFiles) throws Exception {
final Path baseDir = Paths.get(dir);
List<File> filesFromPath = new ArrayList<File>();
File[] files = null;
final BiPredicate<Path, BasicFileAttributes> predicate = (path, attrs) -> attrs.isRegularFile()
&& String.valueOf(path).endsWith(".xml");
List<Path> result;
try (Stream<Path> fileStream = Files.find(baseDir, 1, predicate).limit(numberOfFiles).onClose(() -> LOG.debug("Closing file stream."))){
result = fileStream.collect(Collectors.toList());
result.forEach(path -> {
path.toString();
File file = path.toFile();
LOG.info("adding {} to process.", file.getName());
filesFromPath.add(file);
});
if (filesFromPath != null && !filesFromPath.isEmpty()) {
files = filesFromPath.toArray(new File[filesFromPath.size()]);
}
} catch (Exception e) {
LOG.error("Error during file opening/closing", e);
}
if (files != null) {
return files;
}
return new File[0];
}
I am using the lsof command to see how many opened files I have, and the list of directories is always growing.
I added a log in the onClise method and this is being called all times I open a stream.
Shouldn't the try with resources, close the stream?
[EDIT]
There is also another peace of code that move processed files to another folder. This code does not use stream, and I couldn't found out what is wrong with it, besides the fact it is ugly.
public void move(File file, String archivePath) throws IOException {
File backupFile = new File(archivePath);
if (!backupFile.exists()) {
backupFile.mkdirs();
}
Path source = file.toPath();
if (file.exists()) {
Path target = Paths.get(archivePath + File.separator + file.getName());
Files.move(source, target, StandardCopyOption.REPLACE_EXISTING);
LOG.info("file {} moved to {}", file, archivePath);
} else {
LOG.info("unable to move the file: {} because it was already moved to {}", file, archivePath);
}
}
[EDIT 2]
and all files are being processed like this:
private void processFile(File[] files) throws Exception {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
XPath xpathParser = XPathFactory.newInstance().newXPath();
for (int i = 0; i < files.length; i++) {
File file = files[i];
Document doc = db.parse(file);
// DO STUFF
fileUtils.move(file, processedPath);
}
}
Thanks.

I hit the same issue. In the case of calls to "parse" there is very much files open and I clearly see the implementation opens an input stream but may not close it in case of a failure.

Java: How to create a directory in a zip using java.nio.file.FileSystem

I've followed what this page has told me but I can't get it to work. I want it so that in my test.zip a folder called "new" will be in there. Whenever I run the code below it gives a FileAlreadyExistsException and only creates an empty zip file.
Map<String, String> env = new HashMap<>();
env.put("create", "true");
Path path = Paths.get("test.zip");
URI uri = URI.create("jar:" + path.toUri());
try (FileSystem fs = FileSystems.newFileSystem(uri, env)) {
Path nf = fs.getPath("new/");
Files.createDirectory(path);
} catch (IOException e) {
e.printStackTrace();
}

Because Files.createDirectory() states in the javadoc
throws FileAlreadyExistsException - if dir exists but is not a
directory (optional specific exception)
you need to check if the folder already exits:
try (FileSystem fs = FileSystems.newFileSystem(uri, env)) {
Path nf = fs.getPath("new");
if (Files.notExists(nf)) {
Files.createDirectory(nf);
}
}

Have you tried java.util.zip.ZipEntry ?
FileOutputStream f = new FileOutputStream("test.zip");
ZipOutputStream zip = new ZipOutputStream(new BufferedOutputStream(f));
zip.putNextEntry(new ZipEntry("new/"));

How to search for files in directory that start with String

I have a directory full of text files. The naming convention of these files is {userName} + {dayOfYear}
SuperMan201
JamesBond056
JamesBond101
JamesBond093
JamesBondImposter004
SuperMan051
JamesBond057
JamesBond004
I want to search through my files for files that start with Jamesbond. How can I get the first instance of a text file that starts with Jamesbond and not get JamesBondImposter matching my search.
I have a LinkedHashSet that stores the usernames of the users that have text files in the directory. Essentially I want to get the first instance of a JamesBond text file read the file,move the file to another directory, then read subsequent files if they exist; and then move on to the next username in the LinkedList.

If you want to select files from directory take a look at https://docs.oracle.com/javase/7/docs/api/java/io/File.html#listFiles(java.io.FileFilter) and https://docs.oracle.com/javase/7/docs/api/java/io/File.html#listFiles(java.io.FilenameFilter) - just write the filter that will match what you want

Using Apache commons-io (listFiles and iterateFiles methods). Usually the code looks something like this:
File dir = new File(".");
FileFilter fileFilter = new WildcardFileFilter("James*");
File[] files = dir.listFiles(fileFilter);
if(files!=null && files.length > 0)
{
your desired file is files[0]
}

If you're using Java 8, you can do this:
Path dir = Paths.get(fullPathOfDirectory);
Stream<Path> jamesBondFiles = Files.list(dir).filter(path ->
path.getFileName().toString().matches("JamesBond\\d+"));
Iterator<Path> i = jamesBondFiles.iterator();
while (i.hasNext()) {
Path file = i.next();
try (BufferedReader reader = Files.newBufferedReader(file)) {
// Read and process file
}
}
If you're using Java 7, or don't want to use a Stream, you can do:
Path dir = Paths.get(fullPathOfDirectory);
final PathMatcher matcher =
dir.getFileSystem().getPathMatcher("regex:JamesBond\\d+");
DirectoryStream.Filter<Path> filter = new DirectoryStream.Filter<Path>() {
#Override
public boolean accept(Path path) {
return matcher.matches(path.getFileName());
}
};
try (DirectoryStream<Path> ds = Files.newDirectoryStream(dir, filter)) {
Charset charset = Charset.defaultCharset();
for (Path file : ds) {
try (BufferedReader reader = Files.newBufferedReader(file, charset)) {
// Read and process file
}
}
}