Develop Reference

How to open and display content of multiple files in folder with multiple folder?

I have folders with multiple files. This is how the organization is.
Folder_1
- Folder_1_File_1
- Folder_1_File_2
- Folder_1_File_3
- Folder_1_File_4
...
Folder_2
- Folder_2_File_1
- Folder_2_File_2
- Folder_2_File_3
- Folder_2_File_4
...
Folder_3
- Folder_3_File_1
- Folder_3_File_2
- Folder_3_File_3
- Folder_3_File_4
...
I have to open each folder, read each file and then perform some computations on all files of each folder and display result, then move to next folder and perform same computations on files of those folders. Below is my code. Is there any other easier method to do it? My code is really slow and sometimes it doesn't even work.
public void listFilesForFolder_Test(final File folder) {
for (final File fileEntry : folder.listFiles()) {
if (fileEntry.isDirectory()) {
listFilesForFolder_Test(fileEntry);
} else {
Test_filenames.add(fileEntry.getAbsolutePath());
Test_filename.add(fileEntry.getName());
// String content = FileUtils.readFileToString(file);
// read file and perform computations on it.
}
}
}

Java 8 has introduced the Visitor-based Files.walkFileTree() api which is a lot easier to master:
File file = new File("/base/folder");
Files.walkFileTree(file.toPath(), new SimpleFileVisitor<>() {
#Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
// do your thing here
return FileVisitResult.CONTINUE;
}
});

This code will provide you regular files (not folders) at first the walk the rest of the given path tree.
public class ReadRecurssivelyFromFolder {
public static void main(String[] args) {
try {
Files.walk(Paths.get("/home/rafik/test" ))
.filter(Files::isRegularFile)
.forEach(System.out::println);
} catch (IOException ex) {
Logger.getLogger(ReadRecurssivelyFromFolder.class.getName()).log(Level.SEVERE, null, ex);
}
}
OUTPUT:
Then you can perform your calculation based on the given paths.

Exception in thread "main" java.lang.StackOverflowError at java.io.File.<init>(Unknown Source)

I have the following code. I cant find the source for the problem.
public class ConsoleApp1 {
public static void main(String[] args) {
allFiles("C:\\");
}
private static void allFiles(String root) {
File ro = new File("e:\\");
File[] flist = ro.listFiles();
for (File i: flist) {
if (i.isFile()) {
System.out.println(i.getAbsolutePath());
} else {
allFiles(i.getAbsolutePath());
}
}
}
}
I am in the future going to add this to swing application, A call from button will get the list and populate them into a JTree any advice on that would also help.

You are recursively calling File constructor on E:\. You will eventually get stackoverflow error due to infinite loop. Change the constructor argument to use root variable.
private static void allFiles(String root) {
File ro = new File(root);
File[] flist = ro.listFiles();
for(File i : flist){
if(i.isFile()) {
System.out.println(i.getAbsolutePath());
}
else {
allFiles(i.getAbsolutePath());
}
}
}

"Too many open files in system" failure while listing a recursive directory structure

I've implemented (in Java) a fairly straightforward Iterator to return the names of the files in a recursive directory structure, and after about 2300 files it failed "Too many open files in system" (the failure was actually in trying to load a class, but I assume the directory listing was the culprit).
The data structure maintained by the iterator is a Stack holding the contents of the directories that are open at each level.
The actual logic is fairly basic:
private static class DirectoryIterator implements Iterator<String> {
private Stack<File[]> directories;
private FilenameFilter filter;
private Stack<Integer> positions = new Stack<Integer>();
private boolean recurse;
private String next = null;
public DirectoryIterator(Stack<File[]> directories, boolean recurse, FilenameFilter filter) {
this.directories = directories;
this.recurse = recurse;
this.filter = filter;
positions.push(0);
advance();
}
public boolean hasNext() {
return next != null;
}
public String next() {
String s = next;
advance();
return s;
}
public void remove() {
throw new UnsupportedOperationException();
}
private void advance() {
if (directories.isEmpty()) {
next = null;
} else {
File[] files = directories.peek();
while (positions.peek() >= files.length) {
directories.pop();
positions.pop();
if (directories.isEmpty()) {
next = null;
return;
}
files = directories.peek();
}
File nextFile = files[positions.peek()];
if (nextFile.isDirectory()) {
int p = positions.pop() + 1;
positions.push(p);
if (recurse) {
directories.push(nextFile.listFiles(filter));
positions.push(0);
advance();
} else {
advance();
}
} else {
next = nextFile.toURI().toString();
count++;
if (count % 100 == 0) {
System.err.println(count + " " + next);
}
int p = positions.pop() + 1;
positions.push(p);
}
}
}
}
I would like to understand how many "open files" this requires. Under what circumstances is this algorithm "opening" a file, and when does it get closed again?
I've seen some neat code using Java 7 or Java 8, but I'm constrained to Java 6.

When you call nextFile.listFiles(), an underlying file descriptor is opened to read the directory. There is no way to explicitly close this descriptor, so you are relying on garbage collection. As your code descends a deep tree, it is essentially collecting a stack of nextFile instances which can't be garbaged collected.
Step 1: set nextFile = null before calling advance(). This releases the object for garbage collection.
Step 2: you may need to call System.gc() after nulling nextFile to encourage quick garbage collection. Unfortunately, there is no way to force GC.
Step 3: you may need to increase the open file limit on your operating system. On Linux this may be done with ulimit(1).
If you can migrate to Java 7 or later, then DirectoryStream will solve your problem. Instead of using nextFile.listFiles(), use Files.newDirectoryStream(nextFile.toPath()) to get a DirectoryStream. You can then iterate over the stream and then close() it to release the operating system resources. Each returned path can be converted back to a file with toFile(). However you might like to refactor to use just Path instead of File.

Thanks everyone for the help and advice. I established that the problem is actually in what is being done with the files after they are returned by the iterator: the "client" code is opening the files as they are delivered, and is not tidying up properly. It's complicated by the fact that the files coming back are actually being processed in parallel.
I've also rewritten the DireectoryIterator, which I share incase anyone is interested:
private static class DirectoryIterator implements Iterator<String> {
private Stack<Iterator<File>> directories;
private FilenameFilter filter;
private boolean recurse;
private String next = null;
public DirectoryIterator(Stack<Iterator<File>> directories, boolean recurse, FilenameFilter filter) {
this.directories = directories;
this.recurse = recurse;
this.filter = filter;
advance();
}
public boolean hasNext() {
return next != null;
}
public String next() {
String s = next;
advance();
return s;
}
public void remove() {
throw new UnsupportedOperationException();
}
private void advance() {
if (directories.isEmpty()) {
next = null;
} else {
Iterator<File> files = directories.peek();
while (!files.hasNext()) {
directories.pop();
if (directories.isEmpty()) {
next = null;
return;
}
files = directories.peek();
}
File nextFile = files.next();
if (nextFile.isDirectory()) {
if (recurse) {
directories.push(Arrays.asList(nextFile.listFiles(filter)).iterator());
}
advance();
} else {
next = nextFile.toURI().toString();
}
}
}
}

Recursively process same files inside many sub directories in Java

Hi I would like to process files inside many sub directories using Java. Psuedo code would be
while(mainDir.hasMoreDirectory())
{
getFilesFromCurrentDirectory()
passThoseFilesAsArgumentToProcess()
}
I am currently using the following code
public void list(File file) {
System.out.println(file.getName());
File[] children = file.listFiles();
for (File child : children) {
list(child);
}
}
Above code just lists files. Other thing I can do is I have to store list of files and directories in a list and then process in another loop. But I am not able to come up with what I want as show in pseudo code. I am new to Files Directories please help. Thanks in advance.

If you are using Java 7, you can harness the enhanced functionality of NIO in the form of the Files.walkFileTree method. Traversing the file system has never been easier in Java.
There is a short tutorial on it's usage here.
It implements the visitor pattern so you don't need to worry about the traversal algorithm itself, only specify what you want to do with each entry.

When traveling a directory tree in Java 7 use the Paths and Files functionality. They not only ease reading of directories and files, they're way faster then the "old" File way.
Assume you have two directories: mainDirand otherDirand you want to walk thru all directories of mainDir down to its leaves. With each entry in maiondir (file, sub-directory, symbolic link, ...) you want to compare this entry and its attributes (size, modification time, ...) against the entry at the same position in the otherDir.
Then this would be your code:
public final void test() throws IOException, InterruptedException {
final Path mainDir = Paths.get("absolute path to your main directory to read from");
final Path otherDir = Paths.get("absolute path to your other directory to compare");
// Walk thru mainDir directory
Files.walkFileTree(mainDir, new FileVisitor<Path>() {
#Override
public FileVisitResult preVisitDirectory(Path path,
BasicFileAttributes atts) throws IOException {
return visitFile(path, atts);
}
#Override
public FileVisitResult visitFile(Path path, BasicFileAttributes mainAtts)
throws IOException {
// I've seen two implementations on windows and MacOSX. One has passed the relative path, one the absolute path.
// This works in both cases
Path relativePath = mainDir.relativize(mainDir.resolve(path));
BasicFileAttributes otherAtts = Files.readAttributes(otherDir.resolve(relativePath), BasicFileAttributes.class);
// Do your comparison logic here:
compareEntries(mainDir, otherDir, relativePath, mainAtts, otherAtts);
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult postVisitDirectory(Path path,
IOException exc) throws IOException {
// TODO Auto-generated method stub
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult visitFileFailed(Path path, IOException exc)
throws IOException {
exc.printStackTrace();
// If the root directory has failed it makes no sense to continue
return (path.equals(mainDir))? FileVisitResult.TERMINATE:FileVisitResult.CONTINUE;
}
});
}
What it not does:
Find entries that do exist in otherDirbut not in maindir
Path and BasicFileAttributes are not Serializable, so there's no easy way to do this walk on two different machines.

A method like this would return you a List of all the files recursively within a directory. You can either operate on the returned List or replace the rtn.add calls with your processing.
Beware that this method doesn't have anything to stop it getting stuck in circular symlinks.
public static List<File> getFilesRecursive(File s)
{
ArrayList<File> rtn = new ArrayList<File>();
File[] contents = s.listFiles();
for(int i = 0; i<contents.length; i++)
{
if(contents[i].isDirectory()){
rtn.addAll(getFilesRecursive(contents[i]));
}else{
rtn.add(contents[i]);
}
}
return rtn;
}

Maybe this piece of code helps you:
public void traverse(String path) {
File root = new File(path);
File[] list = root.listFiles();
if (list == null) return;
for (File file : list) {
if (file.isDirectory()) {
traverse(file.getAbsolutePath());
System.out.println("Directory: " + file.getAbsoluteFile());
} else {
System.out.println("File: " + file.getAbsoluteFile());
}
}
}

Will following do?
public void list(File file) {
File[] children = file.listFiles();
if (children != null) {
process(children);
for (File child : children) {
if (child.isDirectory()) {
list(child);
}
}
} else {
process(new File[]{file});
}
}
private void process(File[] children) {
for (File child : children) {
if (child.isFile()) {
// process normal file
}
}
}

private static List<File> allFiles = new ArrayList<File>();
private static void processFiles(String rootDirectory) {
File rootDir = new File(rootDirectory);
if (rootDir.exists()) {
traverseDirectories(rootDir);
}
}
private static void traverseDirectories(File file) {
// add all files and directories to list.
allFiles.add(file);
if (file.isDirectory()) {
File[] fileList = file.listFiles();
for (File fileHandle : fileList) {
traverseDirectories(fileHandle);
}
} else {
// call to process file
System.out.println("Call to process file " + file.getAbsolutePath());
}
}

Not able to load properties file kept outside the code

import static com.crawler.constants.CrawlerConstants;
import static com.crawler.constants.CrawlerConstants.CRAWLER4J;
import java.util.Properties;
public final class Configurations {
private static Properties prop = new Properties();
public static String getStringProperty(String key, String defaultValue) {
if (prop == null || prop.getProperty(key) == null) {
return defaultValue;
}
return prop.getProperty(key);
}
public static int getIntProperty(String key, int defaultValue) {
if (prop == null || prop.getProperty(key) == null) {
return defaultValue;
}
return Integer.parseInt(prop.getProperty(key));
}
public static short getShortProperty(String key, short defaultValue) {
if (prop == null || prop.getProperty(key) == null) {
return defaultValue;
}
return Short.parseShort(prop.getProperty(key));
}
public static long getLongProperty(String key, long defaultValue) {
if (prop == null || prop.getProperty(key) == null) {
return defaultValue;
}
return Long.parseLong(prop.getProperty(key));
}
public static boolean getBooleanProperty(String key, boolean defaultValue) {
if (prop == null || prop.getProperty(key) == null) {
return defaultValue;
}
return prop.getProperty(key).toLowerCase().trim().equals("true");
}
static {
try {
prop.load(Configurations.class.getClassLoader()
.getResourceAsStream(CONFIG_DIR + "/" + CRAWLER4J));
} catch (Exception e) {
prop = null;
System.err.println("WARNING: Could not find crawler4j.properties file in class path. I will use the default values.");
}
}
}
In my try loop above it is not loading crawler4j.properties file. As previously it was like this-
try {
prop.load(Configurations.class.getClassLoader()
.getResourceAsStream("crawler4j.properties"));
}
so it was able to load it directly from the src/main/resources folder. But I want to load this crawler4j.properties file kept outside the code in different directory. So I have stored crawler4j.properties file in /my/dir/crawler4j.properties and this is try loop that I want to modify-
try {
prop.load(Configurations.class.getClassLoader()
.getResourceAsStream(CONFIG_DIR + "/" + CRAWLER4J_PROP));
}
And CONFIG_DIR contains \my\dir and CRAWLER4J has crawler4j.properties but somehow it is not loading and it is going to catch exception block. Any suggestions why is it happening.

If you try to load something as a resource, it must be on the classpath--that's the definition of a resource (give or take).
To load it from a filesystem path, use load(InputStream) (or load(Reader)).

It's most likely a misconfigured path, in your catch try appending in your System.err statement:
e.getMessage();

I think you might find you need to escape the "/" so try using "//". I'm not sure if this is the issue but I had it awhile back and was pulling my hair out trying to figure out why it worked some places and not others.

So, what kind of exception do you get? FileNotFound? Is this a part of web app or regular java application?
Make sure you have read rights to read the properties file in the directory.
Edit:
getResourceAsStream returns null if it doesn't find the resource, so as suggested by others:
1. Make sure the properties file is on the classpath.
2. Escape the / -
prop.load(Configurations.class.getClassLoader()
.getResourceAsStream(CONFIG_DIR + "//" + CRAWLER4J_PROP));