I have a program that goes to a few directories from time to time and and do some kind of processing to the files in those directories.
The problem is from time to time (each two or three days) the program is reaching the OS open files limit.
It is a spring-boot application running in a RHEL 7.
the method that get the files is this:
public File[] getFiles(String dir, int numberOfFiles) throws Exception {
final Path baseDir = Paths.get(dir);
List<File> filesFromPath = new ArrayList<File>();
File[] files = null;
final BiPredicate<Path, BasicFileAttributes> predicate = (path, attrs) -> attrs.isRegularFile()
&& String.valueOf(path).endsWith(".xml");
List<Path> result;
try (Stream<Path> fileStream = Files.find(baseDir, 1, predicate).limit(numberOfFiles).onClose(() -> LOG.debug("Closing file stream."))){
result = fileStream.collect(Collectors.toList());
result.forEach(path -> {
path.toString();
File file = path.toFile();
LOG.info("adding {} to process.", file.getName());
filesFromPath.add(file);
});
if (filesFromPath != null && !filesFromPath.isEmpty()) {
files = filesFromPath.toArray(new File[filesFromPath.size()]);
}
} catch (Exception e) {
LOG.error("Error during file opening/closing", e);
}
if (files != null) {
return files;
}
return new File[0];
}
I am using the lsof command to see how many opened files I have, and the list of directories is always growing.
I added a log in the onClise method and this is being called all times I open a stream.
Shouldn't the try with resources, close the stream?
[EDIT]
There is also another peace of code that move processed files to another folder. This code does not use stream, and I couldn't found out what is wrong with it, besides the fact it is ugly.
public void move(File file, String archivePath) throws IOException {
File backupFile = new File(archivePath);
if (!backupFile.exists()) {
backupFile.mkdirs();
}
Path source = file.toPath();
if (file.exists()) {
Path target = Paths.get(archivePath + File.separator + file.getName());
Files.move(source, target, StandardCopyOption.REPLACE_EXISTING);
LOG.info("file {} moved to {}", file, archivePath);
} else {
LOG.info("unable to move the file: {} because it was already moved to {}", file, archivePath);
}
}
[EDIT 2]
and all files are being processed like this:
private void processFile(File[] files) throws Exception {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
XPath xpathParser = XPathFactory.newInstance().newXPath();
for (int i = 0; i < files.length; i++) {
File file = files[i];
Document doc = db.parse(file);
// DO STUFF
fileUtils.move(file, processedPath);
}
}
Thanks.
I hit the same issue. In the case of calls to "parse" there is very much files open and I clearly see the implementation opens an input stream but may not close it in case of a failure.
Related
File f=new File("C:/");
File fList[] = f.listFiles();
When i use this it list all system file as well as hidden files.
and this cause null pointer exception when i use it to show in jTree like this:
public void getList(DefaultMutableTreeNode node, File f) {
if(f.isDirectory()) {
DefaultMutableTreeNode child = new DefaultMutableTreeNode(f);
node.add(child);
File fList[] = f.listFiles();
for(int i = 0; i < fList.length; i++)
getList(child, fList[i]);
}
}
What should i do so that it do not give NullPointerException and show only non hidden and non system files in jTree?
Do this for hidden files:
File root = new File(yourDirectory);
File[] files = root.listFiles(new FileFilter() {
#Override
public boolean accept(File file) {
return !file.isHidden();
}
});
This will not return hidden files.
As for system files, I believe that is a Windows concept and therefore might not be supported by File interface that tries to be system independent. You can use Command line commands though, if those exist.
Or use what #Reimeus had in his answer.
Possibly like
File root = new File("C:\\");
File[] files = root.listFiles(new FileFilter() {
#Override
public boolean accept(File file) {
Path path = Paths.get(file.getAbsolutePath());
DosFileAttributes dfa;
try {
dfa = Files.readAttributes(path, DosFileAttributes.class);
} catch (IOException e) {
// bad practice
return false;
}
return (!dfa.isHidden() && !dfa.isSystem());
}
});
DosFileAttributes was introduced in Java 7.
If running under Windows, Java 7 introduced DosFileAttributes which enables system and hidden files to be filtered. This can be used in conjunction with a FileFilter
Path srcFile = Paths.get("myDirectory");
DosFileAttributes dfa = Files.readAttributes(srcFile, DosFileAttributes.class);
System.out.println("System File? " + dfa.isSystem());
System.out.println("Hidden File? " + dfa.isHidden());
If you are trying to list all files in C:/ please keep in mind that there are other files also which are neither hidden nor system files, but that still won't open because they require special privileges/permissions. So:
String[] files = file.list();
if (files!=null) {
for (String f : files) open(f);
}
So just compare if the array is null or not and design your recursion in such a way that it just skips those files whose array for the list() function is null.
private void nodes(DefaultMutableTreeNode top, File f) throws IOException {
if (f.isDirectory()) {
File[] listFiles = f.listFiles();
if (listFiles != null) {
DefaultMutableTreeNode b1[] = new DefaultMutableTreeNode[listFiles.length];
for (int i = 0; i < b1.length; i++) {
b1[i] = new DefaultMutableTreeNode(listFiles[i].toString());
top.add(b1[i]);
File g = new File(b1[i].toString());
nodes(b1[i], g);
}
}
}
Here is the code I used to create a window file explorer using jtree.
How do you move a file from one location to another? When I run my program any file created in that location automatically moves to the specified location. How do I know which file is moved?
myFile.renameTo(new File("/the/new/place/newName.file"));
File#renameTo does that (it can not only rename, but also move between directories, at least on the same file system).
Renames the file denoted by this abstract pathname.
Many aspects of the behavior of this method are inherently platform-dependent: The rename operation might not be able to move a file from one filesystem to another, it might not be atomic, and it might not succeed if a file with the destination abstract pathname already exists. The return value should always be checked to make sure that the rename operation was successful.
If you need a more comprehensive solution (such as wanting to move the file between disks), look at Apache Commons FileUtils#moveFile
With Java 7 or newer you can use Files.move(from, to, CopyOption... options).
E.g.
Files.move(Paths.get("/foo.txt"), Paths.get("bar.txt"), StandardCopyOption.REPLACE_EXISTING);
See the Files documentation for more details
Java 6
public boolean moveFile(String sourcePath, String targetPath) {
File fileToMove = new File(sourcePath);
return fileToMove.renameTo(new File(targetPath));
}
Java 7 (Using NIO)
public boolean moveFile(String sourcePath, String targetPath) {
boolean fileMoved = true;
try {
Files.move(Paths.get(sourcePath), Paths.get(targetPath), StandardCopyOption.REPLACE_EXISTING);
} catch (Exception e) {
fileMoved = false;
e.printStackTrace();
}
return fileMoved;
}
File.renameTo from Java IO can be used to move a file in Java. Also see this SO question.
To move a file you could also use Jakarta Commons IOs FileUtils.moveFile
On error it throws an IOException, so when no exception is thrown you know that that the file was moved.
Just add the source and destination folder paths.
It will move all the files and folder from source folder to
destination folder.
File destinationFolder = new File("");
File sourceFolder = new File("");
if (!destinationFolder.exists())
{
destinationFolder.mkdirs();
}
// Check weather source exists and it is folder.
if (sourceFolder.exists() && sourceFolder.isDirectory())
{
// Get list of the files and iterate over them
File[] listOfFiles = sourceFolder.listFiles();
if (listOfFiles != null)
{
for (File child : listOfFiles )
{
// Move files to destination folder
child.renameTo(new File(destinationFolder + "\\" + child.getName()));
}
// Add if you want to delete the source folder
sourceFolder.delete();
}
}
else
{
System.out.println(sourceFolder + " Folder does not exists");
}
Files.move(source, target, REPLACE_EXISTING);
You can use the Files object
Read more about Files
You could execute an external tool for that task (like copy in windows environments) but, to keep the code portable, the general approach is to:
read the source file into memory
write the content to a file at the new location
delete the source file
File#renameTo will work as long as source and target location are on the same volume. Personally I'd avoid using it to move files to different folders.
Try this :-
boolean success = file.renameTo(new File(Destdir, file.getName()));
Wrote this method to do this very thing on my own project only with the replace file if existing logic in it.
// we use the older file i/o operations for this rather than the newer jdk7+ Files.move() operation
private boolean moveFileToDirectory(File sourceFile, String targetPath) {
File tDir = new File(targetPath);
if (tDir.exists()) {
String newFilePath = targetPath+File.separator+sourceFile.getName();
File movedFile = new File(newFilePath);
if (movedFile.exists())
movedFile.delete();
return sourceFile.renameTo(new File(newFilePath));
} else {
LOG.warn("unable to move file "+sourceFile.getName()+" to directory "+targetPath+" -> target directory does not exist");
return false;
}
}
Please try this.
private boolean filemovetoanotherfolder(String sourcefolder, String destinationfolder, String filename) {
boolean ismove = false;
InputStream inStream = null;
OutputStream outStream = null;
try {
File afile = new File(sourcefolder + filename);
File bfile = new File(destinationfolder + filename);
inStream = new FileInputStream(afile);
outStream = new FileOutputStream(bfile);
byte[] buffer = new byte[1024 * 4];
int length;
// copy the file content in bytes
while ((length = inStream.read(buffer)) > 0) {
outStream.write(buffer, 0, length);
}
// delete the original file
afile.delete();
ismove = true;
System.out.println("File is copied successful!");
} catch (IOException e) {
e.printStackTrace();
}finally{
inStream.close();
outStream.close();
}
return ismove;
}
I am trying to move files from one directory to another delete that file from source directory after moving.
for (File file : files) {
if (file != null) {
boolean status = moveFile(file, filePath, name, docGroupId);
if (status) {
//some operations....
}
}
}
public static boolean moveFile(final File file, final String filePath, final String groupName, Integer docGroupId) {
// TODO Auto-generated method stub
String selectedDirectory = filePath + File.separator + groupName;
InputStream in = null;
OutputStream out = null;
try {
if (!file.isDirectory()) {
File dir = new File(selectedDirectory);
if (!dir.exists()) {
dir.mkdirs();
}
String newFilString = dir.getAbsolutePath() +
File.separator + file.getName();
File newFile = new File(newFilString);
in = new FileInputStream(file);
out = new FileOutputStream(newFile);
byte[] moveBuff = new byte[1024];
int butesRead;
while ((butesRead = in.read(moveBuff)) > 0) {
out.write(moveBuff, 0, butesRead);
}
}
in.close();
out.close();
if(file.delete())
return true;
} catch (Exception e) {
return false;
}
}
The program works on Linux-Ubuntu and all files are moved to another directory and deleted from source directory, but in Windows system all files are moved but failed to delete one or two files from source directory. Please note that while debugging the program is working fine.
Consider using Files.delete instead of File.delete. The javadoc says:
Note that the Files class defines the delete method to throw an IOException when a file cannot be deleted. This is useful for error reporting and to diagnose why a file cannot be deleted.
This should provide the information necessary to diagnose the problem.
So, if problem comes with delete, possible explanations:
you do file.delete() on every files and directories. How do you know the directory is empty ? If not, it will fail, then what happen to next instructions ?
file deletion is OS-dependant. On Windows, you can have many security issues, depending on which user, which rights, which location. You should check with a file-delete-alone program;
last: files can be locked by other programs (even explorer), it is also OS-dependant.
You don't need any of this if the source and target are in the same file system. Just use File.renameTo().
I'm curious as if there's a way to define a Parent-folder, then have a program cycle through all of the files, and sub-folders, and rename the file extension.
I know this can be done in the command prompt using the command "*.ext *.newext" however that's not a possible solution for me and I need to rename 2,719 file extentions that are nested inside of this folder.
Yes, you can do it. Here's an example:
// java 6
File parentDir = new File("..");
System.out.println(parentDir.getAbsolutePath());
final File[] files = parentDir.listFiles();
System.out.println(Arrays.toString(files));
// java 7+
File parentDir = new File("..");
try {
Files.walkFileTree(parentDir.toPath(), new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
if (file.toFile().renameTo(new File("othername.txt"))) {
return FileVisitResult.CONTINUE;
} else {
return FileVisitResult.TERMINATE;
}
}
});
} catch (IOException e) {
e.printStackTrace();
}
This one does not go through subdirs, but it is easy to modify that way.
Here's a simple function that should do the job for you. Sorry, if it's not the most elegant code -- my Java is a little rusty.
By default, it's recursive in its implementation; so be aware that it will affect all files of the specified type in the parent directory!
SwapFileExt params
path the parent directory you want to parse
cExt the extension type that you want to replace
nExt the desired extension type
NOTE: Both cExt and nExt are to be represented without the '.' (e.g. "txt", not ".txt")
public static void SwapFileExt(String path, String cExt, String nExt) {
File parentDir = new File(path);
File[] contents = parentDir.listFiles();
for (int i = 0; i < contents.length; i++) {
if (contents[i].isFile()) {
if (contents[i].toString().contains("." + cExt)) {
String item = contents[i].toString().replaceAll("." + cExt, "." + nExt);
contents[i].renameTo(new File(item));
}
} else if (contents[i].isDirectory()) {
SwapFileExt(contents[i].toString(), cExt, nExt);
}
}
}
I want to delete all files inside ABC directory.
When I tried with FileUtils.deleteDirectory(new File("C:/test/ABC/")); it also deletes folder ABC.
Is there a one liner solution where I can delete files inside directory but not directory?
import org.apache.commons.io.FileUtils;
FileUtils.cleanDirectory(directory);
There is this method available in the same file. This will also recursively deletes all sub-folders and files under them.
Docs: org.apache.commons.io.FileUtils.cleanDirectory
Do you mean like?
for(File file: dir.listFiles())
if (!file.isDirectory())
file.delete();
This will only delete files, not directories.
Peter Lawrey's answer is great because it is simple and not depending on anything special, and it's the way you should do it. If you need something that removes subdirectories and their contents as well, use recursion:
void purgeDirectory(File dir) {
for (File file: dir.listFiles()) {
if (file.isDirectory())
purgeDirectory(file);
file.delete();
}
}
To spare subdirectories and their contents (part of your question), modify as follows:
void purgeDirectoryButKeepSubDirectories(File dir) {
for (File file: dir.listFiles()) {
if (!file.isDirectory())
file.delete();
}
}
Or, since you wanted a one-line solution:
for (File file: dir.listFiles())
if (!file.isDirectory())
file.delete();
Using an external library for such a trivial task is not a good idea unless you need this library for something else anyway, in which case it is preferrable to use existing code. You appear to be using the Apache library anyway so use its FileUtils.cleanDirectory() method.
Java 8 Stream
This deletes only files from ABC (sub-directories are untouched):
Arrays.stream(new File("C:/test/ABC/").listFiles()).forEach(File::delete);
This deletes only files from ABC (and sub-directories):
Files.walk(Paths.get("C:/test/ABC/"))
.filter(Files::isRegularFile)
.map(Path::toFile)
.forEach(File::delete);
^ This version requires handling the IOException
Or to use this in Java 8:
try {
Files.newDirectoryStream( directory ).forEach( file -> {
try { Files.delete( file ); }
catch ( IOException e ) { throw new UncheckedIOException(e); }
} );
}
catch ( IOException e ) {
e.printStackTrace();
}
It's a pity the exception handling is so bulky, otherwise it would be a one-liner ...
public class DeleteFile {
public static void main(String[] args) {
String path="D:\test";
File file = new File(path);
File[] files = file.listFiles();
for (File f:files)
{if (f.isFile() && f.exists)
{ f.delete();
system.out.println("successfully deleted");
}else{
system.out.println("cant delete a file due to open or error");
} } }}
rm -rf was much more performant than FileUtils.cleanDirectory.
Not a one-liner solution but after extensive benchmarking, we found that using rm -rf was multiple times faster than using FileUtils.cleanDirectory.
Of course, if you have a small or simple directory, it won't matter but in our case we had multiple gigabytes and deeply nested sub directories where it would take over 10 minutes with FileUtils.cleanDirectory and only 1 minute with rm -rf.
Here's our rough Java implementation to do that:
// Delete directory given and all subdirectories and files (i.e. recursively).
//
static public boolean clearDirectory( File file ) throws IOException, InterruptedException {
if ( file.exists() ) {
String deleteCommand = "rm -rf " + file.getAbsolutePath();
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec( deleteCommand );
process.waitFor();
file.mkdirs(); // Since we only want to clear the directory and not delete it, we need to re-create the directory.
return true;
}
return false;
}
Worth trying if you're dealing with large or complex directories.
For deleting all files from directory say "C:\Example"
File file = new File("C:\\Example");
String[] myFiles;
if (file.isDirectory()) {
myFiles = file.list();
for (int i = 0; i < myFiles.length; i++) {
File myFile = new File(file, myFiles[i]);
myFile.delete();
}
}
Another Java 8 Stream solution to delete all the content of a folder, sub directories included, but not the folder itself.
Usage:
Path folder = Paths.get("/tmp/folder");
CleanFolder.clean(folder);
and the code:
public interface CleanFolder {
static void clean(Path folder) throws IOException {
Function<Path, Stream<Path>> walk = p -> {
try { return Files.walk(p);
} catch (IOException e) {
return Stream.empty();
}};
Consumer<Path> delete = p -> {
try {
Files.delete(p);
} catch (IOException e) {
}
};
Files.list(folder)
.flatMap(walk)
.sorted(Comparator.reverseOrder())
.forEach(delete);
}
}
The problem with every stream solution involving Files.walk or Files.delete is that these methods throws IOException which are a pain to handle in streams.
I tried to create a solution which is more concise as possible.
I think this will work (based on NonlinearFruit previous answer):
Files.walk(Paths.get("C:/test/ABC/"))
.sorted(Comparator.reverseOrder())
.map(Path::toFile)
.filter(item -> !item.getPath().equals("C:/test/ABC/"))
.forEach(File::delete);
Cheers!
package com;
import java.io.File;
public class Delete {
public static void main(String[] args) {
String files;
File file = new File("D:\\del\\yc\\gh");
File[] listOfFiles = file.listFiles();
for (int i = 0; i < listOfFiles.length; i++)
{
if (listOfFiles[i].isFile())
{
files = listOfFiles[i].getName();
System.out.println(files);
if(!files.equalsIgnoreCase("Scan.pdf"))
{
boolean issuccess=new File(listOfFiles[i].toString()).delete();
System.err.println("Deletion Success "+issuccess);
}
}
}
}
}
If you want to delete all files remove
if(!files.equalsIgnoreCase("Scan.pdf"))
statement it will work.