in one requirement, i need to copy multiple files from one location to another network location.
let assume that i have the following files present in the /src location.
a.pdf, b.pdf, a.doc, b.doc, a.txt and b.txt
I need to copy a.pdf, a.doc and a.txt files atomically into /dest location at once.
Currently i am using Java.nio.file.Files packages and code as follows
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1 = Paths.get("/dest/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2 = Paths.get("/dest/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3 = Paths.get("/dest/a.txt");
Files.copy(srcFile1, destFile1);
Files.copy(srcFile2, destFile2);
Files.copy(srcFile3, destFile3);
but this process the file are copied one after another.
As an alternate to this, in order to make whole process as atomic,
i am thinking of zipping all the files and move to /dest and unzip at the destination.
is this approach is correct to make whole copy process as atomic ? any one experience similar concept and resolved it.
is this approach is correct to make whole copy process as atomic ? any one experience similar concept and resolved it.
You can copy the files to a new temporary directory and then rename the directory.
Before renaming your temporary directory, you need to delete the destination directory
If other files are already in the destination directory that you don't want to overwrite, you can move all files from the temporary directory to the destination directory.
This is not completely atomic, however.
With removing /dest:
String tmpPath="/tmp/in/same/partition/as/source";
File tmp=new File(tmpPath);
tmp.mkdirs();
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1 = Paths.get(tmpPath+"/dest/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2 = Paths.get(tmpPath+"/dest/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3 = Paths.get(tmpPath+"/dest/a.txt");
Files.copy(srcFile1, destFile1);
Files.copy(srcFile2, destFile2);
Files.copy(srcFile3, destFile3);
delete(new File("/dest"));
tmp.renameTo("/dest");
void delete(File f) throws IOException {
if (f.isDirectory()) {
for (File c : f.listFiles())
delete(c);
}
if (!f.delete())
throw new FileNotFoundException("Failed to delete file: " + f);
}
With just overwriting the files:
String tmpPath="/tmp/in/same/partition/as/source";
File tmp=new File(tmpPath);
tmp.mkdirs();
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1=paths.get("/dest/a.pdf");
Path tmp1 = Paths.get(tmpPath+"/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2=Paths.get("/dest/a.doc");
Path tmp2 = Paths.get(tmpPath+"/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3=Paths.get("/dest/a.txt");
Path destFile3 = Paths.get(tmpPath+"/a.txt");
Files.copy(srcFile1, tmp1);
Files.copy(srcFile2, tmp2);
Files.copy(srcFile3, tmp3);
//Start of non atomic section(it can be done again if necessary)
Files.deleteIfExists(destFile1);
Files.deleteIfExists(destFile2);
Files.deleteIfExists(destFile2);
Files.move(tmp1,destFile1);
Files.move(tmp2,destFile2);
Files.move(tmp3,destFile3);
//end of non-atomic section
Even if the second method contains a non-atomic section, the copy process itself uses a temporary directory so that the files are not overwritten.
If the process aborts during moving the files, it can easily be completed.
See https://stackoverflow.com/a/4645271/10871900 as reference for moving files and https://stackoverflow.com/a/779529/10871900 for recursively deleting directories.
First there are several possibilities to copy a file or a directory. Baeldung gives a very nice insight into different possibilities. Additionally you can also use the FileCopyUtils from Spring. Unfortunately, all these methods are not atomic.
I have found an older post and adapt it a little bit. You can try using the low-level transaction management support. That means you make a transaction out of the method and define what should be done in a rollback. There is also a nice article from Baeldung.
#Autowired
private PlatformTransactionManager transactionManager;
#Transactional(rollbackOn = IOException.class)
public void copy(List<File> files) throws IOException {
TransactionDefinition transactionDefinition = new DefaultTransactionDefinition();
TransactionStatus transactionStatus = transactionManager.getTransaction(transactionDefinition);
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
#Override
public void afterCompletion(int status) {
if (status == STATUS_ROLLED_BACK) {
// try to delete created files
}
}
});
try {
// copy files
transactionManager.commit(transactionStatus);
} finally {
transactionManager.rollback(transactionStatus);
}
}
Or you can use a simple try-catch-block. If an exception is thrown you can delete the created files.
Your question lacks the goal of atomicity. Even unzipping is never atomic, the VM might crash with OutOfMemoryError right in between inflating the blocks of the second file. So there's one file complete, a second not and a third entirely missing.
The only thing I can think of is a two phase commit, like all the suggestions with a temporary destination that suddenly becomes the real target. This way you can be sure, that the second operation either never occurs or creates the final state.
Another approach would be to write a sort of cheap checksum file in the target afterwards. This would make it easy for an external process to listen for creation of such files and verify their content with the files found.
The latter would be the same like offering the container/ ZIP/ archive right away instead of piling files in a directory. Most archives have or support integrity checks.
(Operating systems and file systems also differ in behaviour if directories or folders disappear while being written. Some accept it and write all data to a recoverable buffer. Others still accept writes but don't change anything. Others fail immediately upon first write since the target block on the device is unknown.)
FOR ATOMIC WRITE:
There is no atomicity concept for standard filesystems, so you need to do only single action - that would be atomic.
Therefore, for writing more files in an atomic way, you need to create a folder with, let's say, the timestamp in its name, and copy files into this folder.
Then, you can either rename it to the final destination or create a symbolic link.
You can use anything similar to this, like file-based volumes on Linux, etc.
Remember that deleting the existing symbolic link and creating a new one will never be atomic, so you would need to handle the situation in your code and switch to the renamed/linked folder once it's available instead of removing/creating a link. However, under normal circumstances, removing and creating a new link is a really fast operation.
FOR ATOMIC READ:
Well, the problem is not in the code, but on the operation system/filesystem level.
Some time ago, I got into a very similar situation. There was a database engine running and changing several files "at once". I needed to copy the current state, but the second file was already changed before the first one was copied.
There are two different options:
Use a filesystem with support for snapshots. At some moment, you create a snapshot and then copy files from it.
You can lock the filesystem (on Linux) using fsfreeze --freeze, and unlock it later with fsfreeze --unfreeze. When the filesystem is frozen, you can read the files as usual, but no process can change them.
None of these options worked for me as I couldn't change the filesystem type, and locking the filesystem wasn't possible (it was root filesystem).
I created an empty file, mount it as a loop filesystem, and formatted it. From that moment on, I could fsfreeze just my virtual volume without touching the root filesystem.
My script first called fsfreeze --freeze /my/volume, then perform the copy action, and then called fsfreeze --unfreeze /my/volume. For the duration of the copy action, the files couldn't be changed, and so the copied files were all exactly from the same moment in time - for my purpose, it was like an atomic operation.
Btw, be sure to not fsfreeze your root filesystem :-). I did, and restart is the only solution.
DATABASE-LIKE APPROACH:
Even databases cannot rely on atomic operations, and so they first write the change to WAL (write-ahead log) and flush it to the storage. Once it's flushed, they can apply the change to the data file.
If there is any problem/crash, the database engine first loads the data file and checks whether there are some unapplied transactions in WAL and eventually apply them.
This is also called journaling, and it's used by some filesystems (ext3, ext4).
I hope this solution would be useful : as per my understanding you need to copy the files from one directory to another directory.
so my solution is as follows:
Thank You.!!
public class CopyFilesDirectoryProgram {
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
String sourcedirectoryName="//mention your source path";
String targetdirectoryName="//mention your destination path";
File sdir=new File(sourcedirectoryName);
File tdir=new File(targetdirectoryName);
//call the method for execution
abc (sdir,tdir);
}
private static void abc(File sdir, File tdir) throws IOException {
if(sdir.isDirectory()) {
copyFilesfromDirectory(sdir,tdir);
}
else
{
Files.copy(sdir.toPath(), tdir.toPath());
}
}
private static void copyFilesfromDirectory(File source, File target) throws IOException {
if(!target.exists()) {
target.mkdir();
}else {
for(String items:source.list()) {
abc(new File(source,items),new File(target,items));
}
}
}
}
I got a problem when i want to update my (lets say orderNumber) and i try to renaming the file of image. lets say the original file ini MainImage15 i want to rename to MainImage16 with Files.Copy or Files.Move and after that i try to delete after succesful copying. and i got the error like this
java.nio.file.FileSystemException: C:\Users\User\apache\webapps\Promotion\030000\MainImage15.jpg: The process cannot access the file because it is being used by another process.
public void renameFileToFileSystem(final String fileName, final String oldFileLocation, final String newFileLocation) {
Path source = Paths.get(oldFileLocation);
Path destination = Paths.get(newFileLocation);
try {
Files.copy(source, destination, StandardCopyOption.REPLACE_EXISTING);
Files.delete(source);
} catch (final IOException ioException) {
throw new ContentManagementException(fileName, ioException.getMessage());
}
}
i dont knwo what to do. should i use buffer close? but i just renaming the file . thankyou.
I don't think Files.copy keeps the file handle open.
To make sure its true - remove the line Files.copy and rerun - the chances are that you still won't be able to delete the file.
So you must find who keeps the handle busy. There are basically two possibilities:
Its somewhere else in your code
Its some kind of external process (antivirus, another application that you've used to render the image and so forth). You can use Process Explorer that will help to find the process that keeps the handle.
Why am I able to delete a file in my Java code despite the Tomcat user not having the deletion permissions?
My server is running the following code, which deletes and recreates a file if it exists:
File fileCSV = new File(filePath);
try {
if (fileCSV.exists()) {
fileCSV.delete();
}
fileCSV.createNewFile();
} catch (IOException ex) {
throw new FooImportException("Error creating new file");
}
It is able to delete the file despite the user used by the server not having deletion permissions - only read and write permissions.
I am certain that these are the relevant permissions, as the code fails on the file creation line without the "Create files / Write data" permissions. However, it does not fail on the deletion line when lacking the "Delete" permission. What might be the reasoning for this?
According to the JavaDocs for File#delete
public boolean delete() Deletes the file or directory denoted by
this abstract pathname. If this pathname denotes a directory, then the
directory must be empty in order to be deleted. Note that the Files
class defines the delete method to throw an IOException when a file
cannot be deleted. This is useful for error reporting and to diagnose
why a file cannot be deleted.
Returns: true if and only if the file or directory is successfully
deleted; false otherwise Throws: SecurityException - If a
security manager exists and its
SecurityManager.checkDelete(java.lang.String) method denies delete
access to the file
So, File#delete does not actually throw an Exception when the file can't be deleted, but instead returns a boolean based on the success of the operation.
If the Exception is important to you, then you should use Files#delete instead.
It's important to note - this only solves the question of "why does it not fail" based on the available code, not the question of "would it fail" based on the available file permissions
For example I want to save large file (3G+) from web. Code sample:
try {
Files.copy(inputstream, destFilePath);
} catch (IOException ex) {
Files.deleteIfExists(destFilePath);
} finally {
IOUtils.closeQuietly(inputstream);
}
According to JavaDoc for deleteIfExists:
On some operating systems it may not be possible to remove a file when
* it is open and in use by this Java virtual machine or other programs.
Is it safe to delete file in a such way? Files.copy release output stream even error occurs, does it guarantee that JVM released lock on the file?
file should not be in use in your case. Take into consideration that Files.copy does not ask you for an OutputStream or File, just for a path. It would be weird that it could leave a file descriptor open upon exit, no matter if exception or not; the File api would be broken in that case IMO. In any case, File javadoc would inform you of that possibility.
In any case, if you want to minimise possibilities of file not being deleted, you can add also a file.deleteOnExit(); then when jvm terminates, it will do another try to delete file (unless jvm terminates abnormally).
I have eclipse plugin jface application.
A thread writes file via BufferedWriter.
After writing is done I close the buffer after that I try to rename the file.
But sometimes file is not renamed!
I tried to add some Thread.Sleep(BIG_NUMBER) between couple of retries this didn't help.
It looks like the file getting some kind of lock. (when I kill the jvm I can rename the file).
Is there something I can do?
OS: Windows XP, windows 7
JAVA version: 1.5
File.RenameTo() is platform dependent and relies on a few conditions to be met in order to succesfully rename a file, a better alternative is using
Path source = currentFile.toPath();
try {
Files.move(source, source.resolveSibling(formattedName));
} catch (IOException e) {
e.printStackTrace();
}
Read more here.
From the javadocs:
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed if a file with the destination abstract pathname
already exists. The return value should always be checked to make sure
that the rename operation was successful.
Note that the Files class defines the move method to move or rename a file in a platform independent manner.
For the File.renameTo() to work,The file will need to be somehow writable by external applications.
You can also do something like below:
File o=new File("oldFile.txt");
File n=new File("newFile.txt");
n.delete();
o.renameTo(n);
n.delete() : We need to delete the file(new.txt) if exists.
o.rename(n) : so that the file(old.txt) is renamed as new.txt
How to find out why renameTo() failed?
Reliable File.renameTo() alternative on Windows?
http://www.bigsoft.co.uk/blog/index.php/2010/02/02/file-renameto-always-fails-on-windows
We have had issues under Windows 7 with UAC and unexpected file permissions. File#canWrite will return true even though any attempts to perform file I/O will fail.
Make sure the file you are trying to rename actually exists
Make sure that the location you are attempting to write the file (or rename the file) to is accessible. We write a simple text file to the location, check to see if it exists and that it's contents is correct (we're paranoid) before we attempt any further I/O.
This is working fine for me. Rename is done using two steps but don't forget to set permissions in manifest.xml with:
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_INTERNAL_STORAGE" />
public boolean RenameFile(String from, String to) {
to.replace(" ", ""); // clear all spaces within file name
File oldfile = new File(from);
File newfile = new File(to);
File tempfile = new File(to + ".tmp"); // add extension .tmp
oldfile.renameTo(tempfile);
return (tempfile.renameTo(newfile));
}