in one requirement, i need to copy multiple files from one location to another network location.
let assume that i have the following files present in the /src location.
a.pdf, b.pdf, a.doc, b.doc, a.txt and b.txt
I need to copy a.pdf, a.doc and a.txt files atomically into /dest location at once.
Currently i am using Java.nio.file.Files packages and code as follows
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1 = Paths.get("/dest/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2 = Paths.get("/dest/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3 = Paths.get("/dest/a.txt");
Files.copy(srcFile1, destFile1);
Files.copy(srcFile2, destFile2);
Files.copy(srcFile3, destFile3);
but this process the file are copied one after another.
As an alternate to this, in order to make whole process as atomic,
i am thinking of zipping all the files and move to /dest and unzip at the destination.
is this approach is correct to make whole copy process as atomic ? any one experience similar concept and resolved it.
is this approach is correct to make whole copy process as atomic ? any one experience similar concept and resolved it.
You can copy the files to a new temporary directory and then rename the directory.
Before renaming your temporary directory, you need to delete the destination directory
If other files are already in the destination directory that you don't want to overwrite, you can move all files from the temporary directory to the destination directory.
This is not completely atomic, however.
With removing /dest:
String tmpPath="/tmp/in/same/partition/as/source";
File tmp=new File(tmpPath);
tmp.mkdirs();
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1 = Paths.get(tmpPath+"/dest/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2 = Paths.get(tmpPath+"/dest/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3 = Paths.get(tmpPath+"/dest/a.txt");
Files.copy(srcFile1, destFile1);
Files.copy(srcFile2, destFile2);
Files.copy(srcFile3, destFile3);
delete(new File("/dest"));
tmp.renameTo("/dest");
void delete(File f) throws IOException {
if (f.isDirectory()) {
for (File c : f.listFiles())
delete(c);
}
if (!f.delete())
throw new FileNotFoundException("Failed to delete file: " + f);
}
With just overwriting the files:
String tmpPath="/tmp/in/same/partition/as/source";
File tmp=new File(tmpPath);
tmp.mkdirs();
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1=paths.get("/dest/a.pdf");
Path tmp1 = Paths.get(tmpPath+"/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2=Paths.get("/dest/a.doc");
Path tmp2 = Paths.get(tmpPath+"/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3=Paths.get("/dest/a.txt");
Path destFile3 = Paths.get(tmpPath+"/a.txt");
Files.copy(srcFile1, tmp1);
Files.copy(srcFile2, tmp2);
Files.copy(srcFile3, tmp3);
//Start of non atomic section(it can be done again if necessary)
Files.deleteIfExists(destFile1);
Files.deleteIfExists(destFile2);
Files.deleteIfExists(destFile2);
Files.move(tmp1,destFile1);
Files.move(tmp2,destFile2);
Files.move(tmp3,destFile3);
//end of non-atomic section
Even if the second method contains a non-atomic section, the copy process itself uses a temporary directory so that the files are not overwritten.
If the process aborts during moving the files, it can easily be completed.
See https://stackoverflow.com/a/4645271/10871900 as reference for moving files and https://stackoverflow.com/a/779529/10871900 for recursively deleting directories.
First there are several possibilities to copy a file or a directory. Baeldung gives a very nice insight into different possibilities. Additionally you can also use the FileCopyUtils from Spring. Unfortunately, all these methods are not atomic.
I have found an older post and adapt it a little bit. You can try using the low-level transaction management support. That means you make a transaction out of the method and define what should be done in a rollback. There is also a nice article from Baeldung.
#Autowired
private PlatformTransactionManager transactionManager;
#Transactional(rollbackOn = IOException.class)
public void copy(List<File> files) throws IOException {
TransactionDefinition transactionDefinition = new DefaultTransactionDefinition();
TransactionStatus transactionStatus = transactionManager.getTransaction(transactionDefinition);
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
#Override
public void afterCompletion(int status) {
if (status == STATUS_ROLLED_BACK) {
// try to delete created files
}
}
});
try {
// copy files
transactionManager.commit(transactionStatus);
} finally {
transactionManager.rollback(transactionStatus);
}
}
Or you can use a simple try-catch-block. If an exception is thrown you can delete the created files.
Your question lacks the goal of atomicity. Even unzipping is never atomic, the VM might crash with OutOfMemoryError right in between inflating the blocks of the second file. So there's one file complete, a second not and a third entirely missing.
The only thing I can think of is a two phase commit, like all the suggestions with a temporary destination that suddenly becomes the real target. This way you can be sure, that the second operation either never occurs or creates the final state.
Another approach would be to write a sort of cheap checksum file in the target afterwards. This would make it easy for an external process to listen for creation of such files and verify their content with the files found.
The latter would be the same like offering the container/ ZIP/ archive right away instead of piling files in a directory. Most archives have or support integrity checks.
(Operating systems and file systems also differ in behaviour if directories or folders disappear while being written. Some accept it and write all data to a recoverable buffer. Others still accept writes but don't change anything. Others fail immediately upon first write since the target block on the device is unknown.)
FOR ATOMIC WRITE:
There is no atomicity concept for standard filesystems, so you need to do only single action - that would be atomic.
Therefore, for writing more files in an atomic way, you need to create a folder with, let's say, the timestamp in its name, and copy files into this folder.
Then, you can either rename it to the final destination or create a symbolic link.
You can use anything similar to this, like file-based volumes on Linux, etc.
Remember that deleting the existing symbolic link and creating a new one will never be atomic, so you would need to handle the situation in your code and switch to the renamed/linked folder once it's available instead of removing/creating a link. However, under normal circumstances, removing and creating a new link is a really fast operation.
FOR ATOMIC READ:
Well, the problem is not in the code, but on the operation system/filesystem level.
Some time ago, I got into a very similar situation. There was a database engine running and changing several files "at once". I needed to copy the current state, but the second file was already changed before the first one was copied.
There are two different options:
Use a filesystem with support for snapshots. At some moment, you create a snapshot and then copy files from it.
You can lock the filesystem (on Linux) using fsfreeze --freeze, and unlock it later with fsfreeze --unfreeze. When the filesystem is frozen, you can read the files as usual, but no process can change them.
None of these options worked for me as I couldn't change the filesystem type, and locking the filesystem wasn't possible (it was root filesystem).
I created an empty file, mount it as a loop filesystem, and formatted it. From that moment on, I could fsfreeze just my virtual volume without touching the root filesystem.
My script first called fsfreeze --freeze /my/volume, then perform the copy action, and then called fsfreeze --unfreeze /my/volume. For the duration of the copy action, the files couldn't be changed, and so the copied files were all exactly from the same moment in time - for my purpose, it was like an atomic operation.
Btw, be sure to not fsfreeze your root filesystem :-). I did, and restart is the only solution.
DATABASE-LIKE APPROACH:
Even databases cannot rely on atomic operations, and so they first write the change to WAL (write-ahead log) and flush it to the storage. Once it's flushed, they can apply the change to the data file.
If there is any problem/crash, the database engine first loads the data file and checks whether there are some unapplied transactions in WAL and eventually apply them.
This is also called journaling, and it's used by some filesystems (ext3, ext4).
I hope this solution would be useful : as per my understanding you need to copy the files from one directory to another directory.
so my solution is as follows:
Thank You.!!
public class CopyFilesDirectoryProgram {
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
String sourcedirectoryName="//mention your source path";
String targetdirectoryName="//mention your destination path";
File sdir=new File(sourcedirectoryName);
File tdir=new File(targetdirectoryName);
//call the method for execution
abc (sdir,tdir);
}
private static void abc(File sdir, File tdir) throws IOException {
if(sdir.isDirectory()) {
copyFilesfromDirectory(sdir,tdir);
}
else
{
Files.copy(sdir.toPath(), tdir.toPath());
}
}
private static void copyFilesfromDirectory(File source, File target) throws IOException {
if(!target.exists()) {
target.mkdir();
}else {
for(String items:source.list()) {
abc(new File(source,items),new File(target,items));
}
}
}
}
Related
I have been searching for a way to get a file object from a file, in the resources folder. I have read a lot of similar questions on this website but non fix my problem exactly.
Link already referred to
how-to-get-a-path-to-a-resource-in-a-java-jar-file
that got really close to answering my question:
String path = this.getClass().getClassLoader().getResource(<resourceFileName>)
.toExternalForm()
I am trying to have a resource file that I can write data into and then bring that file object to another part of my program, I know I can technically create a temp file that, I then write data into then pass it into a part of my program, the problem with this approach is that I think it can take a lot of system recourses, my program will need to create a lot of these temp files.
Is there any way, I can reuse one file in the resource folder? all I need is to get it's path (and it needs to work in a jar).I have tried this snipper of code i created for testing, i don't really know why it returns false, because in the ide it returns true.
public File getFile(String fileName) throws FileNotFoundException {
//Getting file from the resources folder
ClassLoader classLoader = getClass().getClassLoader();
URL fileUrl = classLoader.getResource(fileName);
if (fileUrl == null)
throw new FileNotFoundException("Cannot find file " + fileName);
System.out.println("before: " + fileUrl.toExternalForm());
final String result = fileUrl.toExternalForm()
.replace("jar:" , "")
.replace("file:" , "");
System.out.println("after: " + result);
return new File(result);
}
Output:
before: jar:file:/C:/Users/%myuser%/Downloads/Untitlecd.jar!/Recording.wav
after: /C:/Users/%myuser%/Downloads/Untitlecd.jar!/Recording.wav
false
i have been searching for a way to get a file object from a file in the resources folder.
This is flat out impossible. The resources folder is going to end up jarred into your distribution, and you can't edit jar files, they are read only (or at least, you should consider them so. Non-idiotic deployments will generally mark their own code files (which includes those jars) as read-only to the running process. Even if not, editing jar files is extremely heavy and not something you want to do. Even if you do, on windows, open files can't be edited/replaced like this without significant headaches).
The 'resources' folder simply isn't designed for files that are meant to be modified.
The usual strategy is to make a directory someplace (for example, the user's home dir, accessing via System.getProperty("user.home"), and then make/edit files within that dir. If you wish, you can put templates in your resources folder and use those to 'initialize' that dir hanging off the user's home dir with a skeleton version.
If you have a few ten thousand files to make, whatever process needs this needs to be adjusted to not need this. For example, by using a database (H2, perhaps, if you want to ship it with your java app and have it be as low impact as possible).
I am working on an export that requires the files to be stored in a folder inside the tmp folder, and each folder must be different for different exports.
So my export() method does the below:
System.setProperty("java.io.tmpdir", System.getProperty("java.io.tmpdir")+pathSpecificToFirstExport);
the createTempFile method makes use of System.getProperty("java.io.tmpdir") to store the files in it.
While the above method is running, another call to export() sets the new System.getProperty("java.io.tmpdir") to System.getProperty("java.io.tmpdir")+pathSpecificToFirstExport+pathSpecificToSecondExport while what I really want is only System.getProperty("java.io.tmpdir")+pathSpecificToSecondExport.
I cannot hardcode System.getProperty("java.io.tmpdir") instead of appending new path to it everytime as System.getProperty("java.io.tmpdir") changes for different environments. I cannot change the way temp file is created as it is not done by me, but by write() of SXSSFWorkbook.java:
File tmplFile = TempFile.createTempFile("poi-sxssf-template", ".xlsx");
What I am looking for is to limit the scope of System.getProperty("java.io.tmpdir") only to the instance of method export()
Any thoughts?
You can't do that. The System properties object is effectively global, and there is no scoping mechanism applicable to it.
What you need to do is use a different mechanism for creating temporary files that doesn't depend on the "java.io.tmpdir". Solution: use createTempFile(String prefix, String suffix, File directory), and keep track of the "current" temporary directory using (for example) thread locals.
Better still, use the equivalent method in java.nio.Files.
Unfortunately I cannot make the above change since the createTempDirectory is done by another method that I cannot change (SXSSFWorkbook does that for me).
So I took a look at SXSSFWorkbook, and here is where it is creating temporary files:
/**
* Write out this workbook to an Outputstream.
*
* #param stream - the java OutputStream you wish to write to
* #exception IOException if anything can't be written.
*/
public void write(OutputStream stream) throws IOException {
for (SXSSFSheet sheet : _xFromSxHash.values()) {
sheet.flushRows();
}
//Save the template
File tmplFile = File.createTempFile("poi-sxssf-template", ".xlsx");
tmplFile.deleteOnExit();
FileOutputStream os = new FileOutputStream(tmplFile);
_wb.write(os);
os.close();
//Substitute the template entries with the generated sheet data files
injectData(tmplFile, stream);
tmplFile.delete();
}
First of all, Apache-POI is open source, and that means that you are free to modify it, if you need to. In this case, modifying the write method would be better than trying to make it behave differently by messing around with the global temporary directory.
But this begs the question: Why are you trying to do this? Looking at the code of write, it is pretty clear that the method is designed to clean up after itself. If write terminates normally, the temporary file is deleted before the method returns. If it terminates abnormally, the file should be cleaned up when the JVM exits.
And if, despite the above, temporary files are still "leaking", then it should be a simple matter to write an external script that periodically finds and deletes them.
Can you save the first value of "java.io.tmpdir" in a temp variable and the set it back once you finish?
String defaultDir = System.getProperty("java.io.tmpdir")
System.setProperty("java.io.tmpdir", System.getProperty("java.io.tmpdir")+pathSpecificToFirstExport);
//createTempFile method
System.setProperty("java.io.tmpdir",defaultDir);
You can't. setProperty is always global. What you should do instead is something like
Files.createTempDirectory(System.getProperty("java.io.tempdir") + pathSpecificToFirstExport);
If you can run the jobs from maven commands you can try
mvn <command to execute job> -Djava.io.tmpdir=absolutePathSpecificToFirstExport
and run a mvn command for each job.
I have two folders, source and target, with files and possible subfolders(directory structure is assumed to be the same, subfolders and files can go into any depth) in them. We want to synchronize the target so that for all files:
Exists in source, but not in target -> new, copy over to target
Exists in target, but not in source -> deleted, delete from the target
Exists in both, but binary unequal -> changed, copy over from source
Exists in both, and is binary equal -> unchanged, leave be
One problem I have with this is checking for existence of a file(the return value of listFiles() doesn't seem to have contains() defined), but a far bigger obstacle is referencing the other directory structure. For example, how would I check if target folder contains file "foo.txt" while iterating through the source folder and finding it there? Here's what I have so far:
public void synchronize(File source, File target) {
//first loop; accounts for every case except deleted
if (source.isDirectory()) {
for (File i : source.listFiles()) {
if (i.isDirectory()) {
synchronize(i, /**i's equivalent subdirectory in target*/);
}
else if (/**i is new*/) {
/**Copy i over to the appropriate target folder*/
}
else if (/**i is different*/) {
/**copy i over from source to target*/
}
else {/**i is identical in both*/
/**leave i in target alone*/
}
}
for (File i : target.listFiles()) {
if (/**i exists in the target but not in source*/) {
/**delete in target*/
}
}
}
}
EDIT(important): I thank you guys for all the answers, but the main problem remains unsolved: referring to the other directory, i.e. the stuff in the comments. h22's answer seem to be somewhere in the ballpark, but it's not sufficient, as explained in the comment below it. I'd be very grateful if someone could explain this in even smaller words. From experience, this is exactly the kind of problem that someone more java-savvy could solve in five minutes, whereas I would spend two frustrating weeks rediscovering America.
As wero points out, you can use aFile.exists() to see if a given path exists. You should also combine it with aFile.isFile() to check whether the path is a normal file (and not, say, a folder).
Checking content-equals is more tricky. I propose the following:
boolean sameContents(File fa, File fb) throws IOException {
Path a = a.toPath();
Path b = b.toPath();
if (Files.size(a) != Files.size(b)) return false;
return Arrays.equals(
Files.readAllBytes(a), Files.readAllBytes(b));
}
But only if the files are expected to be small; otherwise you could run out of memory trying to compare them in one go (required to use Arrays.equals). If you have large files in there, this answer proposes Apache Commons IO's FileUtils.contentEquals().
Note that both the above code and contentEquals only compare files, and not folders. To compare folders, you will need to use recursion, calling sameContents or equivalent on each same-named, same-sized file, and erroring out if no match is found for a particular pathname either in source or in destination.
Only visit the source folder recursively. Strip the folder root and address the target location directly:
String subPath = sourceFile.getAbsolutePath().substring(sourceRoot.length);
File targetFile = new File(targetRoot + File.separator + subPath);
if (targetFile.getParentFile().exists()) {
targetFile.getParentFile().mkdirs();
}
// copy, etc
Otherwise you may have difficulties if the target location is missing the required hierarchical folder structure that may go many directories in depth.
If you have a target directory File targetDir and a source file File sourceFile in a source directory you can check the existence of the corresponding target file by writing:
File targetFile = new File(targetDir, sourceFile.getName());
boolean exists = targetFile.exists();
A program we have erred when trying to move files from one directory to another. After much debugging I located the error by writing a small utility program that just moves a file from one directory to another (code below). It turns out that while moving files around on the local filesystem works fine, trying to move a file to another filesystem fails.
Why is this? The question might be platform specific - we are running Linux on ext3, if that matters.
And the second question; should I have been using something else than the renameTo() method of the File class? It seems as if this just works on local filesystems.
Tests (run as root):
touch /tmp/test/afile
java FileMover /tmp/test/afile /root/
The file move was successful
touch /tmp/test/afile
java FileMover /tmp/test/afile /some_other_disk/
The file move was erroneous
Code:
import java.io.File;
public class FileMover {
public static void main(String arguments[] ) throws Exception {
boolean success;
File file = new File(arguments[0]);
File destinationDir = new File(arguments[1]);
File destinationFile = new File(destinationDir,file.getName() );
success = file.renameTo(destinationFile);
System.out.println("The file move was " + (success?"successful":"erroneous"));
}
}
Java 7 and above
Use Files.move(Path source, Path target, CopyOption... opts).
Note that you must not provide the ATOMIC_MOVE option when moving files between file systems.
Java 6 and below
From the docs of File.renameTo:
[...] The rename operation might not be able to move a file from one filesystem to another [...]
The obvious workaround would be to copy the file "manually" by opening a new file, write the content to the file, and delete the old file.
You could also try the FileUtils.moveFile method from Apache Commons.
Javadoc to the rescue:
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed if a file with the destination abstract pathname
already exists. The return value should always be checked to make sure
that the rename operation was successful.
Note that the Files class defines the move method to move or rename a
file in a platform independent manner.
From the docs:
Renames the file denoted by this abstract pathname.
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed if a file with the destination abstract pathname
already exists. The return value should always be checked to make sure
that the rename operation was successful.
If you want to move file between different file system you can use Apache's moveFile
your ider is error
beause /some_other_disk/ is relative url but completely url ,can not find the url
i have example
java FileMover D:\Eclipse33_workspace_j2ee\test\src\a\a.txt D:\Eclipse33_workspace_j2ee\test\src
The file move was successful
java FileMover D:\Eclipse33_workspace_j2ee\test\src\a\a.txt \Eclipse33_workspace_j2ee\test\src
The file move was erronous
result is url is error
How to rename a file using java.io packages?
File oldfile = new File(old_name);
File newfile = new File(new_name);
boolean Rename = oldfile.renameTo(newfile);
The boolean Rename will be true if it successfully renamed the oldfile.
import java.io.File;
import java.io.IOException
public class Rename {
public static void main(String[] argv) throws IOException {
// Construct the file object. Does NOT create a file on disk!
File f = new File("Rename.java~"); // backup of this source file.
// Rename the backup file to "junk.dat"
// Renaming requires a File object for the target.
f.renameTo(new File("junk.dat"));
}
}
Reference: http://www.java2s.com/Code/Java/File-Input-Output/RenameafileinJava.htm
Use the java.io.File's renameTo method.
FWIW, as of Java 7 and later, the preferred answer for this should probably be to use java.nio.file.Files#move:
java.nio.file.Files.move(oldPath, newPath, StandardCopyOption.ATOMIC_MOVE)
The reason why one would prefer this approach is because of this documented behavior in java.io.File#renameTo:
Many aspects of the behavior of this method are inherently
platform-dependent: The rename operation might not be able to move a
file from one filesystem to another, it might not be atomic, and it
might not succeed if a file with the destination abstract pathname
already exists. The return value should always be checked to make sure
that the rename operation was successful.
Note that the Files class defines the move method to move or rename a
file in a platform independent manner.
When using java.nio.file.Files#move, one can specify standard CopyOption parameters that indicate more nuanced behavior (e.g., what you want to happen if the file already exists, whether it must be done atomically, etc.)