in one requirement, i need to copy multiple files from one location to another network location.
let assume that i have the following files present in the /src location.
a.pdf, b.pdf, a.doc, b.doc, a.txt and b.txt
I need to copy a.pdf, a.doc and a.txt files atomically into /dest location at once.
Currently i am using Java.nio.file.Files packages and code as follows
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1 = Paths.get("/dest/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2 = Paths.get("/dest/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3 = Paths.get("/dest/a.txt");
Files.copy(srcFile1, destFile1);
Files.copy(srcFile2, destFile2);
Files.copy(srcFile3, destFile3);
but this process the file are copied one after another.
As an alternate to this, in order to make whole process as atomic,
i am thinking of zipping all the files and move to /dest and unzip at the destination.
is this approach is correct to make whole copy process as atomic ? any one experience similar concept and resolved it.
is this approach is correct to make whole copy process as atomic ? any one experience similar concept and resolved it.
You can copy the files to a new temporary directory and then rename the directory.
Before renaming your temporary directory, you need to delete the destination directory
If other files are already in the destination directory that you don't want to overwrite, you can move all files from the temporary directory to the destination directory.
This is not completely atomic, however.
With removing /dest:
String tmpPath="/tmp/in/same/partition/as/source";
File tmp=new File(tmpPath);
tmp.mkdirs();
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1 = Paths.get(tmpPath+"/dest/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2 = Paths.get(tmpPath+"/dest/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3 = Paths.get(tmpPath+"/dest/a.txt");
Files.copy(srcFile1, destFile1);
Files.copy(srcFile2, destFile2);
Files.copy(srcFile3, destFile3);
delete(new File("/dest"));
tmp.renameTo("/dest");
void delete(File f) throws IOException {
if (f.isDirectory()) {
for (File c : f.listFiles())
delete(c);
}
if (!f.delete())
throw new FileNotFoundException("Failed to delete file: " + f);
}
With just overwriting the files:
String tmpPath="/tmp/in/same/partition/as/source";
File tmp=new File(tmpPath);
tmp.mkdirs();
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1=paths.get("/dest/a.pdf");
Path tmp1 = Paths.get(tmpPath+"/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2=Paths.get("/dest/a.doc");
Path tmp2 = Paths.get(tmpPath+"/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3=Paths.get("/dest/a.txt");
Path destFile3 = Paths.get(tmpPath+"/a.txt");
Files.copy(srcFile1, tmp1);
Files.copy(srcFile2, tmp2);
Files.copy(srcFile3, tmp3);
//Start of non atomic section(it can be done again if necessary)
Files.deleteIfExists(destFile1);
Files.deleteIfExists(destFile2);
Files.deleteIfExists(destFile2);
Files.move(tmp1,destFile1);
Files.move(tmp2,destFile2);
Files.move(tmp3,destFile3);
//end of non-atomic section
Even if the second method contains a non-atomic section, the copy process itself uses a temporary directory so that the files are not overwritten.
If the process aborts during moving the files, it can easily be completed.
See https://stackoverflow.com/a/4645271/10871900 as reference for moving files and https://stackoverflow.com/a/779529/10871900 for recursively deleting directories.
First there are several possibilities to copy a file or a directory. Baeldung gives a very nice insight into different possibilities. Additionally you can also use the FileCopyUtils from Spring. Unfortunately, all these methods are not atomic.
I have found an older post and adapt it a little bit. You can try using the low-level transaction management support. That means you make a transaction out of the method and define what should be done in a rollback. There is also a nice article from Baeldung.
#Autowired
private PlatformTransactionManager transactionManager;
#Transactional(rollbackOn = IOException.class)
public void copy(List<File> files) throws IOException {
TransactionDefinition transactionDefinition = new DefaultTransactionDefinition();
TransactionStatus transactionStatus = transactionManager.getTransaction(transactionDefinition);
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
#Override
public void afterCompletion(int status) {
if (status == STATUS_ROLLED_BACK) {
// try to delete created files
}
}
});
try {
// copy files
transactionManager.commit(transactionStatus);
} finally {
transactionManager.rollback(transactionStatus);
}
}
Or you can use a simple try-catch-block. If an exception is thrown you can delete the created files.
Your question lacks the goal of atomicity. Even unzipping is never atomic, the VM might crash with OutOfMemoryError right in between inflating the blocks of the second file. So there's one file complete, a second not and a third entirely missing.
The only thing I can think of is a two phase commit, like all the suggestions with a temporary destination that suddenly becomes the real target. This way you can be sure, that the second operation either never occurs or creates the final state.
Another approach would be to write a sort of cheap checksum file in the target afterwards. This would make it easy for an external process to listen for creation of such files and verify their content with the files found.
The latter would be the same like offering the container/ ZIP/ archive right away instead of piling files in a directory. Most archives have or support integrity checks.
(Operating systems and file systems also differ in behaviour if directories or folders disappear while being written. Some accept it and write all data to a recoverable buffer. Others still accept writes but don't change anything. Others fail immediately upon first write since the target block on the device is unknown.)
FOR ATOMIC WRITE:
There is no atomicity concept for standard filesystems, so you need to do only single action - that would be atomic.
Therefore, for writing more files in an atomic way, you need to create a folder with, let's say, the timestamp in its name, and copy files into this folder.
Then, you can either rename it to the final destination or create a symbolic link.
You can use anything similar to this, like file-based volumes on Linux, etc.
Remember that deleting the existing symbolic link and creating a new one will never be atomic, so you would need to handle the situation in your code and switch to the renamed/linked folder once it's available instead of removing/creating a link. However, under normal circumstances, removing and creating a new link is a really fast operation.
FOR ATOMIC READ:
Well, the problem is not in the code, but on the operation system/filesystem level.
Some time ago, I got into a very similar situation. There was a database engine running and changing several files "at once". I needed to copy the current state, but the second file was already changed before the first one was copied.
There are two different options:
Use a filesystem with support for snapshots. At some moment, you create a snapshot and then copy files from it.
You can lock the filesystem (on Linux) using fsfreeze --freeze, and unlock it later with fsfreeze --unfreeze. When the filesystem is frozen, you can read the files as usual, but no process can change them.
None of these options worked for me as I couldn't change the filesystem type, and locking the filesystem wasn't possible (it was root filesystem).
I created an empty file, mount it as a loop filesystem, and formatted it. From that moment on, I could fsfreeze just my virtual volume without touching the root filesystem.
My script first called fsfreeze --freeze /my/volume, then perform the copy action, and then called fsfreeze --unfreeze /my/volume. For the duration of the copy action, the files couldn't be changed, and so the copied files were all exactly from the same moment in time - for my purpose, it was like an atomic operation.
Btw, be sure to not fsfreeze your root filesystem :-). I did, and restart is the only solution.
DATABASE-LIKE APPROACH:
Even databases cannot rely on atomic operations, and so they first write the change to WAL (write-ahead log) and flush it to the storage. Once it's flushed, they can apply the change to the data file.
If there is any problem/crash, the database engine first loads the data file and checks whether there are some unapplied transactions in WAL and eventually apply them.
This is also called journaling, and it's used by some filesystems (ext3, ext4).
I hope this solution would be useful : as per my understanding you need to copy the files from one directory to another directory.
so my solution is as follows:
Thank You.!!
public class CopyFilesDirectoryProgram {
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
String sourcedirectoryName="//mention your source path";
String targetdirectoryName="//mention your destination path";
File sdir=new File(sourcedirectoryName);
File tdir=new File(targetdirectoryName);
//call the method for execution
abc (sdir,tdir);
}
private static void abc(File sdir, File tdir) throws IOException {
if(sdir.isDirectory()) {
copyFilesfromDirectory(sdir,tdir);
}
else
{
Files.copy(sdir.toPath(), tdir.toPath());
}
}
private static void copyFilesfromDirectory(File source, File target) throws IOException {
if(!target.exists()) {
target.mkdir();
}else {
for(String items:source.list()) {
abc(new File(source,items),new File(target,items));
}
}
}
}
I am trying to get the directory of a class package as a File within my .jar (the hierarchy is /controller/core):
File directory_file_of_package = new File(getClass().getResource("/controller/core").toURI());
Which works when I run the program within the IDE, but when I run it as a standalone .jar, it gives me this error:
java.lang.IllegalArgumentException: URI is not hierarchical
Looking at these questions (why my URI is not hierarchical? and Java Jar file: use resource errors: URI is not hierarchical) does not help me, because those questions involved obtaining a File within the .jar as a resource (getResourceAsStream()), but what I am trying to obtain is a directory.
Further, the accepted answer states this:
Generally the IDEs separates on file system classes and resources. But when the jar is created they are put all together.
So, is there no way to grab the directory of /controller/core?
My XY Problem:
I am trying to read a .CSV file using CSVJDBC (http://csvjdbc.sourceforge.net/), and the way the DriverManager references the CSV file as as if the containing folder is the database and the CSV file is the table.
public class SearchData {
void SearchData() {
try {
Class.forName("org.relique.jdbc.csv.CsvDriver");
File directory_file_of_package_containing_csv = new File(getClass().getResource("/controller/core").toURI());
// Driver points to directory which contains the CSV file as if it is the database
Connection conn = DriverManager.getConnection("jdbc:relique:csv:" + directory_file_of_package_containing_csv.toString());
System.out.println("Connection established.");
Statement stmt = conn.createStatement();
System.out.println("Statement created.");
// CSV is named "data.csv" so the CsvDriver sees "data" as the table name
String sql = "SELECT id,name FROM data";
ResultSet results = stmt.executeQuery(sql);
...
} catch (ClassNotFoundException | SQLException | URISyntaxException e) {
e.printStackTrace();
}
}
}
How can I point CsvJdbc to my csv file contained within a .jar?
There simply is no directory of a package within the JAR. It works in IDE because your class is not yet packaged in a JAR file and resides in some directory. But this is not the case with JARs.
So if some of the libraries you use require a directory with a file in it, you can't generally achieve this with classpath resources. One of the options would be to create a copy of your resource as a file in a temporary directory.
Alternatively you may want to study the CsvJDBC documentation closer:
To read data that is either held inside the Java application (for example, in a JAR file) or accessed remotely (for example, using HTTP requests), create a Java class that implements the interface org.relique.io.TableReader and give this class name in the connection URL. CsvJdbc then creates an instance of this class and calls the getReader method to obtain a java.io.Reader for each database table being read.
So you should be able to solve this for CsvJDBC without temporary directories.
Can I get and edit the list of files, downloaded with DownloadProvider?
What do I mean? We have an application called Downloads in Android that displays all the downloads made with DownloadProvider. The records it displays are stored in a database somewhere in /data and are not strongly connected with real files. E.g. if I delete a record in Downloads, the file is deleted too, but not vice versa.
So, I want to delete the file in filesystem and delete the record in Downloads application about this file.
Currently I have tried using something like:
DownloadManager dm = (DownloadManager)getSystemService(DOWNLOAD_SERVICE);
DownloadManager.Query query = new DownloadManager.Query();
query.setFilterByStatus(DownloadManager.STATUS_PAUSED|
DownloadManager.STATUS_PENDING|
DownloadManager.STATUS_RUNNING|
DownloadManager.STATUS_SUCCESSFUL);
Cursor cur = dm.query(query);
and I don't see any ID's in query which I could pass to the DownloadManager.remove(long...IDs). Can I do what I want with Android API or the only way to achieve this is to edit the database itself?
Try call
mDownloadManager.remove(long... ids)
I have an application with document uploading.After uploading the document,the document path is newly created and the document is saved within the file path.At the same time the document path and related values are saved with in the database table.In my application after uploading,there is a button to delete the unwanted document.Sometimes the document deletion operation is not properly work.So there is a wastage of memory will occur.I want to avoid the situation by using transaction statement.I don't know how to use the hibernate transaction to my work.Is this possible? Please help me to do the work successfully(I am using spring with hibernate integration and postgresql)
Thank you
In controller
int supDocId=1102;
String docPath=D:/;
String filePath=docPath+supDocId+".pdf";
File file=new File(filePath);
boolean isDelete = servicesService.deleteDocument(supDocId);
if(isDelete)
{
if(file.exists())
{
file.delete();
}
alertMsg = "The file is deleted sucessfully";
}
else{
alertMsg = "Deletion Failed.!!! File is under processing..";
}
In service class
public boolean deleteDocument(int supDocId){
return servicesDAO.deleteDocument(supDocId);
}
In servicesDAO class
public boolean deleteDocument(int supDocId){
int deleteStatus=0;
try {
String deleteQuery = "DELETE FROM tablename WHERE attch_doc_id='"+supDocId+"'";
Query deleteQ = session.createSQLQuery(deleteQuery);
deleteStatus = deleteQ.executeUpdate();
if(deleteStatus>0){
return deleteStatus;
}
} catch (Exception e) {
e.printStackTrace();
}
return deleteStatus;
}
I want to work the two operations(document and database value deletion) within the DAO class using transaction statement.
If DB transaction fails - you will have problem not only with deleted files, but also with uploaded new files.
In most cases file system does not supports transaction so I don't think bullet proof solution can be achieved using XA (distributed transactions using the JTA) or similar approach.
Quite strait forward solution I am using in some of my projects:
Files in file system are created before DB commit (but preferably after hibernate flush).
On file delete operation, reference to file is deleted from DB, but not from file system.
Asynchronous process periodically scans all files in file system and deletes files with no references from DB.
If you have big number of files some optimization can be necessary to prevent always scan all files. For example:
Put new files to folder with name "current date", so only part of files can be checked for unsuccessful transactions involving uploaded new files.
On file delete operation insert new record to table "deleted_files" with reference to file that should be deleted.
Create asynchronous process to periodically scans table "deleted_files", which deletes physical file and if deletion is successful (or file already is missing) removes record from table "deleted_files" and commits transaction.
You shouldn't use +supDocId+ Replace that with ? instead or you would be leaving yourself vulnerable to Sql Injection.
And deleteQ.Execute();is what you want to be doing not an update because you want to be checking boolean true or false if something is deleted.
I am using JGit for my project where i have to maintain the files uploaded by the users.
If the file already existing then it would create a new version of the same and associate the customer with his commited/uploaded set of files.
It works file locally i.e each file one uploads , commit it to repository i.e. on server machine and adds the same recored in the DB , to associate the customer and its respective file versioning lists i.e. commitIds.
But after few iterations of upload files (each upload may consists around 200 files) , JGit able to commit the files and generate the commitIds properly but not able to retreive the content of the files when showing the user back his commmitted files.
Unfortunately log does not show any errors while retreiving the files.
Hence i am lost ....and struggling to understand stand what is wrong here.
My questions are:
Does the JGIT have enough scalability ? i.e. fetch time would be fast enough as it grows?
what to do if i retrieve the files properly.
Below is the piece of code which i am using
FileUtils.copyFile(aFile,destFile);
AddCommand add = git.add();
add.addFilepattern(".").call();
CommitCommand commit = git.commit();
commit.setAuthor(user.getFirstName(), user.getUserId());
commit.setMessage("comments" ).call();
ObjectId lastCommitId = git.getRepository().resolve(org.eclipse.jgit.lib.Constants.HEAD);
Above destFile is GIT repository and aFile is the file name
Using the lastCommitId i am trying to retrive the content of the file but me getting the:
MissingObjectException: Missing unknown 0000000000000000000000000000000000000000
Code used to retreive the file is:
ObjectId lastCommitId = repo.resolve(lastCommitId);
RevTree tree = commit.getTree();
TreeWalk treeWalk = new TreeWalk(repo);
treeWalk.addTree(tree);
treeWalk.setRecursive(true);
treeWalk.setFilter(PathFilter.create("actial_File_Name")); //this is actual file name i used
boolean next = treeWalk.next();
if (next)
{
ObjectId objectId = treeWalk.getObjectId(0);
log.logError(" objectId :" + objectId );
try{
ObjectLoader loader = repo.open(objectId);
OutputStream out = new FileOutputStream(targetFile); ///targetFile is the actual file name i wanted to retreive the content i.e orginal name
loader.copyTo(out);
}
}
The 'missing unknown 0000' seems to suggest that one of your references is missing, and the null is being returned for the objectId. You should check to see if the objectId.equals(ObjectId.zeroId()) when you do the lookup.
You might also want to check to see if the repository needs garbage collection (either with JGit or a standalone Git client) - by default JGit doesn't do automatic garbage collection.
Note also that a commit doesn't get inserted into the database until you call the call() method and you get a commit back again. Maybe that's why you can't find it in your code?