Develop Reference

Getting a resource's path

I have been searching for a way to get a file object from a file, in the resources folder. I have read a lot of similar questions on this website but non fix my problem exactly.
Link already referred to
how-to-get-a-path-to-a-resource-in-a-java-jar-file
that got really close to answering my question:
String path = this.getClass().getClassLoader().getResource(<resourceFileName>)
.toExternalForm()
I am trying to have a resource file that I can write data into and then bring that file object to another part of my program, I know I can technically create a temp file that, I then write data into then pass it into a part of my program, the problem with this approach is that I think it can take a lot of system recourses, my program will need to create a lot of these temp files.
Is there any way, I can reuse one file in the resource folder? all I need is to get it's path (and it needs to work in a jar).I have tried this snipper of code i created for testing, i don't really know why it returns false, because in the ide it returns true.
public File getFile(String fileName) throws FileNotFoundException {
//Getting file from the resources folder
ClassLoader classLoader = getClass().getClassLoader();
URL fileUrl = classLoader.getResource(fileName);
if (fileUrl == null)
throw new FileNotFoundException("Cannot find file " + fileName);
System.out.println("before: " + fileUrl.toExternalForm());
final String result = fileUrl.toExternalForm()
.replace("jar:" , "")
.replace("file:" , "");
System.out.println("after: " + result);
return new File(result);
}
Output:
before: jar:file:/C:/Users/%myuser%/Downloads/Untitlecd.jar!/Recording.wav
after: /C:/Users/%myuser%/Downloads/Untitlecd.jar!/Recording.wav
false

i have been searching for a way to get a file object from a file in the resources folder.
This is flat out impossible. The resources folder is going to end up jarred into your distribution, and you can't edit jar files, they are read only (or at least, you should consider them so. Non-idiotic deployments will generally mark their own code files (which includes those jars) as read-only to the running process. Even if not, editing jar files is extremely heavy and not something you want to do. Even if you do, on windows, open files can't be edited/replaced like this without significant headaches).
The 'resources' folder simply isn't designed for files that are meant to be modified.
The usual strategy is to make a directory someplace (for example, the user's home dir, accessing via System.getProperty("user.home"), and then make/edit files within that dir. If you wish, you can put templates in your resources folder and use those to 'initialize' that dir hanging off the user's home dir with a skeleton version.
If you have a few ten thousand files to make, whatever process needs this needs to be adjusted to not need this. For example, by using a database (H2, perhaps, if you want to ship it with your java app and have it be as low impact as possible).

How to copy multiple files atomically from src to dest in java?

in one requirement, i need to copy multiple files from one location to another network location.
let assume that i have the following files present in the /src location.
a.pdf, b.pdf, a.doc, b.doc, a.txt and b.txt
I need to copy a.pdf, a.doc and a.txt files atomically into /dest location at once.
Currently i am using Java.nio.file.Files packages and code as follows
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1 = Paths.get("/dest/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2 = Paths.get("/dest/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3 = Paths.get("/dest/a.txt");
Files.copy(srcFile1, destFile1);
Files.copy(srcFile2, destFile2);
Files.copy(srcFile3, destFile3);
but this process the file are copied one after another.
As an alternate to this, in order to make whole process as atomic,
i am thinking of zipping all the files and move to /dest and unzip at the destination.
is this approach is correct to make whole copy process as atomic ? any one experience similar concept and resolved it.

is this approach is correct to make whole copy process as atomic ? any one experience similar concept and resolved it.
You can copy the files to a new temporary directory and then rename the directory.
Before renaming your temporary directory, you need to delete the destination directory
If other files are already in the destination directory that you don't want to overwrite, you can move all files from the temporary directory to the destination directory.
This is not completely atomic, however.
With removing /dest:
String tmpPath="/tmp/in/same/partition/as/source";
File tmp=new File(tmpPath);
tmp.mkdirs();
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1 = Paths.get(tmpPath+"/dest/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2 = Paths.get(tmpPath+"/dest/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3 = Paths.get(tmpPath+"/dest/a.txt");
Files.copy(srcFile1, destFile1);
Files.copy(srcFile2, destFile2);
Files.copy(srcFile3, destFile3);
delete(new File("/dest"));
tmp.renameTo("/dest");
void delete(File f) throws IOException {
if (f.isDirectory()) {
for (File c : f.listFiles())
delete(c);
}
if (!f.delete())
throw new FileNotFoundException("Failed to delete file: " + f);
}
With just overwriting the files:
String tmpPath="/tmp/in/same/partition/as/source";
File tmp=new File(tmpPath);
tmp.mkdirs();
Path srcFile1 = Paths.get("/src/a.pdf");
Path destFile1=paths.get("/dest/a.pdf");
Path tmp1 = Paths.get(tmpPath+"/a.pdf");
Path srcFile2 = Paths.get("/src/a.doc");
Path destFile2=Paths.get("/dest/a.doc");
Path tmp2 = Paths.get(tmpPath+"/a.doc");
Path srcFile3 = Paths.get("/src/a.txt");
Path destFile3=Paths.get("/dest/a.txt");
Path destFile3 = Paths.get(tmpPath+"/a.txt");
Files.copy(srcFile1, tmp1);
Files.copy(srcFile2, tmp2);
Files.copy(srcFile3, tmp3);
//Start of non atomic section(it can be done again if necessary)
Files.deleteIfExists(destFile1);
Files.deleteIfExists(destFile2);
Files.deleteIfExists(destFile2);
Files.move(tmp1,destFile1);
Files.move(tmp2,destFile2);
Files.move(tmp3,destFile3);
//end of non-atomic section
Even if the second method contains a non-atomic section, the copy process itself uses a temporary directory so that the files are not overwritten.
If the process aborts during moving the files, it can easily be completed.
See https://stackoverflow.com/a/4645271/10871900 as reference for moving files and https://stackoverflow.com/a/779529/10871900 for recursively deleting directories.

First there are several possibilities to copy a file or a directory. Baeldung gives a very nice insight into different possibilities. Additionally you can also use the FileCopyUtils from Spring. Unfortunately, all these methods are not atomic.
I have found an older post and adapt it a little bit. You can try using the low-level transaction management support. That means you make a transaction out of the method and define what should be done in a rollback. There is also a nice article from Baeldung.
#Autowired
private PlatformTransactionManager transactionManager;
#Transactional(rollbackOn = IOException.class)
public void copy(List<File> files) throws IOException {
TransactionDefinition transactionDefinition = new DefaultTransactionDefinition();
TransactionStatus transactionStatus = transactionManager.getTransaction(transactionDefinition);
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
#Override
public void afterCompletion(int status) {
if (status == STATUS_ROLLED_BACK) {
// try to delete created files
}
}
});
try {
// copy files
transactionManager.commit(transactionStatus);
} finally {
transactionManager.rollback(transactionStatus);
}
}
Or you can use a simple try-catch-block. If an exception is thrown you can delete the created files.

Your question lacks the goal of atomicity. Even unzipping is never atomic, the VM might crash with OutOfMemoryError right in between inflating the blocks of the second file. So there's one file complete, a second not and a third entirely missing.
The only thing I can think of is a two phase commit, like all the suggestions with a temporary destination that suddenly becomes the real target. This way you can be sure, that the second operation either never occurs or creates the final state.
Another approach would be to write a sort of cheap checksum file in the target afterwards. This would make it easy for an external process to listen for creation of such files and verify their content with the files found.
The latter would be the same like offering the container/ ZIP/ archive right away instead of piling files in a directory. Most archives have or support integrity checks.
(Operating systems and file systems also differ in behaviour if directories or folders disappear while being written. Some accept it and write all data to a recoverable buffer. Others still accept writes but don't change anything. Others fail immediately upon first write since the target block on the device is unknown.)

FOR ATOMIC WRITE:
There is no atomicity concept for standard filesystems, so you need to do only single action - that would be atomic.
Therefore, for writing more files in an atomic way, you need to create a folder with, let's say, the timestamp in its name, and copy files into this folder.
Then, you can either rename it to the final destination or create a symbolic link.
You can use anything similar to this, like file-based volumes on Linux, etc.
Remember that deleting the existing symbolic link and creating a new one will never be atomic, so you would need to handle the situation in your code and switch to the renamed/linked folder once it's available instead of removing/creating a link. However, under normal circumstances, removing and creating a new link is a really fast operation.
FOR ATOMIC READ:
Well, the problem is not in the code, but on the operation system/filesystem level.
Some time ago, I got into a very similar situation. There was a database engine running and changing several files "at once". I needed to copy the current state, but the second file was already changed before the first one was copied.
There are two different options:
Use a filesystem with support for snapshots. At some moment, you create a snapshot and then copy files from it.
You can lock the filesystem (on Linux) using fsfreeze --freeze, and unlock it later with fsfreeze --unfreeze. When the filesystem is frozen, you can read the files as usual, but no process can change them.
None of these options worked for me as I couldn't change the filesystem type, and locking the filesystem wasn't possible (it was root filesystem).
I created an empty file, mount it as a loop filesystem, and formatted it. From that moment on, I could fsfreeze just my virtual volume without touching the root filesystem.
My script first called fsfreeze --freeze /my/volume, then perform the copy action, and then called fsfreeze --unfreeze /my/volume. For the duration of the copy action, the files couldn't be changed, and so the copied files were all exactly from the same moment in time - for my purpose, it was like an atomic operation.
Btw, be sure to not fsfreeze your root filesystem :-). I did, and restart is the only solution.
DATABASE-LIKE APPROACH:
Even databases cannot rely on atomic operations, and so they first write the change to WAL (write-ahead log) and flush it to the storage. Once it's flushed, they can apply the change to the data file.
If there is any problem/crash, the database engine first loads the data file and checks whether there are some unapplied transactions in WAL and eventually apply them.
This is also called journaling, and it's used by some filesystems (ext3, ext4).

I hope this solution would be useful : as per my understanding you need to copy the files from one directory to another directory.
so my solution is as follows:
Thank You.!!
public class CopyFilesDirectoryProgram {
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
String sourcedirectoryName="//mention your source path";
String targetdirectoryName="//mention your destination path";
File sdir=new File(sourcedirectoryName);
File tdir=new File(targetdirectoryName);
//call the method for execution
abc (sdir,tdir);
}
private static void abc(File sdir, File tdir) throws IOException {
if(sdir.isDirectory()) {
copyFilesfromDirectory(sdir,tdir);
}
else
{
Files.copy(sdir.toPath(), tdir.toPath());
}
}
private static void copyFilesfromDirectory(File source, File target) throws IOException {
if(!target.exists()) {
target.mkdir();
}else {
for(String items:source.list()) {
abc(new File(source,items),new File(target,items));
}
}
}
}

Reading files from a windows shared directory in parallel

I have a server that reads a list of text files from a windows shared directory and save it contents to the db before its starts to accepts user messages. This server will be running in multiple machines at one time.
I see that when I run the server in multiple machines, the server that starts processing the files, first processes all the files and the others keep waiting to get access to the files in that directory.
My code does this - (cannot post code due to security policy)
Get a list all files in the shared directory.
Sort them by modified date (its saving time series data)
While(true) till more files exist in directory
Get the first file in the list, and move it to InProgess folder and read
Save contents to database.
Move file to Archive directory.
Process the next file.
I see that when I run the same program in 2 different machines, one of them get holds of the files first and loads them all. The other one keeps waiting to get a handle on the files and if it does find a handle, they have already been processed. So it moves on.
My aim is to have the process when run in both or multiple machines to process all the file in parallel and finish faster. For now I am testing with 500 files on disk, but I can have more files on disk at any given time.
PseudoCode -
if(files exist on disk){
LOGGER.info("Files exist on disk. Lets process them up first....");
while (true) {
File dir = new File(directory);
List<File> fileList = new LinkedList<File>(Arrays.asList(dir.listFiles((FileFilter)FileFileFilter.FILE)));
LOGGER.info("No of files in this process: "+ sortedFileList.size());
if (fileList.size() > 0) {
Collections.sort(fileList, new Server().new FileComparator());
File file = fileList.get(0);
//If I cannot rename the file in the same directory, the file maybe open and I move to the next file
if(!file.renameTo(file.getAbsoluteFile())) {
LOGGER.info("Read next file...");
continue;
}
LOGGER.info("Get file handle...");
if (file.exists()) {
File inprogressFile = new File(dataDirName + FileBackupOnDisk.INPROGRESS + fileName);
saveToDB(inprogressFile);
if (savedToDB)
if(inprogressFile.renameTo(new File(dataDirName+ARCHIVE+fileName)))
LOGGER.info("Moved file to archive - " + fileName);
else
LOGGER.error("Move file " + fileName + " to failed directory!");
}
}
}
}
That's my file comparator code. This cannot be opening files -
final Map<File, Long> staticLastModifiedTimes = new HashMap<File,Long>();
for(final File f : sortedFileList) {
staticLastModifiedTimes.put(f, f.lastModified());
}
Collections.sort(sortedFileList, new Comparator<File>() {
#Override
public int compare(final File f1, final File f2) {
return
staticLastModifiedTimes.get(f1).compareTo(staticLastModifiedTimes.get(f2));
}
});
How do I make sure that both my servers/multiple servers running on different machines are able to access the shared directly in parallel. Right now it looks like the 2nd process find that files exist in the dir but hang at one point waiting to get a file handle.
Let me know if anyone has done this before and how?

I found out that my solution above works perfectly fine!!!!
Its just that running one instance from my eclipse with another from a m/c in the network was causing this latency issues.
If I run the program with 2 machines in the same network it works fine. Just that my computer was slower. Both the instances read the files when they get are able to get handle on it.
Thank you all for your help.

Java List array showing non existing folders.. Why?

I am in the process of making a program, which allows you to view your file system.
I was testing it, and ran into a problem: It was saying a directory called "Documents and Settings" was on my C:\ drive, while it wasn't there.
This is how I get my file array:
File f = new File(path); //path being a path sent by the client, for example C:\
if(f.isFile()){
//TODO start downloading it.
out.println("ERR: no dir!");
return;
}
Server.log.log("System path requested: " + f.getAbsolutePath());
File[] files = f.listFiles();
for(int i = 0; i < files.length; i++){
File found = files[i];
if(!found.exists()){
continue;
}
if(found.isDirectory()){
out.println("dir:" + found.getName());
}else{
out.println(found.getName());
}
System.out.println("Printed " + found.getName());
}
out.println("ENDOFLIST"); //Notify the client it has to stop receiving data
For some reason, this outputs quite a lot of directories that I can't seem to find, even with the "Show hidden folders" option on.
When trying to access these directories, it tries to read the contents of the directory, but since the directory doesn't exist it throws an exception, causing no data to get sent over sockets and my client freezing.
My question is: Is there a way to either check if the file/directory REALLY exists? Note, if you look at my code block, if the file/dir doesn't exist it already continues instead of writing it to the socket.
I've given it a google, but no matches were found. Also, I've given the search function a go, but it didn't come up with anything similar.

These are hidden system folders.
They do exist. Really.
You get exceptions because a lot of them don't have read access.

I suggest to use the new Fil I/O API introduced by Java 7, it features greatly improved support of the features a specific file system offers. It also offers the possibility to use walk the file tree.
Have a look at the FileVisitor http://docs.oracle.com/javase/7/docs/api/java/nio/file/FileVisitor.html that will greatly help you.

What are possible reasons for java.io.IOException: "The filename, directory name, or volume label syntax is incorrect"

I am trying to copy a file using the following code:
File targetFile = new File(targetPath + File.separator + filename);
...
targetFile.createNewFile();
fileInputStream = new FileInputStream(fileToCopy);
fileOutputStream = new FileOutputStream(targetFile);
byte[] buffer = new byte[64*1024];
int i = 0;
while((i = fileInputStream.read(buffer)) != -1) {
fileOutputStream.write(buffer, 0, i);
}
For some users the targetFile.createNewFile results in this exception:
java.io.IOException: The filename, directory name, or volume label syntax is incorrect
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:850)
Filename and directory name seem to be correct. The directory targetPath is even checked for existence before the copy code is executed and the filename looks like this: AB_timestamp.xml
The user has write permissions to the targetPath and can copy the file without problems using the OS.
As I don't have access to a machine this happens on yet and can't reproduce the problem on my own machine I turn to you for hints on the reason for this exception.

This can occur when filename has timestamp with colons, eg. myfile_HH:mm:ss.csv Removing colons fixed the issue.

Try this, as it takes more care of adjusting directory separator characters in the path between targetPath and filename:
File targetFile = new File(targetPath, filename);

I just encountered the same problem. I think it has to something do with write access permission. I got the error while trying to write to c:\ but on changing to D:\ everything worked fine.
Apparently Java did not have permission to write to my System Drive (Running Windows 7 installed on C:)

Here is the test program I use
import java.io.File;
public class TestWrite {
public static void main(String[] args) {
if (args.length!=1) {
throw new IllegalArgumentException("Expected 1 argument: dir for tmp file");
}
try {
File.createTempFile("bla",".tmp",new File(args[0]));
} catch (Exception e) {
System.out.println("exception:"+e);
e.printStackTrace();
}
}
}

Try to create the file in a different directory - e.g. "C:\" after you made sure you have write access to that directory. If that works, the path name of the file is wrong.
Take a look at the comment in the Exception and try to vary all the elements in the path name of the file. Experiment. Draw conclusions.

Remove any special characters in the file/folder name in the complete path.

Do you check that the targetPath is a directory, or just that something exists with that name? (I know you say the user can copy it from the operating system, but maybe they're typing something else).
Does targetPath end with a File.separator already?
(It would help if you could log and tell us what the value of targetPath and filename are on a failing case)

Maybe the problem is that it is copying the file over the network, to a shared drive? I think java can have problems when writing files using NFS when the path is something like \mypc\myshared folder.
What is the path where this problem happens?

Try adding some logging to see exactly what is the name and path the file is trying to create, to ensure that the parent is well a directory.
In addition, you can also take a look at Channels instead of using a loop. ;-)

You say "for some users" - so it works for others? What is the difference here, are the users running different instances on different machines, or is this a server that services concurrent users?
If the latter, I'd say it is a concurrency bug somehow - two threads check try to create the file with WinNTFileSystem.createFileExclusively(Native Method) simultaniously.
Neither createNewFile or createFileExclusively are synchronized when I look at the OpenJDK source, so you may have to synchronize this block yourself.

Maybe the file already exists. It could be the case if your timestamp resolution is not good enough. As it is an IOException that you are getting, it might not be a permission issue (in which case you would get a SecurityException).
I would first check for file existence before trying to create the file and try to log what's happening.
Look at public boolean createNewFile() for more information on the method you are using.

As I was not able to reproduce the error on my own machine or get hands on the machine of the user where the code failed I waited until now to declare an accepted answer.
I changed the code to the following:
File parentFolder = new File(targetPath);
... do some checks on parentFolder here ...
File targetFile = new File(parentFolder, filename);
targetFile.createNewFile();
fileInputStream = new FileInputStream(fileToCopy);
fileOutputStream = new FileOutputStream(targetFile);
byte[] buffer = new byte[64*1024];
int i = 0;
while((i = fileInputStream.read(buffer)) != -1) {
fileOutputStream.write(buffer, 0, i);
}
After that it worked for the user reporting the problem.
So it seems Alexanders answer did the trick - although I actually use a slightly different constructor than he gave, but along the same lines.
I yet have to talk that user into helping me verifying that the code change fixed the error (instead of him doing something differently) by running the old version again and checking if it still fails.
btw. logging was in place and the logged path seemed ok - sorry for not mentioning that. I took that for granted and found it unnecessarily complicated the code in the question.
Thanks for the helpful answers.

A very similar error:-
" ... java.io.IOException: The filename, directory name, or volume label syntax is incorrect"
was generated in Eclipse for me when the TOMCAT home setting had a training backslash.
The minor edit suggested at:-
http://www.coderanch.com/t/556633/Tomcat/java-io-IOException-filename-directory
fixed it for me.

FileUtils.copyFile(src,new File("C:\\Users\\daiva\\eclipse-workspace\\PracticeProgram\\Screenshot\\adi.png"));
Try to copy file like this.