I have a server that reads a list of text files from a windows shared directory and save it contents to the db before its starts to accepts user messages. This server will be running in multiple machines at one time.
I see that when I run the server in multiple machines, the server that starts processing the files, first processes all the files and the others keep waiting to get access to the files in that directory.
My code does this - (cannot post code due to security policy)
Get a list all files in the shared directory.
Sort them by modified date (its saving time series data)
While(true) till more files exist in directory
Get the first file in the list, and move it to InProgess folder and read
Save contents to database.
Move file to Archive directory.
Process the next file.
I see that when I run the same program in 2 different machines, one of them get holds of the files first and loads them all. The other one keeps waiting to get a handle on the files and if it does find a handle, they have already been processed. So it moves on.
My aim is to have the process when run in both or multiple machines to process all the file in parallel and finish faster. For now I am testing with 500 files on disk, but I can have more files on disk at any given time.
PseudoCode -
if(files exist on disk){
LOGGER.info("Files exist on disk. Lets process them up first....");
while (true) {
File dir = new File(directory);
List<File> fileList = new LinkedList<File>(Arrays.asList(dir.listFiles((FileFilter)FileFileFilter.FILE)));
LOGGER.info("No of files in this process: "+ sortedFileList.size());
if (fileList.size() > 0) {
Collections.sort(fileList, new Server().new FileComparator());
File file = fileList.get(0);
//If I cannot rename the file in the same directory, the file maybe open and I move to the next file
if(!file.renameTo(file.getAbsoluteFile())) {
LOGGER.info("Read next file...");
continue;
}
LOGGER.info("Get file handle...");
if (file.exists()) {
File inprogressFile = new File(dataDirName + FileBackupOnDisk.INPROGRESS + fileName);
saveToDB(inprogressFile);
if (savedToDB)
if(inprogressFile.renameTo(new File(dataDirName+ARCHIVE+fileName)))
LOGGER.info("Moved file to archive - " + fileName);
else
LOGGER.error("Move file " + fileName + " to failed directory!");
}
}
}
}
That's my file comparator code. This cannot be opening files -
final Map<File, Long> staticLastModifiedTimes = new HashMap<File,Long>();
for(final File f : sortedFileList) {
staticLastModifiedTimes.put(f, f.lastModified());
}
Collections.sort(sortedFileList, new Comparator<File>() {
#Override
public int compare(final File f1, final File f2) {
return
staticLastModifiedTimes.get(f1).compareTo(staticLastModifiedTimes.get(f2));
}
});
How do I make sure that both my servers/multiple servers running on different machines are able to access the shared directly in parallel. Right now it looks like the 2nd process find that files exist in the dir but hang at one point waiting to get a file handle.
Let me know if anyone has done this before and how?
I found out that my solution above works perfectly fine!!!!
Its just that running one instance from my eclipse with another from a m/c in the network was causing this latency issues.
If I run the program with 2 machines in the same network it works fine. Just that my computer was slower. Both the instances read the files when they get are able to get handle on it.
Thank you all for your help.
Related
I have been searching for a way to get a file object from a file, in the resources folder. I have read a lot of similar questions on this website but non fix my problem exactly.
Link already referred to
how-to-get-a-path-to-a-resource-in-a-java-jar-file
that got really close to answering my question:
String path = this.getClass().getClassLoader().getResource(<resourceFileName>)
.toExternalForm()
I am trying to have a resource file that I can write data into and then bring that file object to another part of my program, I know I can technically create a temp file that, I then write data into then pass it into a part of my program, the problem with this approach is that I think it can take a lot of system recourses, my program will need to create a lot of these temp files.
Is there any way, I can reuse one file in the resource folder? all I need is to get it's path (and it needs to work in a jar).I have tried this snipper of code i created for testing, i don't really know why it returns false, because in the ide it returns true.
public File getFile(String fileName) throws FileNotFoundException {
//Getting file from the resources folder
ClassLoader classLoader = getClass().getClassLoader();
URL fileUrl = classLoader.getResource(fileName);
if (fileUrl == null)
throw new FileNotFoundException("Cannot find file " + fileName);
System.out.println("before: " + fileUrl.toExternalForm());
final String result = fileUrl.toExternalForm()
.replace("jar:" , "")
.replace("file:" , "");
System.out.println("after: " + result);
return new File(result);
}
Output:
before: jar:file:/C:/Users/%myuser%/Downloads/Untitlecd.jar!/Recording.wav
after: /C:/Users/%myuser%/Downloads/Untitlecd.jar!/Recording.wav
false
i have been searching for a way to get a file object from a file in the resources folder.
This is flat out impossible. The resources folder is going to end up jarred into your distribution, and you can't edit jar files, they are read only (or at least, you should consider them so. Non-idiotic deployments will generally mark their own code files (which includes those jars) as read-only to the running process. Even if not, editing jar files is extremely heavy and not something you want to do. Even if you do, on windows, open files can't be edited/replaced like this without significant headaches).
The 'resources' folder simply isn't designed for files that are meant to be modified.
The usual strategy is to make a directory someplace (for example, the user's home dir, accessing via System.getProperty("user.home"), and then make/edit files within that dir. If you wish, you can put templates in your resources folder and use those to 'initialize' that dir hanging off the user's home dir with a skeleton version.
If you have a few ten thousand files to make, whatever process needs this needs to be adjusted to not need this. For example, by using a database (H2, perhaps, if you want to ship it with your java app and have it be as low impact as possible).
In some place I create temporary file:
Files.createTempDirectory("chunk");
In some other place after processsing, I try to delete file:
Files.deleteIfExists(somePath)
and experience following trace:
java.nio.file.FileSystemException: C:\....\Temp\chunk11607697185854596263\chunk-3.csv: The process cannot access the file because it is being used by another process.
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:270)
at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110)
at java.base/java.nio.file.Files.deleteIfExists(Files.java:1180)
at my.some.project.batch.MyClass.afterStep(MyClass.java:31)
It happens when I start application on my local Windows mashine and doesn't happen in docker. Also I don't experience such error when I run application locally on MacOS. What the problem and how can I fix it ?
Check that the file isn't opened by some other process. This check can be done with Process Explorer. When you have started the program, select Find in the menu and then Find Handle or DLL..., or press Ctrl+F. Enter the name of the file that is being locked by some process and click Search.
There's a difference between Windows and Unix-like operating systems regarding the handling of removal of opened files.
On Unix-like systems a file can be removed even if it's opened by other processes. The actual file is not removed until all other processes has closed the file, but the filename entry on the filesystem is removed. The low-level file remove operation returns success as soon as the first part of the operation succeeds. Removal of the actual file data is delayed until the open count has reached 0.
On Windows the file may not be opened by any process for the low-level file remove operation to be able to succeed.
This could be a reason why you see a different behaviour when running your program on Windows versus Docker or MacOS.
There are quite a few problems around when file is deleted with Java in Windows. The most important thing is to really make sure there are no unclosed streams before trying to delete a file as VGR already mentioned.
For more info about the usual problems you could take a look at this:
delete & GC
and this: file delete
I am using this kind of awful looking "solution" for deleting files, feel free to try:
public static boolean deleteFile(File file)
throws Exception {
if (!file.exists()) {
throw new Exception("File not found: " + file.getName());
}
try {
file.setWritable(true);
} catch (Exception ignored) {
}
int delay = 50;
for (int i = 0; i < 20; i++) {
if (file.delete()) {
return true;
}
System.gc();
try {
Thread.sleep(delay);
} catch (InterruptedException ignored) {
}
delay = delay + 50;
}
throw new Exception("Could not delete the file '"
+ file.getName() + "'");
}
I have a piece of code which writes a file in Java to a shared Samba folder on a virtual server, but I have a performance issue/question.
The first thing my code does, is checking if the Samba folder exists. If it does, it instantiates the file it is going to write. Then more irrelevant stuff happens.
If I check my log file, there are always 6 seconds between Instantiated destination samba directory... and Checking if the file already exists.... This means that the if(!destination.exists) takes 6 seconds, which looks awefully long, since the if(smbOutputFile.exists)) doesn't even take 1 second. (In my testcase, both of them do exist).
What could be any of the factors of this performance issue? And is there a way to speed this up?
SmbFile destinationShare = new SmbFile(sambaDestinationPath, destinationAuthentication);
logger.info("Instantiated destination samba directory : " + destinationShare);
if (!destinationShare.exists()) { //destinationShare = a directory
destinationShare.mkdir();
logger.debug("Shared directory created");
}
smbOutputFile = new SmbFile(sambaDestinationPath + filename, destinationAuthentication);
logger.debug("Checking if the file already exists and rename it to '.old'");
if (smbOutputFile.exists()) { //smbOutputFile = a file
//Do something
}
Thanks!
I would like to ask if its possible to put text files into my jar, I use them to make my map in my game, but users can get Highscores. now I want to save the Highscores with the map, so I have to save the map on the user their PC. Is there any way how I could do this? I've searched the internet for some ideas but I could not find anything that even came close to what I've wanted. I only had 3/4th of a year java so I don't know much about these things, everything that happens outside the debug of eclipse are problems for me(files are mainly one of those things, null exceptions, etc).
The main question now.
Is it possible to do? If yes, do you have any terms I could search on, or some sites/guides/tutorials? If no, is there any other way how I could save the highscores?
EDIT:
to make clear
Can I get the text file (the text inside the file) to be extracted to a different file in like the home directory of my game (where I save the settings and stuff) the basic maps are inside the jar file, so I want them to be extracted on the first start-up of the program
Greetings Carolien
"extracted to a different file in like the home directory of my game (where i save the settings and stuff) the basic maps are inside the jar file, so i want them to be extracted on the first startup of the program"
You can get the URL by using getClass().getResource()
URL url = getClass().getResource("/res/myfile.txt");
Then create a File object from the URI of the URL
File file = new File(url.toURI());
Then just perform your normal file operations.
if (file.renameTo(new File(System.getProperty("user.home") + "\\" + file.getName()))) {
System.out.println("File is moved successful!");
} else {
System.out.println("File is failed to move!");
}
Assuming your file structure is like below, it should work fine
ProjectRoot
src
res
myfile.txt
Note: the above is moving the entire file. If you want to extract just the data inside the file, then you can simple use
InputStream is = getClass().getResourceAsStream("/res/myfile.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
The just do normal IO operation with the reader. See here for help with writing the file.
I am in the process of making a program, which allows you to view your file system.
I was testing it, and ran into a problem: It was saying a directory called "Documents and Settings" was on my C:\ drive, while it wasn't there.
This is how I get my file array:
File f = new File(path); //path being a path sent by the client, for example C:\
if(f.isFile()){
//TODO start downloading it.
out.println("ERR: no dir!");
return;
}
Server.log.log("System path requested: " + f.getAbsolutePath());
File[] files = f.listFiles();
for(int i = 0; i < files.length; i++){
File found = files[i];
if(!found.exists()){
continue;
}
if(found.isDirectory()){
out.println("dir:" + found.getName());
}else{
out.println(found.getName());
}
System.out.println("Printed " + found.getName());
}
out.println("ENDOFLIST"); //Notify the client it has to stop receiving data
For some reason, this outputs quite a lot of directories that I can't seem to find, even with the "Show hidden folders" option on.
When trying to access these directories, it tries to read the contents of the directory, but since the directory doesn't exist it throws an exception, causing no data to get sent over sockets and my client freezing.
My question is: Is there a way to either check if the file/directory REALLY exists? Note, if you look at my code block, if the file/dir doesn't exist it already continues instead of writing it to the socket.
I've given it a google, but no matches were found. Also, I've given the search function a go, but it didn't come up with anything similar.
These are hidden system folders.
They do exist. Really.
You get exceptions because a lot of them don't have read access.
I suggest to use the new Fil I/O API introduced by Java 7, it features greatly improved support of the features a specific file system offers. It also offers the possibility to use walk the file tree.
Have a look at the FileVisitor http://docs.oracle.com/javase/7/docs/api/java/nio/file/FileVisitor.html that will greatly help you.