I'm using the java AWS SDK in order to download a big amount of files from one S3 bucket, edit the files, and copy them back to a different S3 bucket.
I think it's supposed to work fine, but there is one line that keeps throwing me exceptions:
when I use
myClient.getObject(myGetObjectRequest, myFile)
I get an AmazonClientException saying there are too many files open.
Now, each time I download a file, edit it and copy it back to the bucket, I delete the temporary files I create.
I'm assuming it's taking a few milliseconds to delete the file, and maybe that's why I'm getting these errors.
Or is it maybe because of the open files on Amazon's side?
Anyway, I made my application sleep for 3 seconds each time it encounters this exception, that way it would have time to close the files, but that just takes too much time. Even if I'll take it down to 1 second.
Has anybody encountered this problem?
What should I do?
Thanks
Do you actually call "myFile.close()" at some point?
Related
I'm currently trying to write a simple journal-like program in Java that allows me to add "entries" and be able to browse all the "entries" I have added since the very beginning. The problem is, if I run the program, add two entries, exit the program, and then run the program again, I want to be able to have access to the two entries I previously added. I guess my questions is then, how am I able to "save" (if that's the right word) the entries that I add so that they won't be wiped out every time the program terminates?
I did some looking around, and it appears there's a tool I can use called the Java Cache System, but I'm not entirely sure if that's what I need for my situation. I'd appreciate if somebody could point me in the right direction.
When you run the program and create the entries your storing them in primary storage aka RAM. As you have discovered these entries will not persist across different executions of your program.
You need to store the entries in secondary storage aka the hard drive. This can be done by writing the entries to a file saved on disk and then reading those entries upon startup of the program. Java provides several mechanisms to read and write files to the file system on a machine.
Some applications use a database to store information in a relational manner so that it is available via a SQL request, however I would recommend using a simple file to store your entries.
The simplest way would be to store this data somehow in a file, and then read it from the file when the application starts, a few simple examples on how to write/read from file:
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileReader.java
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileReader.txt
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileWriter.java
http://www2.cs.uic.edu/~sloan/CLASSES/java/MyFileWriter.txt
Now, you store your objects in memory instead of this you can try to serialize them to some format like xml. And then in next run load them from xml. Or you can try to use dataBase for storing objects.
I faced same problem in past but little bit different.I clearly understood your problem , My solution is whatever the journal you are entering and getting saved should be saved in a particular location in your Location such as "C:\Your_Directory\Journal_folder\"
so it will be easier when you initially enter the journal it stores in above location ,again if u exit and reopen the application just try to retrieve the data from the above Mentioned target Location.
therefore every time when ever you enter the application it retrieves the data from that location if not it displays empty
I have to create a jar with a java application that fulfills the following features:
There are xml data packed in the jar which are read the first time the application is started. with every consecutive start of the application the data are loaded from a dynamically created binary file.
A customer should not be able to reset the application to its primary state (e.g. if the binary file gets deleted for some reason, the application should fail to run again and give an error message).
All this should not depend on the os it is running on (which means e.g. setting a registry entry in windows won't do the job)
Summarizing I want to prevent a once started application to be reset in order to limit illegitimate reuse of the application.
Now to my ideas on how to accomplish that:
Delete the xml from the jar at the first run (so far I came to the understanding that it is not possible to let an application edit it's own jar. is that true?)
Set a variable/property/setting/whatever in the jar permanently at the first run (is that possible)
Any suggestions/ideas on how to accomplish that?
update:
I did not find a solution for this exact problem, but I found a simple workaround: along with my software I ship a certain file which gets changed after the program is started the first time. of course if someone keeps a copy of the original file he can always replace it and start over.
Any user able to delete the binary file, will, with enough time, also be able to revert any changes made in the jar. When the only existing part of the application is in the hand of the user, you won't able to prevent changes to it.
You can easily just store a backup of the original jar, make a copy, use that for one run, delete, copy the original jar, etc. You would need some sort of mechanism outside the users machine, like an activation server. The user gets one code to activate an account, and can't use that code again.
I want to keep track of multiple files in a location in a unix box, and delete them if no one is using them for a long time.
I am trying to search for reference counting but did not get much help on google.
I also saw FileTime in java 7 which can give me the last accessed time , but i have to the above work using java 6.
If anyone has any ideas or good reference to reference counting and how i can use it, it will be great.
You can use apache common IO. Set observer on your dir which checks the dir and notifying listeners of create, change or delete eventslisten. By this it is possible to track on which dir has been working by user and others are not.
Okay, so we want last access time.
You can have a read of Get the Last Access Time for a File for some further info.
You could try and have a look at http://jdevel.wordpress.com/2011/04/08/file-last-access-time-in-java-on-linux/ for a possible soultion, but this is Linux, not Unix.
Finally, you could borrow the same idea and use JNA or JNI
There is no module for File watching till JDK 1.6. What you can do is, you can write your own file watcher by seeing the
lastModified() method of the java.io.File class.
It returns you the last modified time as a long value and you keep a watch on this file during a regular interval. If the total time difference
(presentTime - lastModifiedTime)
exceeds you time criteria, you can delete the file.
Hi there
I'm building a little p2p program, so I want to make a file unable to be deleted while it is being downloaded. The simple solution is to use a lock, but then again I want it to be possible for multiple clients to download the file (meaning many thread can access the download method at the same time).
I hope the situation is clear.
any ideas of how to implement it?
Thanks!
Make use of java.util.concurrent.locks.ReentrantReadWriteLock. Use a java.util.concurrent.locks.ReentrantReadWriteLock.WriteLock fot the thread that is downloading the file and java.util.concurrent.locks.ReentrantReadWriteLock.ReadLock for the other threads that should access it while it its being downloaded.
There's a bunch of lock implementations at http://download.java.net/jdk7/docs/api/java/util/concurrent/locks/package-frame.html which should help if all possible deleters are in the same VM, but there's no flock equivalent in the java core libraries.
You can't handle the case when the deletion comes from a system call, that's unfortunately impossible in java
So either you cope with that and you guard your 'delete' method by a simple FileLockManager or whatever, or given the size of the file is small, you can copy it to another directory (1 temporary file per client/group of client for example) then the user can do whatever he wants with the original file
just my 2 cents
I have a programmatic solution for this problem
Keep a stack data structure for each file. Keep this synchronized.
Whenever a thread is invoked for downloading a file, it will push an element in the stack and when its finished it will pop the element.
Now the delete request for a particular file comes, it will always check the stack size and it succeeds only when the stack size is zero.
Problem with this approach : If a thread crashes due to some reason or the other, stack will always have an entry and that file will never get deleted.
I am developing a Java Desktop Application. This app needs a configuration to be started. For this, I want to supply a defaultConfig.properties or defaultConfig.xml file with the application so that If user doesn't select any configuration, then the application will start with the help of defaultConfig file.
But I am afraid of my application crash if the user accidentally edit the defaultConfig file. So Is there any mechanism through which I can check before the start of the application that whether the config file has changed or not.
How other applications (out in the market) deal with this type of situation in which their application depends on a configuration file?
If the user edited the config file accidentally or intentionally, then the application won't run in future unless he re-installs the application.
I agree with David in that using a MD5 hash is a good and simple way to accomplish what you want.
Basically you would use the MD5 hashing code provided by the JDK (or somewhere else) to generate a hash-code based on the default data in Config.xml, and save that hash-code to a file (or hardcode it into the function that does the checking). Then each time your application starts load the hash-code that you saved to the file, and then load the Config.xml file and again generate a hash-code from it, compare the saved hash-code to the one generated from the loaded config file, if they are the same then the data has not changed, if they are different, then the data has been modified.
However as others are suggesting if the file should not be editable by the user then you should consider storing the configuration in a manner that the user can not easily edit. The easiest thing I can think of would be to wrap the Output Stream that you are using to write the Config.xml file in a GZIP Output Stream. Not only will this make it difficult for the user to edit the configuration file, but it will also cause the Config.xml file to take up less space.
I am not at all sure that this is a good approach but if you want to go ahead with this you can compute a hash of the configuration file (say md5) and recompute and compare every time the app starts.
Come to think of it, if the user is forbidden to edit a file why expose it? Stick it in a jar file for example, far away from the user's eyes.
If the default configuration is not supposed to be edited, perhaps you don't really want to store it in a file in the first place? Could you not store the default values of the configuration in the code directly?
Remove write permissions for the file. This way the user gets a warning before trying to change the file.
Add a hash or checksum and verify this before loading file
For added security, you can replace the simple hash with a cryptographic signature.
From I have found online so far there seems to be different approaches code wise. none appear to be a 100 hundred percent fix, ex:
The DirectoryWatcher implements
AbstractResourceWatcher to monitor a
specified directory.
Code found here twit88.com develop-a-java-file-watcher
one problem encountered was If I copy
a large file from a remote network
source to the local directory being
monitored, that file will still show
up in the directory listing, but
before the network copy has completed.
If I try to do almost anything non
trivial to the file at that moment
like move it to another directory or
open it for writing, an exception will
be thrown because really the file is
not yet completely there and the OS
still has a write lock on it.
found on the same site, further below.
How the program works It accepts a ResourceListener class, which is FileListener. If a change is detected in the program a onAdd, onChange, or onDelete event will be thrown and passing the file to.
will keep searching for more solutions.