I need to rename a file (keeping it in the same directory);
I can't seem to find a way to see if my program has the required permissions:
Files.isWritable(directory) && Files.isWritable(oldFile);
always returns true, wether or not the running user has the permission to write the file (I guess they only check if the file is read-only, even if this violates the contract stated in the javadoc);
I'm not running under a security manager so I can't call
System.getSecurityManager.checkDelete(oldFile.toString());
I need to check if the renaming of several files will (probably) succeed so I can't just try and catch the exception.
Is there a way out? Obviously a portable solution would be lovable but I would settle for a windows-specific one...
Well, you can't check Windows ACLs that way. The "native" solution is fairly easy, since Windows supports transactions on the file system. Just call DeleteFileTransacted, and roll back the transaction if any one deletion fails.
If you're not using tranactions, then the second option is to first open handles with explicit DELETE desired access (DELETE is one of the standard WinNT access rights), denying any sharing. If and only if this succeeds for all files, delete them all with SetFileInformationByHandle(handle, FileDispositionInfo, &fdiObj, sizeof(fdiObj));
(The latter is not a transaction and may have Isolation issues as a result, which in turn affect Atomicity).
Try new SecurityManager().checkDelete(oldFile.toString()).
Just try to move it! If the move failed, you didn't have permissions, or something else went wrong.
This is a general principle. Don't try to foretell the future, guessing whether an impending operation will succeed. Try the operation. Otherwise you introduce all sorts of extra problems:
You might make the wrong test.
The condition might change between the test and the operation.
The operation usually returns an error or throws an exception anyway, which you have to write code to handle: why write it all twice?
Related
In some automated tests, I am trying to delete and immediately recreate an index at the start of every test, using ElasticSearch's high-level rest client (version 6.4), as follows:
DeleteIndexRequest deleteIndexRequest = new DeleteIndexRequest(indexName);
deleteIndexRequest.indicesOptions(IndicesOptions.lenientExpandOpen());
client.indices().delete(deleteIndexRequest, RequestOptions.DEFAULT);
CreateIndexRequest createIndexRequest = new CreateIndexRequest(indexName);
request.mapping("_doc", "{...}", XContentType.JSON);
client.indices().create(request, RequestOptions.DEFAULT);
The problem I have is that, intermittently, my tests fail at the point of creating the index, with an error:
{"error": {"root_cause":[{"type":"resource_already_exists_exception","reason":"index [(index-name)/(UUID)] already exists, ...,}] "status":400}
The more tests I run, the more likely I am to see the error, which seems to be a strong indicator that it's a race condition - presumably when I try to recreate the index, the previous delete operation hasn't always completed.
This is backed-up with the fact that if I put a breakpoint immediately after the delete operation, and manually run a curl request to look at the index that I tried to delete, I find that it's still there some of the time; on those occasions the error above appears if I continue the test.
I've tried asserting the isAcknowledged() method of the response to the delete operation, but that always returns true, even in cases when the error occurs.
I've also tried doing an exists() check before the create operation. Interestingly in that case if I run the tests without breakpoints, the exists() check always returns false (i.e. that the index doesn't exist) even in cases where the error will then occur, but if I put a breakpoint in before the create operation, then the exists() check returns true in cases where the error will happen.
I'm at a bit of a loss. As far as I understand, my requests should be synchronous, and from a comment on this question, this should mean that the delete() operation only returns when the index has definitely been deleted.
I suspect a key part of the problem might be that these tests are running on a cluster of 3 nodes. In setting up the client, I'm only addressing one of the nodes:
client = new RestHighLevelClient(RestClient.builder(new HttpHost("example.com", 9200, "https")));
but I can see that each operation is being replicated to the other two nodes.
When I stop a breakpoint before the create operation, in cases where the index is not deleted, I can see that it's not being deleted on any of the nodes, and it seems not to matter how long I wait, it never gets deleted.
Is there some way I can reliably determine whether the index has been deleted before I create it? Or perhaps something I need to do before I attempt the delete operation, to guarantee that it will succeed?
Hey I think there are quite a few things to think about. For one I'd test everything with curl or some kind of rest client till I start doing anything in code. Might just help you conceptually, but that's just my opinion.
This is one thing you should consider:
"If an external versioning variant is used, the delete operation automatically creates an index if it has not been created before (check out the create index API for manually creating an index)."
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete.html
Which kind of would explain why exists() would return false. So if external versioning variant is used then the delete option would actually create an index with the same name prior to deleting it.
You mentioned about the fact that you are working with a three node cluster. Something you can try is:
"When making delete requests, you can set the wait_for_active_shards parameter to require a minimum number of shard copies to be active before starting to process the delete request." Here is a super detailed explanation which is certainly worth reading: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-wait-for-active-shards
I suggest you try:
curl -X DELETE 127.0.0.1:9200/fooindex?wait_for_active_shards=3
You said you have 3 nodes in your cluster,so this means that:"...indexing operation will require 3 active shard copies before proceeding, a requirement which should be met because there are 3 active nodes in the cluster, each one holding a copy of the shard."
This check is probably not 100% water tight since according to the docs here:https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-wait-for-active-shards
"It is important to note that this setting greatly reduces the chances of the write operation not writing to the requisite number of shard copies, but it does not completely eliminate the possibility, because this check occurs before the write operation commences. Once the write operation is underway, it is still possible for replication to fail on any number of shard copies but still succeed on the primary. The _shards section of the write operation’s response reveals the number of shard copies on which replication succeeded/failed." so perhaps use this parameter, but have your code check the response to see if any operations failed.
Something you can also try is:
(I can't seem to find good documentation to back this info up)
This should be able to tell you if the cluster isn't ready to accept deletes.
curl -X DELETE 127.0.0.1:9200/index?wait_for_completion=true
I'm working on Java code that checks whether a file exists in the system and whether it's checked out. After these checks it calls the CHECKIN_UNIVERSAL service. This is where it stops. Checking in a new file works just fine, but it's the checking in of an existing file that's giving errors.
The specific error displayed (without making modifications to my original code) is !cscheckinitemexists. A bunch of googling turned up the solution to clear the data binder, yet then it comes up with the error that it cannot retrieve or use the security token.
Here's the code I use to clear and retrieve the data binder:
m_binder.clearResultSets();
m_binder.getLocalData().clear();
m_binder.setEnvironment(new IdcProperties(SharedObjects.getSecureEnvironment()));
What does the rest of your code look like? You can link to a Gist.
Generally, I have run into this due to data pollution (as you stated).
Is there a reason you are using m_binder instead of creating a brand new DataBinder?
After looking at your gist, you are using m_binder (the DataBinder from the service) to execute CHECKIN_UNIVERSAL. Don't do this. Use a separate DataBinder (as you did for the DOC_INFO_BY_NAME service call).
Either use requestBinder or a new DataBinder.
Another way to avoid this issue is to simply not look for the checkout. CHECKIN_UNIVERSAL supports a flag that checks out a content item if it's not already checked out.
Add the flag "isForceCheckout" to your binder, with a value of "1".
For example, this snippet throws a NullPointerException(!) on the stream.read() line, assuming the com.google package exists in a JAR somewhere (Guava, for example).
ClassLoader classLoader = getClass().getClassLoader();
URL resource = classLoader.getResource("com/google");
InputStream stream = resource.openStream();
System.out.println(stream.toString()); // Fine -- stream is not null
stream.read(); // NPE inside FilterInputStream.read()!
If com/google is swapped with a package that's in the file system rather than a JAR, then the snippet doesn't crash at all. In fact, it seems to read the files in that directory, separated by newlines, though I can't imagine that behaviour is specified anywhere.
Is there a way test if the resource path "com/google" points to a "normal" resource file or to a directory?
This is a bit of a mess due to some unspecified behaviour for the protocol handlers involved in loading these resources. In this particular situation, there are two: sun.net.www.protocol.file.Handler and sun.net.www.protocol.jar.Handler, and they each handle the directory case a bit differently. Based on some experiments, here's what they each do:
sun.net.www.protocol.file.Handler:
What this Handler does is open a FileURLConnection, which does exactly what you discovered it did when confronted with a directory. You can check if it's a directory just with:
if (resource.getProtocol().equals("file")) {
return new File(resource.getPath()).isDirectory();
}
sun.net.www.protocol.jar.Handler:
This Handler, on the other hand, opens a JarURLConnection which eventually makes its way to a ZipCoder. If you take a look at that code, you'll notice something interesting: jzentry will come back null from the native JNI call because the JAR zip file does not, in fact, contain a file called com/google, and so it returns null to the stream that wraps it.
However, there is a solution. Although the ZipCoder won't find com/google, it will find com/google/ (this is how most ZIP interfaces work, for some reason). In that case, the jzentry will be found, and it'll just return a null byte.
So, cutting through all these random implementation-specific behaviours, you can probably figure out if it's a directory by first trying to access the resource with a trailing / (which is what URLClassLoaders expect for directories anyway). If ClassLoader.getResource() returns non-null, then it's a directory. If it doesn't, try without the trailing slash. If it returns non-null, it's a file. If it still returns null, then it's not even an existing resource.
Kinda hacky, but I don't think there's anything better. I hope this helps!
There is no safe and generic way to detect this. When you use ClassLoader.getResource(), the ClassLoader can return practically anything in the URL, in principle even something you have never seen before if the ClassLoader implements its own URL scheme (and protocol).
Your only option is to analyze the URL returned by getResource(), the protocol should hint at what it is (e.g. "file://"). But beware, depending on environment it may return things you did not plan for.
But to just access a resource, you don't care where it comes from (you may care if you're debugging a configuration issue, but your code should not care).
In general you should not make assumptions about the returned InputStream's capabilities, i.e. do not rely on it supporting mark/reset etc. The only safe operation would be simply reading the Stream. If an IOException occurs during read it indicates a problem with access to the resource (network connection lost etc.).
EDIT: getResource() should IMO only return resources (e.g. files or zip file entries), but never directories (since they are not resources). However I wouldn't count on every possible ClassLoader to do so, and I'm not sure what the correct behavior is (if its even specified somewhere).
I think that there are 2 solutions.
Naive solution based on analysis of the path itself. If it ends with .jar or .zip or .war or .ear it is a file. Otherwise it is a directory. I think that this approach will work in 99.99% of cases unless somebody tries to make you you to fail on purpose. For example by defining soft link that looks like a directory but is a file or vise versa.
Try to mimic the JVM logic that interprets paths of classpath relatively to the current working directory. So, retrieve current working directory by using new File("."), then take classpath, split it and for each its element use new File(".", classPathElement) unless it is defined using absolute path.
Good luck with this.
I want to write some lines to a file, and I need each line of writing is a atomic operation.
For example, I have 3 lines:
111111111111111111111111
222222222222222222222222
333333333333333333333333
When I write them into a file line by line, the program may be exit by error, so the saved data may be:
11111111111111111111111
222222
This is not what I expected. I hope each line is a transaction, a atomic operation.
How should I do this?
Currently I use Java to do this.
There isn't a 100% reliable way to guarantee this.
I think the closest you can get is by calling flush() on the output stream and then sync() on the underlying file descriptor. Again, there are failure modes where this won't help.
If you really need atomic writing of new lines to a file, I guess the only way is to create a copy under a new name, write the new line and rename the new file to the original name. The rename operation is atomic, at least under POSIX. On Windows you would need to remove the original file before renaming, which bears the problem of not being able to restore the file if a problem occurs in the that process.
You can use flush/sync as #aix suggests. Otherwise (and better -- 99.999% reliable) is to use some sort of environment (such as a database) that includes transaction support and use commit.
We are validating XML files and depending on the result of the validation we have to move the file into a different folder.
When the XML is valid the validator returns a value and we can move the file without a problem. Same thing happens when the XML is not valid according to the schema.
If however the XML is not well formed the validator throws an exception and when we try to move the file, it fails. We believe there is still a handle in the memory somewhere that keeps hold of the file. We tried putting System.gc() before moving the file and that sorted the problem but we can't have System.gc() as a solution.
The code looks like this. We have a File object from which we create a StreamSource. The StreamSource is then passed to the validator. When the XML is not well formed it throws a SAXException. In the exception handling we use the .renameTo() method to move the file.
sc = new StreamSource(xmlFile);
validator.validate(sc);
In the catch we tried
validator.reset();
validator=null;
sc=null;
but still .renameTo() is not able to move the file. If we put System.gc() in the catch, the move will succeed.
Can someone enlight me how to sort this without System.gc()?
We use JAXP and saxon-9.1.0.8 as the parser.
Many thanks
Try creating a FileInputStream and passing that into StreamSource then close the FileInputStream when you're done. By passing in a File you have lost control of how/when to close the file handle.
When you set sc = null, you are indicating to the garbage collector that the StreamSource file is no longer being used, and that it can be collected. Streams close themselves in their destroy() method, so if they are garbage collected, they will be closed, and therefore can be moved on a Windows system (you will not have this problem on a Unix system).
To solve the problem without manually invoking the GC, simply call sc.getInputStream().close() before sc = null. This is good practice anyway.
A common pattern is to do a try .. finally block around any file handle usage, eg.
try {
sc = new StreamSource(xmlFile);
// check stuff
} finally {
sc.getInputStream().close();
}
// move to the appropriate place
In Java 7, you can instead use the new try with resources block.
Try sc.getInputStream().close() in the catch
All the three answers already given are right : you must close the underlying stream, either with a direct call to StramSource, or getting getting the stream and closing it, or creating the stream yourself and closing it.
However, I've already seen this happening, under windows, since at least three years : even if you close the stream, really every stream, if you try to move or delete the file, it will throw exception .. unless ... you explicitly call System.gc().
However, since System.gc() is not mandatory for a JVM to actually execute a round of garbage collection, and since even if it was the JVM is not mandated to remove all possible garbage object, you have no real way of being sure that the file can be deleted "now".
I don't have a clear explanation, I can only imagine that probably the windows implementation of java.io somehow caches the file handle and does not close it, until the handle gets garbage collected.
It has been reported, but I haven't confirmed it, that java.nio is not subject to this behavior, cause it has more low level control on file descriptors.
A solution I've used in the past, but which is quite a hack, was to :
Put files to delete on a "list"
Have a background thread check that list periodically, calla System.gc and try to delete those files.
Remove from the list the files you managed to delete, and keep there those that are not yet ready to.
Usually the "lag" is in the order of a few milliseconds, with some exceptions of files surviving a bit more.
It could be a good idea to also call deleteOnExit on those files, so that if the JVM terminates before your thread finished cleaning some files, the JVM will try to delete them. However, deleteOnExit had it's own bug at the time, preventing exactly the removal of the file, so I didn't. Maybe today it's resolved and you can trust deleteOnExit.
This is the JRE bug that i find most annoying and stupid, and cannot believe it is still in existence, but unfortunately I hit it just a month ago on windows Vista with latest JRE installed.
Pretty old, but some people may still find this question.
I was using Oracle Java 1.8.0_77.
The problem occurs on Windows, not on Linux.
The StreamSource instanciated with a File seems to automatically allocate and release the file resource when processed by a validator or transformer. (getInputStream() returns null)
On Windows moving a file into the place of the source file (deleting the source file) after the processing is not possible.
Solution/Workaround: Move the file using
Files.move(from.toPath(), to.toPath(), REPLACE_EXISTING, ATOMIC_MOVE);
The use of ATOMIC_MOVE here is the critical point. Whatever the reason ist, it has something to do with the annoying behavior of Windows locking files.