Develop Reference

Using Java can I programmatically undelete a file under Windows?

I'm successfully using Desktop.getDesktop().moveToTrash(File) under MacOS to delete files and then retrieve them from the Trash folder. I'd like to do the same under Windows. But I don't see a native Java way to access to Recycle Bin so I can undelete them.
Under MacOS I simply rename files from the Trash folder back to where they were. Is there a way I can do that with the Windows Recycle Bin? Or do something similar?

There's nothing in the core API. You have a bunch of options.
But first.. there's trash, and delete
"move to trash" means the file is literally undestroyable - as long as it is remains in the trashcan it remains on your disk. Said differently, if you have a completely filled up harddisk, trash 100GB worth of files, that disk is... still completely filled up. Possibly certain OSes have a 'bot' that runs when needed or on a schedule that truly deletes files in the trashcan that fit certain rules (such as 'deleted more than 30 days ago').
"Actually delete" means the disk space is now available - if you have a full disk, actually delete 100GB worth of files, then there's now 100GB available, but those files are STILL on disk!! - the only thing that 'delete' does, is flag the space. It doesn't overwrite the actual bits on disk with zeroes or random data or whatnot. Further use of this disk will eventually mean some other file is written 'over' the deleted file at which point it truly becomes entirely unrecoverable, but if you have some extremely sensitive data, you delete the files, then you toss the computer in the garbage bin, anybody who gets their hands on that machine can trivially recover your data. Because what 'delete' does is set a flag "I am deleted", nothing more. All you need to do to undo this, is to unset the flag.
The reason I mention this, is because you used the term 'undelete' in your question. Which usually means something else.
Verb
UnVerb
Action
Trash
Recover, Untrash, Restore, Put Back
Disk space remains unavailable. File visible in OS trash can tool
Delete
Undelete
Disk space is now available; data is still on disk but could be overwritten at any time.
Wipe
N/A
Data is overwritten with zeroes. Some wiping tools overwrite 7 times with random data to be really sure1
Trim
N/A
Pulse all cells on an SSD2 - intended to make data unrecoverable, applies only to SSDs
[1] This fights largely hypothetical trickery where you recover data by scanning for minute magnetic differences in potential. If it's doable at all it requires equipment that costs multiple millions. Still, if you're paranoid, you write random data, and repeat that procedure 7 to 30 times.
[2] SSDs are 'weird' in that they are overprovisioned: They can store more data than the label says. It's because SSDs work in 'cells' and there's a limit to how often a cell can be rewritten. Eventually a cell is near death (it's clear it's only got a few rewrites left in it), at which point the data is copied to an unused cell, and the near-death cell is marked off as no longer usable. The SSD is a 'fake harddrive', exposing itself as a simple contiguous block of writable and addressable space. It's like a mini computer, and will map write ranges to available cells. As a consequence, using basic OS/kernel driver calls to tell the SSD to write 7x random data over a given range of bits does not actually work, in that it is possible that there's a cell with the file data that's been marked as not to be used, and it won't be wiped. While somewhat hard to do, you can send special commands, so-called TRIM commands, to most SSDs to explicitly tell them to pulse-clear all cells on the entire drive, even the ones that have been marked as near-death. This low-level call to the SSD firmware is the only way to securely delete anything off of an SSD. Naturally, the whole point of this exercise is that you can't undo it.
So, to be clear, the one and only thing on this list that is meaningfully doable without writing extremely complex software that scans the raw disk out, byte for byte (which is a tool you should not be writing in java, as you'll be programming a lot towards the OS/arch which java is not good at), is the Untrash part: Undoing the 'trash' action.
Not available in basic java
... unfortunately even that is not available normally. There's an API to tell the OS to 'trash' a file, there is no API call to untrash it. That means you'll have to code an implementation for untrashing for each and every OS you want to support. On mac, you could handroll it by moving files out of ~/.Trash. On windows its a little trickier.
One "simple" (heh) way is to use JNI to write C code (targeting the windows API, to be compiled on windows with windows oriented C tools) that does the job, and then use JNI to call this C function (compiled to a .dll) on windows specifically. You can ship the DLL and simply not use it on non-windows OSes. You will have to compile this DLL once for every arch you want to target (presumably, x64, possibly x86 and aarch64 (ARM)). This is all highly complicated and requires some knowledge about how to write fairly low-level windows code.
Use command line tools
You can invoke command line tools. For example, windows has fsutil which can be used to make hard links. I think you can do it - C:\$Recycle.bin is the path, more or less. Where C is itself already a little tricky to attempt to find from java (you can have multiple disks in a system, so do you just scan for C:, D:, etc? But if the machine still has a CD-ROM drive that'll make it spin up which surely you didn't want. You can ask windows about what kind of disk a letter is, but this again requires JNI, it's not baked into java).
You could write most of the untrash functionality in a powershell script and then simply use java's ProcessBuilder to run it, and have it do the bulk of the work.
Use C:\$Recycle.bin
You can try accessing Paths.get("C:\$Recycle.bin") and see what happens. Presumably, you can just move files out of there. But note that each file has associated with it, knowledge of where it used to be. The files still have their extension but their names are mangled, containing only the drive letter they came from + a number. There's a separate mapping file that tells you where the file was deleted from and what its name was. You will have to open this mapping file and parse through it. You'll have to search the web to figure out what the format of this mapping file is. You'll have to take care not to corrupt it, and probably to remove the file you just recovered from it (without corrupting it).
Note that files in the recycle bin have all sorts of flags set, such as the system flag. You may have to write it all in powershell or batch scripts, and start those from java instead, to e.g. run ATTRIB.EXE to change properties first. Hopefully you can do it with the java.nio.file API which exposes some abilities to interact with windows-specific file flag stuff.
Build your own trashcan
In general it's a bad idea to use java to write highly-OS-specific tooling. You can do it, but it'll hurt the entire time. The designers of java didn't make it for this (Project Panama is trying to fix this, but it's not in JDK18 and won't be in 19 either, it's a few years off – and it wasn't really designed for this kind of thing either), and your average java coder wouldn't use it for this, so that means: Few to no libraries, and hard to find support.
Hence, it's a better idea to consider desktop java apps to do things more in its own way than your average desktop tool. Which can include 'having its own trashcan'. Let's say you have a code editor written in java, and it has a 'delete' feature. You're free to implement 'delete' by moving files to a trashcan dir you made, where you track (Via a DB or shadow files) when the delete occurred, who did it, and where the file came from. Then you build code that can move it back, and code that 'empties the trash', possibly on a schedule.
You can do all that simply with Files.move.

Is java.io.File.createNewFile() atomic in a network file system?

EDIT : Well, I'm back a bunch of months later, the lock mechanism that I was trying to code doesn't work, because createNewFile isn't reliable on the NFS. Check the answer below.
Here is my situation : I have only 1 application which may access the files, so I don't have any constraint about what other applications may do, but the application is running concurrently on several servers in the production environment for redundancy and performance purposes (a couple of machines are hosting each a couple of JVM with our apps).
Basically, what I need is to put some kind of flag in a folder to tell the other instances to leave this folder alone as another instance is already dealing with it.
Many search results are telling to use FileLock to achieve this, but I checked the Javadoc, and from my understanding it will not help much, since it's using the hosting OS's locking possibilities. So I doubt that it will help much since there are different hosting machines.
This question covers a similar subject : Java file locking on a network , and the accepted answer is recommending to implement your own kind of cooperative locking process (using the File.createNewFile() as asked by the OP).
The Javadoc of File.createNewFile() says that the process is atomically creating the file if it doesn't already exist. Does that work reliably in a network file system ?
I mean, how is it possible with the potential network lag to do both existence check and creation simultaneously ? :
The check for the existence of the file and the creation of the file if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the file.

No, createNewFile doesn't work properly on a network file system.
Even if the system call is atomic, it's only atomic regarding the OS, and not over the network.
Over the time, I got a couple of collisions, like once every 2-3 months (approx. once every 600k files).
The thing that happens is my program is running in 6 separates instances over 2 separate servers, so let's call them A1,A2,A3 and B1,B2,B3.
When A1, A2, and A3 try to create the same file, the OS can properly ensure that only one file is created, since it is working with itself.
When A1 and B1 try to create the same file at the same exact moment, there is some form of network cache and/or network delays happening, and they both get a true return from File.createNewFile().
My code then proceeds by renaming the parent folder to stop the other instances of the program from unnecessarily trying to process the folder and that's where it fails :
On A1, the folder renaming operation is successful, but the lock file can't be removed, so A1 just lets it like that and keeps on processing new incoming folders.
On B1, the folder renaming operation (File.renameTo(), can't do much to fix it) gets stuck in a infinite loop because the folder was already renamed (also causing a huge I/O traffic according to my sysadmin), and B1 is unable to process any new file until the program is rebooted.

The check for the existence of the file and the creation of the file if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the file.
That can be implemented easily via the open() system call or its equivalents in any operating system I have ever used.

I mean, how is it possible with the potential network lag to do both
existence check and creation simultaneously ?
There is a difference between simultaneously and atomically. Java doc is not saying anything about this function being a set of two simultaneous actions but two actions designed to work in atomic way. If this method is built to do two operations atomically than means file will never be created without checking file existence first and if file gets created by current call then it means there were no files present and if file doesn't get created that means there was already a file by that name.
I don't see a reason to doubt function being atomic or working reliably despite call being on network or local disk. Local call is equally unreliable - so many things can go wrong in an IO.
What you have to doubt is when trying to use empty file created by this function as a Lock as explained D-Mac's answer for this question and that is what explicitly mentioned in Java Doc for this function too.
You are looking for a directory lock and empty files working as a directory lock ( to signal other processes and threads to not touch it ) has worked quite well for me provided due care is taken to write logic to check for file existence,lock file clean up and orphaned locks.

Why java FileOutputStream's write() or flush() doesn't make NFS client really send data to NFS server?

My Java web application use NFS file system, I use FileOutputStream to open, write multiple chunks and then close the file.
From the profiler stats I found that stream.write(byte[] payload,int begin, int length) and even stream.flush() takes zero milliseconds. Only the method call stream.close() takes non-zero milliseconds.
It seems that java FileOutputStream's write() or flush() doesn't really cause NFS client to send data to NFS server. Is there any other Java class will make NFS client flush data in real time? or there is some NFS client tuning need to be done?

You are probably running into Unix client-side caching. There are lots of details here in the O'Reilly NFS book.
But in short:
Using the buffer cache and allowing async threads to cluster multiple buffers introduces some problems when several machines are reading from and writing to the same file. To prevent file inconsistency with multiple readers and writers of the same file, NFS institutes a flush-on-close policy:
All partially filled NFS data buffers for a file are written to the NFS server when the file is closed.
For NFS Version 3 clients, any writes that were done with the stable flag set to off are forced onto the server's stable storage via the commit operation.
NFS cache consistency uses an approach called close-to-open cache consistency - that is, you have to close the file before your server (and other clients) get a consistent up-to-date view of the file. You are seeing the downsides of this approach, which aims to minimize server hits.
Avoiding the cache is hard from Java. You'd need to set the file open() O_DIRECT flag if you're using Linux; see this answer for more https://stackoverflow.com/a/16259319/5851520, but basically it disables the client's OS cache for that file, though not the server's.
Unfortunately, the standard JDK doesn't expose O_DIRECT. as discussed here: Force JVM to do all IO without page cache (e.g. O_DIRECT) - essentially, use JNI youself or use a nice 3rd party lib. I've heard good things about JNA: https://github.com/java-native-access/jna ...
Alternatively, if you have control over the client mount point, you can use the sync mount option, as per NFS manual. It says:
If the sync option is specified on a mount point, any system call
that writes data to files on that mount point causes that data to be
flushed to the server before the system call returns control to user
space. This provides greater data cache coherence among clients, but
at a significant performance cost.
This could be what you're looking for.

Generally, Java's streams make no guarantee about the effects of flush apart from maybe the flushing of the buffers in the Java classes involved.
To overcome that limitation, Java NIO's Channels can be used, see e.g https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#force(boolean). However, if "the file does not reside on a local device then no such guarantee is made." And Java cannot make such a guarantee, because the underlying remote file system or protocol may not be able to provide that function at all. However, you should be able to achieve (almost) the same level of synchronization with force() that you'd get from a native O_DIRECT access that #SusanW mentioned.

Hack for a "real" Java flush on a remote/virtual disk

I'm looking for a "trick" or an "hack" to be certain that a file has been persisted on a remote disk, passing through vmware cache, NAS cache, etc.
Flushing and closing a FileOutputStream is not enough. I think Channel.force(true) is neither.
I'm thinking about something like these:
write the file and read back the file
write the file, check timestamp, rename the file, check for a different timestamp
write the file with "wrong content", overwrite with the original content, read it back and check the content
maybe someone had the same problem and found a solution.
My requirement is not to lose data. The java application works in this way:
accept a file from a remote source
add a digital signature and a certified timestamp creating a new file. If this file is lost it cannot be recreated in any way.
write this file to the storage
mark the file as signed on the database
tell the remote side that everything is ok
Tonight we had a crash and three transactions failed after step 5 but before the data was actually flushed to the remote store. So the database says that everything is fine, the remote side was told the same but 15 seconds of signed data was lost. And this is no good.
The correct solution could be to do a "synch mount" of the remote file-system. But this is not going to happen in a short time. Even in this case I do not completely trust this scenario given that the app is running on a VMWare server.
So I'd like to have a "best effort hack" to prevent (mitigate) incidents like this one.

Let's start with one assumption: you cannot guarantee any single write to any single disk. There are just too many layers of software and hardware between your write and the disk platter. And even if you could guarantee the write, you cannot guarantee that the data will be readable. It's possible that the disk will crash between the write and the read.
The only solution is redundancy, either provided by a framework (eg, RDMS) or your app.
When you receive and sign the file, you need to send it to multiple destinations on different physical hosts, and wait for them to reply that they saved the file. One of them might crash. Two of them might crash. How important the data is will determine how many remote hosts you need.
Incidentally, redundancy also applies to your database. The fact that a transaction committed does not mean that you'll be able to recover it after a database crash (although DBMS engineers have a lot more experience than either you or I in ensuring writes, all of it depends on a sysadmin who understands things like "logs and datafiles must reside on separate physical drives). I strongly recommend that you (redundantly) store enough metadata along with the file to be able to reconstruct the database entry.

How do you write zero copy in java? What are the main differences

I was reading about how you can use the java nio library to take advantage of file transfer/buffering at the O/S level which is called 'zero copy'.
What are the differences in how you create/write to files then?
Are there any drawbacks to using zero-copy?

zero copy means that your program will not transfer the data from the kernel space to the user space and so on. this is faster
nice article can be found here:
https://developer.ibm.com/articles/j-zerocopy/

Zero copy is a technique where the application is no longer the 'middleman' in transferring data from a disk to the socket. Applications that use zero copy request that the kernel copy the data directly from the disk file to the socket, without going through the application, which improves performance and reduces context switches.
It all depends on what the application will do with the data it reads from disks. If it is a web application serving a lot of static content by reading files and relaying them over sockets, then zero copy is the way to go in order to get better performance. However, if the application is using the data locally (either crunching it in some way and then writing it back, or displaying it locally without persisting it back), you would not use zero copy.
This IBM DeveloperWorks article about zero copy is a good read.
Other ways of file I/O in java are via the use of Stream classes based on the type of file you would want to read/write. This involves both buffered and unbuffered streams, although usually buffered streams promise better performance since they cause less I/O seek cycles and hence lesser context switches.