Develop Reference

Java: How to hold off read/writing a file while it's locked

I have a number of separate applications (all written in Java) that need to access a file for short periods of time. Some of these processes will need read access while others will need to update the file. So that all of these applications play nicely I want to write some code that will use OS locks to gain exclusive access to the file for the time that each application requires it.
The obvious way to do this RandomAccessFile myFile = new RandomAccessFile(file, "rw"), however this will fail if another process already has the lock. What I need is the ability to back off and try again.
I was hoping to write some code that uses channel.tryLock() to test if a lock has been taken out. The trouble is I need a channel, and I don't appear able to get that channel object with out taking out a lock!
Update
I need to find a way of checking to see if there is a lock on a file. I want to do this without throwing an exception.
A simplified version of my code is:
void myMethod(File myFile) {
try (
RandomAccessFile myFile = new RandomAccessFile(myFile, "rw"); // Exception here
FileChannel myChannel = myFile.getChannel();
FileLock myLock = lockFile(myChannel )
) {
// Stuff to read and/or write the file
}
}
private FileLock lockFile(FileChannel channel) throws Exception {
FileLock lock;
while (lock = channel.tryLock() == null) {
Thread.sleep(100);
}
return lock;
}
the problem is that if the file is locked it fails (by throwing an exception) on the highlighted line - before the point that can get a lock for the file.
Other variations for obtaining the channel such as FileChannel channel = FileChannel.open(myFile.toPath(), StandardOpenOption.WRITE, StandardOpenOption.APPEND) also throw an exception:
Exception in thread "main" java.nio.file.FileSystemException: path\to\my\file.txt: The process cannot access the file because it is being used by another process.
So how can I get a channel to test the lock with with out throwing an exception?

The standard way to use FileLock, is to open the file, e.g. via FileChannel.open, followed by tryLock. The presence of a lock does not prevent other processes from opening the file.
This can be demonstrated by the following program:
import java.io.IOException;
import java.nio.channels.*;
import java.nio.file.*;
class Locking {
public static void main(String[] args) throws IOException, InterruptedException {
if(args.length > 0) {
String me = String.format("%6s ", ProcessHandle.current());
Path p = Paths.get(args[0]);
try(FileChannel fc = FileChannel.open(p,
StandardOpenOption.WRITE, StandardOpenOption.APPEND)) {
FileLock l = fc.tryLock();
if(l == null) System.out.println(me + "could not acquire lock");
else {
System.out.println(me + "got lock");
Thread.sleep(3000);
System.out.println(me + "releasing lock");
l.release();
}
}
}
else {
Path p = Files.createTempFile("lock", "test");
String[] command = {
Paths.get(System.getProperty("java.home"), "bin", "java").toString(),
"-cp", System.getProperty("java.class.path"),
"Locking", p.toString()
};
ProcessBuilder b = new ProcessBuilder(command).inheritIO();
Process p1 = b.start(), p2 = b.start(), p3 = b.start();
p1.waitFor();
p2.waitFor();
p3.waitFor();
Files.delete(p);
}
}
}
which prints something alike
12116 got lock
13948 could not acquire lock
13384 could not acquire lock
12116 releasing lock
which can be demonstrated online on tio.run
While this program works the same under Windows, this operating system supports opening files unshared, preventing other processes from opening. If a different process has opened the file in that way, we can’t even open it to probe the locking state.
This is not the way, Java opens the file, however, there’s a non-standard open option to replicate the behavior, com.sun.nio.file.ExtendedOpenOption.NOSHARE_WRITE. In recent JDKs, it’s in the jdk.unsupported module.
When we run the following extended test program under Windows
import java.io.IOException;
import java.nio.channels.*;
import java.nio.file.*;
import java.util.HashSet;
import java.util.Set;
class LockingWindows {
public static void main(String[] args) throws IOException, InterruptedException {
if(args.length > 0) {
String me = String.format("%6s ", ProcessHandle.current());
Path p = Paths.get(args[0]);
Set<OpenOption> options
= Set.of(StandardOpenOption.WRITE, StandardOpenOption.APPEND);
if(Boolean.parseBoolean(args[1])) options = addExclusive(options);
try(FileChannel fc = FileChannel.open(p, options)) {
FileLock l = fc.tryLock();
if(l == null) System.out.println(me + "could not acquire lock");
else {
System.out.println(me + "got lock");
Thread.sleep(3000);
System.out.println(me + "releasing lock");
l.release();
}
}
}
else {
Path p = Files.createTempFile("lock", "test");
String[] command = {
Paths.get(System.getProperty("java.home"), "bin", "java").toString(),
"-cp", System.getProperty("java.class.path"),
"LockingWindows", p.toString(), "false"
};
ProcessBuilder b = new ProcessBuilder(command).inheritIO();
for(int run = 0; run < 2; run++) {
Process p1 = b.start(), p2 = b.start(), p3 = b.start();
p1.waitFor();
p2.waitFor();
p3.waitFor();
if(run == 0) {
command[command.length - 1] = "true";
b.command(command);
System.out.println("\nNow with exclusive mode");
}
}
Files.delete(p);
}
}
private static Set<OpenOption> addExclusive(Set<OpenOption> options) {
OpenOption o;
try {
o = (OpenOption) Class.forName("com.sun.nio.file.ExtendedOpenOption")
.getField("NOSHARE_WRITE").get(null);
options = new HashSet<>(options);
options.add(o);
} catch(ReflectiveOperationException | ClassCastException ex) {
System.err.println("opening exclusive not supported");
}
return options;
}
}
we will get something like
2356 got lock
6412 could not acquire lock
9824 could not acquire lock
2356 releasing lock
Now with exclusive mode
9160 got lock
Exception in thread "main" java.nio.file.FileSystemException: C:\Users\...\Temp\lock13936982436235244405test: The process cannot access the file because it is being used by another process
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
at LockingWindows.main(LockingWindows.java:148)
Exception in thread "main" java.nio.file.FileSystemException: C:\Users\...\Temp\lock13936982436235244405test: The process cannot access the file because it is being used by another process
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
at LockingWindows.main(LockingWindows.java:148)
9160 releasing lock
The similarity to the outcome of your test suggests that the Windows program you ran concurrently to your Java program did use such a mode.
For your Java programs, no such issue should arise, as long as you don’t use that mode. Only when you have to interact with another Windows program not using the collaborative locking, you have to deal with this.

read/write to the same file from multiple docker containers

I have 4 containers(java app) in 4 docker hosts. they need to read/write to the same file.
file lock cannot be used because of different OS.
So, I tried to create a .lock file, if one of the 4 containers created the .lock file, the other containers will have to wait.
But this still isn't working well. The other containers cannot see the .lock file created by other containers sometimes(not real-time).
Are there other solutions?

I suggest you rethink your assumptions:
What if you have not 4 but 400 containers?
What if they are on servers not sharing a file system?
The clean way to do this is to write a very basic server (if the load allows it, this can be nginx+PHP and be done in 10 minutes) that does the file writing, run this in another container and connect to it from the other containers. This will give you:
file locking easy and reliable, as the file is seen only by one server
scalability
clusterability
abstraction

Try File lock api to implement this. Demo in Java.
public void modifyFile() {
try {
File file = new File("/tmp/fileToLock.dat");
// Creates a random access file stream to read from, and optionally to write to
FileChannel channel = new RandomAccessFile(file, "rw").getChannel();
// Acquire an exclusive lock on this channel's file (blocks until lock can be retrieved)
FileLock lock = null;
// Attempts to acquire an exclusive lock on this channel's file (returns null or throws
// an exception if the file is already locked.
try {
lock = channel.tryLock();
if (null != lock) {
List<String> fileToString = FileUtils.readLines(file, StandardCharsets.UTF_8);
long l = 0l;
if (null != fileToString && fileToString.size() > 0) {
l = Long.valueOf(fileToString.get(fileToString.size() - 1));
}
l++;
FileUtils.writeStringToFile(file, String.valueOf(l) + "\r\n", StandardCharsets.UTF_8, true);
}
} catch (OverlappingFileLockException e) {
// thrown when an attempt is made to acquire a lock on a a file that overlaps
// a region already locked by the same JVM or when another thread is already
// waiting to lock an overlapping region of the same file
System.out.println("Overlapping File Lock Error: " + e.getMessage());
channel.close();
}
// release the lock
if (null != lock) {
lock.release();
}
// close the channel
channel.close();
} catch (IOException e) {
System.out.println("I/O Error: " + e.getMessage());
}
}

Disappointment when trying to lock file for reading

I'm trying to make each thread read its own text file.
private void burden() {
File mainFolder = new File("C:\\FilesToRead");
File[] files = mainFolder.listFiles();
String freeFile;
for (File file : files) {
FileChannel channel;
FileLock lock = null;
try {
channel = new RandomAccessFile(file, "r").getChannel();;
lock = channel.lock();
// Ok. We get the lock
readFile(file.getName());
} catch (OverlappingFileLockException e) {
continue; // File is open by someone else
} catch (FileNotFoundException f) {
} catch (IOException ex) {
} catch (NonWritableChannelException n) {
System.out.println("NonWritableChannelException");
} finally {
try {
lock.release();
} catch (IOException ex) {
System.out.println("IOException!");
}
}
}
} // burden();
I get this picture in the debugger:
The next step will bring me to the NonWritableChannelException.
I can't understand why as I tried to lock file for reading.

The javadoc gives the answer. You use .lock(), which is equivalent to calling .lock(0L, Long.MAX_VALUE, false).
And the third parameter, a boolean, is described as (emphasis mine):
true to request a shared lock, in which case this channel must be open for reading (and possibly writing); false to request an exclusive lock, in which case this channel must be open for writing (and possibly reading)
You have to .lock(0L, Long.MAX_VALUE, true)
Also, if you are using Java 7, use FileChannel.open()

you also need to lock the writable channel here so make your code,
channel = new RandomAccessFile(file, "rw").getChannel();
This should work.

Locking a process in Java

I have a Java servlet which calls another software (say S) over a TCP connection. This software S uses a network resource, and the output has to be retrived from a hyperlink(using wget).
Since it's the same hyperlink I need to download my result from (irrespective of the request), it results into incorrect results few requests. I basically need to lock the use of this network resource across different processes (I believe each call from the servlet is going to create a new process).
I tried to use ReentrantLock (but I guess it only works with threads and not accross processes).
Please let me know how can this be achieved.
Thanks

Here is how to do cross-process locking in Java. Adjust to your needs and add error/exception checking/handling as necessary.
// Tester
try {
if (crossProcessLockAcquire(SomeClassInYourApp.class, 3000)) {
// Success - This process now has the lock. (Don't keep it too long.)
}
else {
// Fail (Timeout) - Another process still had the lock after 3 seconds.
}
} finally {
crossProcessLockRelease(); // try/finally is very important.
}
// Acquire - Returns success ( true/false )
private static boolean crossProcessLockAcquire(final Class<?> c, final long waitMS) {
if (fileLock == null && c != null && waitMS > 0) {
try {
long dropDeadTime = System.currentTimeMillis() + waitMS;
File file = new File(lockTempDir, c.getName() + ".lock");
RandomAccessFile randomAccessFile = new RandomAccessFile(file, "rw");
FileChannel fileChannel = randomAccessFile.getChannel();
while (System.currentTimeMillis() < dropDeadTime) {
fileLock = fileChannel.tryLock();
if (fileLock != null) {
break;
}
Thread.sleep(250); // 4 attempts/sec
}
} catch (Exception e) {
e.printStackTrace();
}
}
return fileLock == null ? false : true;
}
// Release
private static void crossProcessLockRelease() {
if (fileLock != null) {
try {
fileLock.release();
fileLock = null;
} catch (IOException e) {
e.printStackTrace();
}
}
}
// Some class vars and a failsafe lock release.
private static File lockTempDir = new File(System.getProperty("java.io.tmpdir") + File.separator + "locks");
private static FileLock fileLock = null;
static {
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run(){
crossProcessLockRelease();
}
});
}

Why are you reusing this TCP connection? If it's easy to set up, just set one up every time you need it. For example, with an HTTP request, you should just make a new request every time.
My guess is that you have something static that shouldn't be, so multiple threads are using it when they should all have their own version.
If they're expensive, consider creating one-per-thread with ThreadLocal.
If even that doesn't work, and you don't mind threads blocking, just add "synchronized" to the method that's causing the problem.

The resource you are trying to lock has to support looking. It would be better if the service didn't need to be locked externally.
As a work around you can use a ServerSocket to lock a resource between processes.

Java - locking a file for exclusive access

My problem is this: I'm using a WatchService to get notified about new files in a specific folder, now if a file gets moved/copied or created in said folder an event gets triggered and the name of the new file gets returned. The problem now is, if I try to access the file and it is not fully there yet (e.g. the copy is still in progress) an exception gets raised. What i tried was to do something like this:
RandomAccessFile raf = new RandomAccessFile(fileName, "rw");
FileChannel fc = raf.getChannel();
FileLock lck = fc.lock();
But even if a lock gets acquired, sometimes still an Exception gets raised if I try to write to the file because another process has still an open handle to it.
Now, how can a file in Java be locked for truly exclusive access?

For me, the statement
RandomAccessFile raf = new RandomAccessFile(fileName, "rw");
returns a FileNotFoundException if I cannot acquire a lock on the file. I catch the filenotfound exception and treat it...
public static boolean isFileLocked(String filename) {
boolean isLocked=false;
RandomAccessFile fos=null;
try {
File file = new File(filename);
if(file.exists()) {
fos=new RandomAccessFile(file,"rw");
}
} catch (FileNotFoundException e) {
isLocked=true;
}catch (Exception e) {
// handle exception
}finally {
try {
if(fos!=null) {
fos.close();
}
}catch(Exception e) {
//handle exception
}
}
return isLocked;
}
you could run this in a loop and wait until you get a lock on the file. Wouldn't the line
FileChannel fc = raf.getChannel();
never reach if the file is locked? You will get a FileNotFoundException thrown..

its better not to use classes in thejava.io package, instead use the java.nio package .
The latter has a FileLock class that you can use for a lock to a FileChannel.
try {
// Get a file channel for the file
File file = new File("filename");
FileChannel channel = new RandomAccessFile(file, "rw").getChannel();
// Use the file channel to create a lock on the file.
// This method blocks until it can retrieve the lock.
FileLock lock = channel.lock();
/*
use channel.lock OR channel.tryLock();
*/
// Try acquiring the lock without blocking. This method returns
// null or throws an exception if the file is already locked.
try {
lock = channel.tryLock();
} catch (OverlappingFileLockException e) {
// File is already locked in this thread or virtual machine
}
// Release the lock - if it is not null!
if( lock != null ) {
lock.release();
}
// Close the file
channel.close();
} catch (Exception e) {
}