I guess I miss something, but I cannot understand how file locks work in Java. To be more exact - how it is implemented.
It seems I cannot acquire (even cannot attempt acquiring) two or more locks for the same file inside single JVM. First lock will be successfully acquired, all further attempts to acquire more locks will result in OverlapingFileLockException. Nevertheless it works for separate processes.
I want to implement data-storage backed by file-system which is intended to work with multiple concurrent requests (both read and write). I want to use file locks to lock on particular files in the storage.
It seems that I have to introduce one more synchronization (exclusive) on JVM-level and only then sync on files to avoid this exception.
Did anyone do anything like that?
I prepared simple test case to show what my problem is. I use Mac OS X, Java 6.
import junit.framework.*;
import javax.swing.*;
import java.io.*;
import java.nio.channels.*;
/**
* Java file locks test.
*/
public class FileLocksTest extends TestCase {
/** File path (on Windows file will be created under the root directory of the current drive). */
private static final String LOCK_FILE_PATH = "/test-java-file-lock-tmp.bin";
/**
* #throws Exception If failed.
*/
public void testWriteLocks() throws Exception {
final File file = new File(LOCK_FILE_PATH);
file.createNewFile();
RandomAccessFile raf = new RandomAccessFile(file, "rw");
System.out.println("Getting lock...");
FileLock lock = raf.getChannel().lock();
System.out.println("Obtained lock: " + lock);
Thread thread = new Thread(new Runnable() {
#Override public void run() {
try {
RandomAccessFile raf = new RandomAccessFile(file, "rw");
System.out.println("Getting lock (parallel thread)...");
FileLock lock = raf.getChannel().lock();
System.out.println("Obtained lock (parallel tread): " + lock);
lock.release();
}
catch (Throwable e) {
e.printStackTrace();
}
}
});
thread.start();
JOptionPane.showMessageDialog(null, "Press OK to release lock.");
lock.release();
thread.join();
}
/**
* #throws Exception If failed.
*/
public void testReadLocks() throws Exception {
final File file = new File(LOCK_FILE_PATH);
file.createNewFile();
RandomAccessFile raf = new RandomAccessFile(file, "r");
System.out.println("Getting lock...");
FileLock lock = raf.getChannel().lock(0, Long.MAX_VALUE, true);
System.out.println("Obtained lock: " + lock);
Thread thread = new Thread(new Runnable() {
#Override public void run() {
try {
RandomAccessFile raf = new RandomAccessFile(file, "r");
System.out.println("Getting lock (parallel thread)...");
FileLock lock = raf.getChannel().lock(0, Long.MAX_VALUE, true);
System.out.println("Obtained lock (parallel thread): " + lock);
lock.release();
}
catch (Throwable e) {
e.printStackTrace();
}
}
});
thread.start();
JOptionPane.showMessageDialog(null, "Press OK to release lock.");
lock.release();
thread.join();
}
}
From the Javadoc:
File locks are held on behalf of the
entire Java virtual machine. They are
not suitable for controlling access to
a file by multiple threads within the
same virtual machine.
You can only acquire a lock once per file. Locks are not re-entrant AFAIK.
IMHO: Using files to communicate between process is a very bad idea. Perhaps you will be able to get this to work reliably, let me know if you can ;)
I would have one and only one thread read/write in only one process.
Have you checked the documentation? The FileChannel.lock() method returns an exclusive lock across the entire file. If you want to have multiple locks active concurrently across different threads, then you cannot use this method.
Instead you need to use FileChannel.locklock(long position, long size, boolean shared) in order to lock a specific region of the file. This will allow you to have multiple locks active at the same time, provided that each one is applied to a different region of the file. If you attempt to lock the same region of the file twice, you will encounter the same exception.
Related
I want to exclusively lock text file from Java code, so I found following example:
public class Main
{
public static void main(String args[]) throws IOException
{
String strFilePath = "M:/Projects/SafeFile/ClientSide/dump/data6.txt";
writeFileWithLock(new File(strFilePath), "some_content");
}
public static void writeFileWithLock(File file, String content) {
// auto close and release the lock
try (RandomAccessFile reader = new RandomAccessFile(file, "rw");
FileLock lock = reader.getChannel().lock()) {
// Simulate a 10s locked
TimeUnit.SECONDS.sleep(10);
reader.write(content.getBytes());
} catch (IOException | InterruptedException e) {
e.printStackTrace();
}
}
}
When I double click on data6.txt before 10 seconds elapse, my expectation is that I will receive some message like "File already opened by other process" or something similar. But I manage to open it without any problems. Does anybody see what is wrong with this code? Thanks!
FileLock does not generally use mandatory locks. This means that FileLock will generally provide locking only against other applications that use FileLock or equivalent.
Depending on the OS, there may be different system calls to lock files. For example fcntl or lockf.
To summarize, you may not safely assume that FileLock is a mandatory lock in every platform.
I have a number of separate applications (all written in Java) that need to access a file for short periods of time. Some of these processes will need read access while others will need to update the file. So that all of these applications play nicely I want to write some code that will use OS locks to gain exclusive access to the file for the time that each application requires it.
The obvious way to do this RandomAccessFile myFile = new RandomAccessFile(file, "rw"), however this will fail if another process already has the lock. What I need is the ability to back off and try again.
I was hoping to write some code that uses channel.tryLock() to test if a lock has been taken out. The trouble is I need a channel, and I don't appear able to get that channel object with out taking out a lock!
Update
I need to find a way of checking to see if there is a lock on a file. I want to do this without throwing an exception.
A simplified version of my code is:
void myMethod(File myFile) {
try (
RandomAccessFile myFile = new RandomAccessFile(myFile, "rw"); // Exception here
FileChannel myChannel = myFile.getChannel();
FileLock myLock = lockFile(myChannel )
) {
// Stuff to read and/or write the file
}
}
private FileLock lockFile(FileChannel channel) throws Exception {
FileLock lock;
while (lock = channel.tryLock() == null) {
Thread.sleep(100);
}
return lock;
}
the problem is that if the file is locked it fails (by throwing an exception) on the highlighted line - before the point that can get a lock for the file.
Other variations for obtaining the channel such as FileChannel channel = FileChannel.open(myFile.toPath(), StandardOpenOption.WRITE, StandardOpenOption.APPEND) also throw an exception:
Exception in thread "main" java.nio.file.FileSystemException: path\to\my\file.txt: The process cannot access the file because it is being used by another process.
So how can I get a channel to test the lock with with out throwing an exception?
The standard way to use FileLock, is to open the file, e.g. via FileChannel.open, followed by tryLock. The presence of a lock does not prevent other processes from opening the file.
This can be demonstrated by the following program:
import java.io.IOException;
import java.nio.channels.*;
import java.nio.file.*;
class Locking {
public static void main(String[] args) throws IOException, InterruptedException {
if(args.length > 0) {
String me = String.format("%6s ", ProcessHandle.current());
Path p = Paths.get(args[0]);
try(FileChannel fc = FileChannel.open(p,
StandardOpenOption.WRITE, StandardOpenOption.APPEND)) {
FileLock l = fc.tryLock();
if(l == null) System.out.println(me + "could not acquire lock");
else {
System.out.println(me + "got lock");
Thread.sleep(3000);
System.out.println(me + "releasing lock");
l.release();
}
}
}
else {
Path p = Files.createTempFile("lock", "test");
String[] command = {
Paths.get(System.getProperty("java.home"), "bin", "java").toString(),
"-cp", System.getProperty("java.class.path"),
"Locking", p.toString()
};
ProcessBuilder b = new ProcessBuilder(command).inheritIO();
Process p1 = b.start(), p2 = b.start(), p3 = b.start();
p1.waitFor();
p2.waitFor();
p3.waitFor();
Files.delete(p);
}
}
}
which prints something alike
12116 got lock
13948 could not acquire lock
13384 could not acquire lock
12116 releasing lock
which can be demonstrated online on tio.run
While this program works the same under Windows, this operating system supports opening files unshared, preventing other processes from opening. If a different process has opened the file in that way, we can’t even open it to probe the locking state.
This is not the way, Java opens the file, however, there’s a non-standard open option to replicate the behavior, com.sun.nio.file.ExtendedOpenOption.NOSHARE_WRITE. In recent JDKs, it’s in the jdk.unsupported module.
When we run the following extended test program under Windows
import java.io.IOException;
import java.nio.channels.*;
import java.nio.file.*;
import java.util.HashSet;
import java.util.Set;
class LockingWindows {
public static void main(String[] args) throws IOException, InterruptedException {
if(args.length > 0) {
String me = String.format("%6s ", ProcessHandle.current());
Path p = Paths.get(args[0]);
Set<OpenOption> options
= Set.of(StandardOpenOption.WRITE, StandardOpenOption.APPEND);
if(Boolean.parseBoolean(args[1])) options = addExclusive(options);
try(FileChannel fc = FileChannel.open(p, options)) {
FileLock l = fc.tryLock();
if(l == null) System.out.println(me + "could not acquire lock");
else {
System.out.println(me + "got lock");
Thread.sleep(3000);
System.out.println(me + "releasing lock");
l.release();
}
}
}
else {
Path p = Files.createTempFile("lock", "test");
String[] command = {
Paths.get(System.getProperty("java.home"), "bin", "java").toString(),
"-cp", System.getProperty("java.class.path"),
"LockingWindows", p.toString(), "false"
};
ProcessBuilder b = new ProcessBuilder(command).inheritIO();
for(int run = 0; run < 2; run++) {
Process p1 = b.start(), p2 = b.start(), p3 = b.start();
p1.waitFor();
p2.waitFor();
p3.waitFor();
if(run == 0) {
command[command.length - 1] = "true";
b.command(command);
System.out.println("\nNow with exclusive mode");
}
}
Files.delete(p);
}
}
private static Set<OpenOption> addExclusive(Set<OpenOption> options) {
OpenOption o;
try {
o = (OpenOption) Class.forName("com.sun.nio.file.ExtendedOpenOption")
.getField("NOSHARE_WRITE").get(null);
options = new HashSet<>(options);
options.add(o);
} catch(ReflectiveOperationException | ClassCastException ex) {
System.err.println("opening exclusive not supported");
}
return options;
}
}
we will get something like
2356 got lock
6412 could not acquire lock
9824 could not acquire lock
2356 releasing lock
Now with exclusive mode
9160 got lock
Exception in thread "main" java.nio.file.FileSystemException: C:\Users\...\Temp\lock13936982436235244405test: The process cannot access the file because it is being used by another process
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
at LockingWindows.main(LockingWindows.java:148)
Exception in thread "main" java.nio.file.FileSystemException: C:\Users\...\Temp\lock13936982436235244405test: The process cannot access the file because it is being used by another process
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
at LockingWindows.main(LockingWindows.java:148)
9160 releasing lock
The similarity to the outcome of your test suggests that the Windows program you ran concurrently to your Java program did use such a mode.
For your Java programs, no such issue should arise, as long as you don’t use that mode. Only when you have to interact with another Windows program not using the collaborative locking, you have to deal with this.
I have 4 containers(java app) in 4 docker hosts. they need to read/write to the same file.
file lock cannot be used because of different OS.
So, I tried to create a .lock file, if one of the 4 containers created the .lock file, the other containers will have to wait.
But this still isn't working well. The other containers cannot see the .lock file created by other containers sometimes(not real-time).
Are there other solutions?
I suggest you rethink your assumptions:
What if you have not 4 but 400 containers?
What if they are on servers not sharing a file system?
The clean way to do this is to write a very basic server (if the load allows it, this can be nginx+PHP and be done in 10 minutes) that does the file writing, run this in another container and connect to it from the other containers. This will give you:
file locking easy and reliable, as the file is seen only by one server
scalability
clusterability
abstraction
Try File lock api to implement this. Demo in Java.
public void modifyFile() {
try {
File file = new File("/tmp/fileToLock.dat");
// Creates a random access file stream to read from, and optionally to write to
FileChannel channel = new RandomAccessFile(file, "rw").getChannel();
// Acquire an exclusive lock on this channel's file (blocks until lock can be retrieved)
FileLock lock = null;
// Attempts to acquire an exclusive lock on this channel's file (returns null or throws
// an exception if the file is already locked.
try {
lock = channel.tryLock();
if (null != lock) {
List<String> fileToString = FileUtils.readLines(file, StandardCharsets.UTF_8);
long l = 0l;
if (null != fileToString && fileToString.size() > 0) {
l = Long.valueOf(fileToString.get(fileToString.size() - 1));
}
l++;
FileUtils.writeStringToFile(file, String.valueOf(l) + "\r\n", StandardCharsets.UTF_8, true);
}
} catch (OverlappingFileLockException e) {
// thrown when an attempt is made to acquire a lock on a a file that overlaps
// a region already locked by the same JVM or when another thread is already
// waiting to lock an overlapping region of the same file
System.out.println("Overlapping File Lock Error: " + e.getMessage());
channel.close();
}
// release the lock
if (null != lock) {
lock.release();
}
// close the channel
channel.close();
} catch (IOException e) {
System.out.println("I/O Error: " + e.getMessage());
}
}
I want to read from a file that's actively being written by another application/process.
This code which I found in another question, does the job: It reads a file while it is being written and only read the new content.
The problem that it consumes a lot of CPU even if no data was added to the file. How can I optimize this code ?
Use a timer ? or a thread.sleep to pause ?
Another thing to add, the whole program I am trying to write reads a file in real-time and process its content. So this means that thread.sleep or the timer will pause my whole program. The ideal improvement I am looking for is not to wait few seconds, but for a certain event to happen => New data is added. Is this possible ?
public class FileReader {
public static void main(String args[]) throws Exception {
if(args.length>0){
File file = new File(args[0]);
System.out.println(file.getAbsolutePath());
if(file.exists() && file.canRead()){
long fileLength = file.length();
readFile(file,0L);
while(true){
if(fileLength<file.length()){
readFile(file,fileLength);
fileLength=file.length();
}
}
}
}else{
System.out.println("no file to read");
}
}
public static void readFile(File file,Long fileLength) throws IOException {
String line = null;
BufferedReader in = new BufferedReader(new java.io.FileReader(file));
in.skip(fileLength);
while((line = in.readLine()) != null)
{
System.out.println(line);
}
in.close();
}
}
The ideal improvement I am looking for is not to wait few seconds, but
for a certain event to happen => New data is added. Is this possible ?
Best solution : data pushing :
The application that produces the content should inform the other application as it may read the content.
You could use any channel that may convey the information between two distinct applications.
For example by using a specific file that the writer updates to notify new things to read.
The writer could write/overwrite in this file a update date and the reader would read the data file only if it doesn't read any content since this date.
A more robust way but with more overhead could be exposing a notification service from the reader side.
Why not a REST service.
In this way, the writer could notify the reader via the service as new content is ready.
Another thing to add, the whole program I am trying to write reads a
file in real-time and process its content. So this means that
thread.sleep or the timer will pause my whole program.
A workaround solution : data pulling performed by a specific thread :
You have probably a multi-core CPU.
So create a separate thread to read the produced file to allow other threads of your application to be runnable.
Besides, you also could performs some regular pause : Thread.sleep() to optimize the core use done by the reading thread.
It could look like :
Thread readingThread = new Thread(new MyReadingProcessing());
readingThread.start();
Where MyReadingProcessing is a Runnable :
public class MyReadingProcessing implements Runnable{
public void run(){
while (true){
readFile(...);
try{
Thread.sleep(1000); // 1 second here but choose the deemed reasonable time according the metrics of your producer application
}
catch(InterruptedException e){
Thread.currentThread().interrupt();
}
if (isAllReadFile()){ // condition to stop the reading process
return;
}
}
}
}
Instead of a busy wait loop use a WatchService on directory entries changing.
Path path = Paths.get("...");
try (WatchService watchService =
path.getFileSystem().newWatchService()) {
WatchKey watchKey = path.register(watchService,
StandardWatchEventKinds.ENTRY_MODIFY);
for (;;) { // watchKey.poll with timeout, or take, blocking
WatchKey taken = watchService.take();
for (WatchEvent<?> event : taken.pollEvents()) {
Path changedPath = (Path) event.context();
if (changedPath.equals(path)) {
...
}
}
boolean valid = taken.reset();
if (!valid) {
... unregistered
}
}
}
Note that the above has to be adapted to use poll or take.
We have a situation where in two different main() programs access the same file for read/write operation.
One of the program tries to serialize a HashMap into the file whereas the other program tries to read the same HashMap.
The aim is to prevent the read operation while the write operation is ongoing.
I am able to get a lock for the file using java.nio.channels.FileChannel.lock(). But now I am unable to write to the file using ObjectOutputStream, since the lock is acquired by the FileChannel.
The main method for the write looks like as given below:
public static void main(String args[]) {
try {
HashMap<String, Double> h = new HashMap<>();
File f = new File("C:\\Sayan\\test.txt");
FileChannel channel = new RandomAccessFile(f, "rw").getChannel();
FileLock lock = channel.lock();
System.out.println("Created File object");
ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(f));
System.out.println("Created output stream");
oos.writeObject(h);
lock.release();
} catch (Exception e) {
e.printStackTrace();
}
}
The read operation is also similar it just reads the data from the file modified by above code.
Constraints:
We cannot create any other class or threading for attaining synchronization.
One more doubt: Can threads created by two different main programs communicate with each other?
Get the channel you're locking with from the FileOutputStream instead of from a separate RandomAccessFile.