Develop Reference

How to read/copy a (partially locked) log file, or at least the unlocked parts?

I am working on a utility that zips up a number of files (for diagnostics purposes). At it's core, it uses the following function:
private void write(ZipOutputStream zipStream, String entryPath, ByteSource content) throws IOException {
try (InputStream contentStream = content.openStream()) {
zipStream.putNextEntry(new ZipEntry(entryPath));
ByteStreams.copy(contentStream, zipStream);
zipStream.closeEntry();
}
}
But one of the files I want to read is a log file that another application runs and locks. Because that file is locked, I get an IO exception.
<ERROR>java.io.IOException: The process cannot access the file because another process has locked a portion of the file
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:257)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:112)
If I am willing to accept that I might get some garbage because of conflicts between my reads and the other application's writes, what is the best/easiest way to work around this? Is there a file reader that ignores locks or perhaps only reads all the unlocked sections only?
Update -- To clarify, I am looking to read a log file, or as much of it as possible. So, I could just start reading the file, wait until I get a block I can't read, catch the error, append a file end and go. Notepad++ and other programs can read files that are partially locked. I'm just looking for a way to do that without re-inventing the ByteStreams.copy function to create a "Copy as much as I can" function.
I should have perhaps asked "How to read all the unlocked parts of a log file" and I will update the title.

One possible answer (which I don't like) is to create a method almost identical to ByteStreams.copy(), which I call "copyUntilLock" which catches any IOException, then it checks to see if the exception is a because another process has locked a portion of the file.
If that is the case, then simply stop writing and return the number of bytes so far. If its some other exception go ahead and throw it. (You could also write a note to the stream like "READING FAILED DUE TO LOCK").
Still looking for a better answer. Code included below.
private static long copyUntilLock (InputStream from, OutputStream to) throws IOException {
checkNotNull(from);
checkNotNull(to);
byte[] buf = createBuffer();
long total = 0;
try {
while (true) {
int r = from.read(buf);
if (r == -1) {
break;
}
to.write(buf, 0, r);
total += r;
}
return total;
} catch (IOException iox) {
if (iox.getMessage() != null && iox.getMessage().contains("another process has locked a portion of the file")) {
return total;
} else {
throw iox;
}
}
}

OutputStream not writing mp3 fully

public void getMp4FromYoutube(String httpPath, String absolutePath, String jobName, String email) {
try {
byte[] mp3ByteArray = youtubeToMP3(httpPath);
File mp3File = new File("src/main/resources/audio.mp3");
OutputStream os = new FileOutputStream(mp3File);
os.write(mp3ByteArray);
os.flush();
os.close();
convertToBase64AndSend(jobName, mp3File, email);
} catch (IOException e) {
e.printStackTrace();
}
}
In my method, I need it to write an mp3 of a youTube page. For anyone who is worrying, its not to copy videos - I've written a transcription that I want to now download the audio, translate, and delete. When I run this method without continuing the code (though commenting out convertToBase64AndSend) it downloads the whole video audio. However, when I continue to the convertToBase64AndSend method to continue with the translation it only downloads one second of the audio. I'm assuming its something to do with the OutputStream and the way I'm using it? If anybody has any ideas I'd be very grateful. Thanks.

Rather than calling the convertToBase64AndSend method in the getMp4FromYoutube method I called it in the class that had called getMp4FromYouTube. Worked for some reason but I have no idea why! Anybody can give me an explanation I'd be grateful!

ParcelFileDescriptor.createPipe in JNI

I'm trying to figure out how I can pass a stream of data within ContentProvider.openFile. The data to be sent is created in JNI. I tried createPipe with a transfer thread but I had a ton of trouble with broken pipes. So I thought I might just pass the 'write' pipe to JNI and write the data directly to it.
Java:
ParcelFileDescriptor[] pipe = ParcelFileDescriptor.createPipe();
boolean result = ImageProcessor.getThumb(fd/*source fd*/, pipe[1].getFd()); //JNI call (formerly returned a byte[])
return pipe[0];
C:
unsigned char* jpeg = NULL;
unsigned long jpegSize = 0;
getThumbnail(env, &jpeg, &jpegSize, rawProcessor); // Populates jpeg thumb, works when converted to byte[] in second segment
FILE* out = fdopen(dest, "wb");
int written = fwrite(jpeg, 1, jpegSize, out);
return TRUE;
When I convert to byte[] everything works fine, just not within a ContentProvider obviously:
jbyteArray thumb = env->NewByteArray(jpegSize);
env->SetByteArrayRegion(thumb, 0, jpegSize, (jbyte *) jpeg);
free(jpeg);
return thumb;
When I debug it gets to fwrite then the stack trace just seems to disappear. Never hits return TRUE or return pipe[0], but also doesn't crash or throw. Very strange...
Has anyone done something similar? Is it sufficient to simply write binary to the "write" pipe? Am I doing anything fundamentally wrong here? Thanks.
Update (after discussion with #pskink)
I tried implementing the PipeDataWriter. I used FileProvider.java as an example.
#Override
public void writeDataToPipe(#NonNull ParcelFileDescriptor output, #NonNull Uri uri, #NonNull String mimeType, #Nullable Bundle opts, #Nullable byte[] args)
{
try (FileOutputStream fout = new FileOutputStream(output.getFileDescriptor()))
{
fout.write(args, 0, args.length);
}
catch (IOException e)
{
Log.e(TAG, "Failed transferring", e);
}
}
byte[] rawData = ImageUtil.getRawThumb(fd.getParcelFileDescriptor().getFd());
return openPipeHelper(Uri.parse("invalid"), "image/jpg", null, rawData, this);
However, I'm getting the same errors I got when I used the transfer thread above:
java.io.IOException: write failed: EBADF (Bad file descriptor)
at libcore.io.IoBridge.write(IoBridge.java:498)
at java.io.FileOutputStream.write(FileOutputStream.java:186)
at
com.anthonymandra.content.MetaProvider.writeDataToPipe(MetaProvider.java:273)
and
java.io.IOException: write failed: EPIPE (Broken pipe)
at libcore.io.IoBridge.write(IoBridge.java:498)
at java.io.FileOutputStream.write(FileOutputStream.java:186)
at
com.anthonymandra.content.MetaProvider.writeDataToPipe(MetaProvider.java:273)
When I stepped through to make sure the data was fine for the images I found that everything loaded fine. It looks to me like this is actually a thread safety issue.

There were actually a bunch of things going wrong that all rolled up into a confusing mess:
I wasn't closing the ParcelFileDescriptor in a finally.
I use Glide for an image cache and it uses two fetchers when you load a Uri, meaning openFile was being called twice per file.
(2) caused endless broken pipe errors.
StrictMode was killing the app because of (1) and I missed it in the flurry of errors from (3).

Accessing Windows disks directly with Java NIO

I am using a library that uses Java NIO in order to directly map files to memory, but I am having trouble reading disks directly.
I can read the disks directly using FileInputStream with UNC, such as
File disk = new File("\\\\.\\PhysicalDrive0\\");
try (FileInputStream fis = new FileInputStream(disk);
BufferedInputStream bis = new BufferedInputStream(fis)) {
byte[] somebytes = new byte[10];
bis.read(somebytes);
} catch (Exception ex) {
System.out.println("Oh bother");
}
However, I can't extend this to NIO:
File disk = new File("\\\\.\\PhysicalDrive0\\");
Path path = disk.toPath();
try (FileChannel fc = FileChannel.open(path, StandardOpenOption.READ)){
System.out.println("No exceptions! Yay!");
} catch (Exception ex) {
System.out.println("Oh bother");
}
The stacktrace (up to the cause) is:
java.nio.file.FileSystemException: \\.\PhysicalDrive0\: The parameter is incorrect.
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:334)
at hdreader.HDReader.testcode(HDReader.java:147)
I haven't been able to find a solution, though I saw something close on How to access specific raw data on disk from java. The answer by Daniel Alder suggesting the use of GLOBALROOT seems to be relevant, as the answer uses FileChannel in the answer, but I can't seem to find the drive using this pattern. Is there a way to list all devices under GLOBALROOT or something like that?
At the moment I am looking at replacing uses of NIO with straight InputStreams, but I want to avoid this if I can. Firstly, NIO was used for a reason, and secondly, it runs through a lot of code and will require a lot of work. Finally, I'd like to know how to implement something like Daniel's solution so that I can write to devices or use NIO in the future.
So in summary: how can I access drives directly with Java NIO (not InputStreams), and/or is there a way to list all devices accessible through GLOBALROOT so that I might use Daniel Alser's solution?
Summary of Answers:
I have kept the past edits (below) to avoid confusion. With the help of EJP and Apangin I think I have a workable solution. Something like
private void rafMethod(long posn) {
ByteBuffer buffer = ByteBuffer.allocate(512);
buffer.rewind();
try (RandomAccessFile raf = new RandomAccessFile(disk.getPath(), "r");
SeekableByteChannel sbc = raf.getChannel()) {
sbc.read(buffer);
} catch (Exception ex) {
System.out.println("Oh bother: " + ex);
ex.printStackTrace();
}
return buffer;
}
This will work as long as the posn parameter is a multiple of the sector size (set at 512 in this case). Note that this also works with the Channels.newChannel(FileInputStream), which seems to always return a SeekableByteStream in this case and it appears it is safe to cast it to one.
From quick and dirty testing it appears that these methods truly do seek and don't just skip. I searched for a thousand locations at the start of my drive and it read them. I did the same but added an offset of half of the disk size (to search the back of the disk). I found:
Both methods took almost the same time.
Searching the start or the end of the disk did not affect time.
Reducing the range of the addresses did reduce time.
Sorting the addresses did reduce time, but not by much.
This suggests to me that this is truly seeking and not merely reading and skipping (as a stream tends to). The speed is still terrible at this stage and it makes my hard drive sound like a washing machine, but the code was designed for a quick test and has yet to be made pretty. It may still work fine.
Thanks to both EJP and Apangin for the help. Read more in their respective answers.
Edit:
I have since run my code on a Windows 7 machine (I didn't have one originally), and I get a slightly different exception (see below). This was run with admin privileges, and the first piece of code still works under the same conditions.
java.nio.file.FileSystemException: \\.\PhysicalDrive0\: A device attached to the system is not functioning.
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at testapp.TestApp.doStuff(TestApp.java:30)
at testapp.TestApp.main(TestApp.java:24)
Edit 2:
In response to EJP, I have tried:
byte[] bytes = new byte[20];
ByteBuffer bb = ByteBuffer.wrap(bytes);
bb.rewind();
File disk = new File("\\\\.\\PhysicalDrive0\\");
try (FileInputStream fis = new FileInputStream(disk);
ReadableByteChannel rbc = Channels.newChannel(new FileInputStream(disk))) {
System.out.println("Channel created");
int read = rbc.read(bb);
System.out.println("Read " + read + " bytes");
System.out.println("No exceptions! Yay!");
} catch (Exception ex) {
System.out.println("Oh bother: " + ex);
}
When I try this I get the following output:
Channel created
Oh bother: java.io.IOException: The parameter is incorrect
So it appears that I can create a FileChannel or ReadableByteChannel, but I can't use it; that is, the error is simply deferred.

When accessing physical drive without buffering, you can read only complete sectors. This means, if a sector size is 512 bytes, you can read only multiple of 512 bytes. Change your buffer length to 512 or 4096 (whatever your sector size is) and FileChannel will work fine:
ByteBuffer buf = ByteBuffer.allocate(512);
try (RandomAccessFile raf = new RandomAccessFile("\\\\.\\PhysicalDrive0", "r");
FileChannel fc = raf.getChannel()) {
fc.read(buf);
System.out.println("It worked! Read bytes: " + buf.position());
} catch (Exception e) {
e.printStackTrace();
}
See Alignment and File Access Requirements.
Your original FileInputStream code works obviously because of BufferedInputStream which has the default buffer size of 8192. Take it away - and the code will fail with the same exception.

Using NIO your original code only needs to change very slightly.
Path disk = Paths.get("d:\\.");
try (ByteChannel bc = Files.newByteChannel(disk, StandardOpenOption.READ)) {
ByteBuffer buffer = ByteBuffer.allocate(10);
bc.read(buffer);
} catch (Exception e){
e.printStackTrace();
}
Is fine, workable code, but I get an access denied error in both your version and mine.

Run this as administrator. It really does work, as it's only a thin wrapper over java.io:
try (FileInputStream fis = new FileInputStream(disk);
ReadableByteChannel fc = Channels.newChannel(fis))
{
System.out.println("No exceptions! Yay!");
ByteBuffer bb = ByteBuffer.allocate(4096);
int count = fc.read(bb);
System.out.println("read count="+count);
}
catch (Exception ex)
{
System.out.println("Oh bother: "+ex);
ex.printStackTrace();
}
EDIT If you need random access, you're stuck with RandomAccessFile. There's no mapping from that via Channels. But the solution above isn't NIO anyway, just a Java NIO layer over FileInput/OutputStream.

Problems writing to a unix pipe through Java

I am writing to a framebuffer located at "/dev/fb0". Everything works fine until I try to write again to the pipe using an OutputStream, which hangs the program. I have resolved this by closing the output stream and then recreating it, but this seems awfully slow and blunt.
Framebuffer.java
public class Framebuffer extends Autobuffer {
private FileOutputStream out = null;
private File pipe = null;
public Framebuffer() {
super(320, 240);
}
public Framebuffer(File pipe) {
super(320, 240);
try {
out = new FileOutputStream(pipe);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
this.pipe = pipe;
}
public void sync() throws IOException {
out.write(getBytes());
out.close();
out = new FileOutputStream(pipe);
}
}
Any ideas?
Thanks.

Firstly, unless something really weird is going on, "/dev/fb0" is a device file not a pipe. [This is a nitpick, but if you use the wrong terminology, 1) people won't understand you and 2) you will have difficulty searching for answers.]
Secondly, this looks like a weird way to interact with a framebuffer!!
I suspect that the problem is that you need to do the equivalent of a POSIX lseek call to set the stream position to zero each time you draw a frame. I've found two ways to do this:
Use RandomAccessFile instead of OutputStream / FileOutputStream, and call seek(long) to seek the file.
Call FileOutputStream.getChannel(), and then use position(long) to seek the file.

Changing the Output Stream to RandomAccessFile fixed all of my problems. I bet the stream wasn't working because it can't seek to position 0. Thanks to all who replied.

What if you flush your output with flush (from OutputStream)?