Develop Reference

How to read/copy a (partially locked) log file, or at least the unlocked parts?

I am working on a utility that zips up a number of files (for diagnostics purposes). At it's core, it uses the following function:
private void write(ZipOutputStream zipStream, String entryPath, ByteSource content) throws IOException {
try (InputStream contentStream = content.openStream()) {
zipStream.putNextEntry(new ZipEntry(entryPath));
ByteStreams.copy(contentStream, zipStream);
zipStream.closeEntry();
}
}
But one of the files I want to read is a log file that another application runs and locks. Because that file is locked, I get an IO exception.
<ERROR>java.io.IOException: The process cannot access the file because another process has locked a portion of the file
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:257)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:112)
If I am willing to accept that I might get some garbage because of conflicts between my reads and the other application's writes, what is the best/easiest way to work around this? Is there a file reader that ignores locks or perhaps only reads all the unlocked sections only?
Update -- To clarify, I am looking to read a log file, or as much of it as possible. So, I could just start reading the file, wait until I get a block I can't read, catch the error, append a file end and go. Notepad++ and other programs can read files that are partially locked. I'm just looking for a way to do that without re-inventing the ByteStreams.copy function to create a "Copy as much as I can" function.
I should have perhaps asked "How to read all the unlocked parts of a log file" and I will update the title.

One possible answer (which I don't like) is to create a method almost identical to ByteStreams.copy(), which I call "copyUntilLock" which catches any IOException, then it checks to see if the exception is a because another process has locked a portion of the file.
If that is the case, then simply stop writing and return the number of bytes so far. If its some other exception go ahead and throw it. (You could also write a note to the stream like "READING FAILED DUE TO LOCK").
Still looking for a better answer. Code included below.
private static long copyUntilLock (InputStream from, OutputStream to) throws IOException {
checkNotNull(from);
checkNotNull(to);
byte[] buf = createBuffer();
long total = 0;
try {
while (true) {
int r = from.read(buf);
if (r == -1) {
break;
}
to.write(buf, 0, r);
total += r;
}
return total;
} catch (IOException iox) {
if (iox.getMessage() != null && iox.getMessage().contains("another process has locked a portion of the file")) {
return total;
} else {
throw iox;
}
}
}

Resume last downloaded file upon Exception

I'm using Java SDK 1.11.534
In my tool I declared a download named 'down' using TransferManager,
since the call:
down.waitForCompletion();
is a blocking call and stops the ProgressBar acknowledgement by ProgressListener I had to introduce a SwingWorker as follows:
SwingWorker worker = new SwingWorker<Void,Integer>(){
#Override
protected void process(List<Integer> chunks) {
int j = chunks.get(chunks.size()-1);
if (i<=fileNum) jLabel4.setText("Scaricamento file " + i+ " di " + fileNum + " del DCP "+ DCPname+" in corso, attendere....");
else jLabel4.setText("Scaricamento DCP "+ DCPname+" completato con successo.");
}
#Override
protected Void doInBackground(){
for (S3ObjectSummary file: fileList){
if((!isPresent(destination,file.getKey().substring(file.getKey().lastIndexOf("/") + 1),file.getSize())) && (!(file.getKey().substring(0, file.getKey().length()-1).equals(DCPname)))){
publish(i);
GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, file.getKey());
down = tx.download(getObjectRequest,new File(percorso+File.separator + file.getKey().substring(file.getKey().lastIndexOf("/") + 1)));
down.addProgressListener(progressListener);
try {
down.waitForCompletion();
} catch (AmazonClientException ex) {
ex.printStackTrace();
tx.shutdownNow(true);
//jButton4.setEnabled(true);
jButton4.doClick();
} catch (InterruptedException ex) {
ex.printStackTrace();
tx.shutdownNow(true);
//jButton4.setEnabled(true);
jButton4.doClick();
}
i++;
}
This is a portion of the code where doInBackground() shows the operations to do.
It happens sometimes to have an AmazonClientException reporting:
Not all bytes from S3inputstream were read
And this leads to have a corrupted file and a stop of the program itself upon exception.
At the beginning of my code (not reported here) before reaching the SwingWorker declaration, I stated that when the jButton4 is clicked the action starts checking if there's a size mismatch between files in the download folder and the ones on Amazon s3 and if there's a truncated file it gets deleted and the name is added to the download list again.
So the only solution I've found so far is to add the following line code:
jButton4.doClick();
In the exception code, meaning when an exception is hit the progress restarts and checks for truncated files and restarts downloads adding such a file too.
My question is:
Is there any way in the SDK to resume or better cancel and then download file again upon exception without restarting the program? I find the usage of:
jButton4.doClick();
is not a professional way of coding.

You could extract the content of the click action method into a new method and call that method instead.

Problems with facebooks conceal library

I'm having issues with reading decrypted data from conceal. It looks like I can't correctly finish streaming.
I pretend there is some issue with conceal, because of when I switch my proxyStream (just the encryption part) to not run it through conceal, everything works as expected. I'm also assuming that writing is ok, there is no exception whatsoever and I can find the encrypted file on disk.
I'm proxying my data through contentprovider to allow other apps read decrypted data when the user wants it. (sharing,...)
In my content provider I'm using the openFile method to allow contentResolvers read the data
#Override
public ParcelFileDescriptor openFile(Uri uri, String mode) throws FileNotFoundException {
try {
ParcelFileDescriptor[] pipe = ParcelFileDescriptor.createPipe();
String name = uri.getLastPathSegment();
File file = new File(name);
InputStream fileContents = mStorageProxy.getDecryptInputStream(file);
ParcelFileDescriptor.AutoCloseOutputStream stream = new ParcelFileDescriptor.AutoCloseOutputStream(pipe[1]);
PipeThread pipeThread = new PipeThread(fileContents, stream);
pipeThread.start();
return pipe[0];
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
I guess in the Facebook app Facebook android team could be rather using a standard query() method with a byte array sent in MediaStore.MediaColumns() which is not suitable for me because of I'm not only encrypting media files and I also like the approach of streams better.
This is how I'm reading from the Inpustream. It's basically a pipe between two parcelFileDescriptors. The inputstream comes from conceal and it is a FileInputstream wrapped into a BufferedInputStream originaly.
static class PipeThread extends Thread {
InputStream input;
OutputStream out;
PipeThread(InputStream inputStream, OutputStream out) {
this.input=inputStream;
this.out=out;
}
#Override
public void run() {
byte[] buf=new byte[1024];
int len;
try {
while ((len=input.read(buf)) > 0) {
out.write(buf, 0, len);
}
input.close();
out.flush();
out.close();
}
catch (IOException e) {
Log.e(getClass().getSimpleName(),
"Exception transferring file", e);
}
}
}
I've tried other methods how to read the stream, so it really shouldn't be the issue.
Finally here's the exception I'm constantly ending up with. Do you know what could be the issue? It points to native calls, which I got lost in..
Exception transferring file
com.facebook.crypto.cipher.NativeGCMCipherException: decryptFinal
at com.facebook.crypto.cipher.NativeGCMCipher.decryptFinal(NativeGCMCipher.java:108)
at com.facebook.crypto.streams.NativeGCMCipherInputStream.ensureTagValid(NativeGCMCipherInputStream.java:126)
at com.facebook.crypto.streams.NativeGCMCipherInputStream.read(NativeGCMCipherInputStream.java:91)
at com.facebook.crypto.streams.NativeGCMCipherInputStream.read(NativeGCMCipherInputStream.java:76)
EDIT:
It looks like the stream is working ok, but what fails is the last iteration of reading from it. As I'm using buffer it seems like the fact that the buffer is bigger then the amount of remaiming data is causing the issue. I've been looking into sources of conceal and it seems to be ok from this regard there. Couldn't it be failing somewhere in the native layer?
Note: I've managed to get the decrypted file except its final chunk of bytes..So I have for example an incomplete image file (with last few thousands of pixels not being displayed)

From my little experience with conceal, I have noticed that, only the same application that encrypts a file could decrypt it successfully irrespective whether it has the same package or not. Be sure to put this in mind

This was resolved in https://github.com/facebook/conceal/issues/24. For posterity's sake, the problem here is that the author forgot to call close() on the output stream.

Accessing Windows disks directly with Java NIO

I am using a library that uses Java NIO in order to directly map files to memory, but I am having trouble reading disks directly.
I can read the disks directly using FileInputStream with UNC, such as
File disk = new File("\\\\.\\PhysicalDrive0\\");
try (FileInputStream fis = new FileInputStream(disk);
BufferedInputStream bis = new BufferedInputStream(fis)) {
byte[] somebytes = new byte[10];
bis.read(somebytes);
} catch (Exception ex) {
System.out.println("Oh bother");
}
However, I can't extend this to NIO:
File disk = new File("\\\\.\\PhysicalDrive0\\");
Path path = disk.toPath();
try (FileChannel fc = FileChannel.open(path, StandardOpenOption.READ)){
System.out.println("No exceptions! Yay!");
} catch (Exception ex) {
System.out.println("Oh bother");
}
The stacktrace (up to the cause) is:
java.nio.file.FileSystemException: \\.\PhysicalDrive0\: The parameter is incorrect.
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:334)
at hdreader.HDReader.testcode(HDReader.java:147)
I haven't been able to find a solution, though I saw something close on How to access specific raw data on disk from java. The answer by Daniel Alder suggesting the use of GLOBALROOT seems to be relevant, as the answer uses FileChannel in the answer, but I can't seem to find the drive using this pattern. Is there a way to list all devices under GLOBALROOT or something like that?
At the moment I am looking at replacing uses of NIO with straight InputStreams, but I want to avoid this if I can. Firstly, NIO was used for a reason, and secondly, it runs through a lot of code and will require a lot of work. Finally, I'd like to know how to implement something like Daniel's solution so that I can write to devices or use NIO in the future.
So in summary: how can I access drives directly with Java NIO (not InputStreams), and/or is there a way to list all devices accessible through GLOBALROOT so that I might use Daniel Alser's solution?
Summary of Answers:
I have kept the past edits (below) to avoid confusion. With the help of EJP and Apangin I think I have a workable solution. Something like
private void rafMethod(long posn) {
ByteBuffer buffer = ByteBuffer.allocate(512);
buffer.rewind();
try (RandomAccessFile raf = new RandomAccessFile(disk.getPath(), "r");
SeekableByteChannel sbc = raf.getChannel()) {
sbc.read(buffer);
} catch (Exception ex) {
System.out.println("Oh bother: " + ex);
ex.printStackTrace();
}
return buffer;
}
This will work as long as the posn parameter is a multiple of the sector size (set at 512 in this case). Note that this also works with the Channels.newChannel(FileInputStream), which seems to always return a SeekableByteStream in this case and it appears it is safe to cast it to one.
From quick and dirty testing it appears that these methods truly do seek and don't just skip. I searched for a thousand locations at the start of my drive and it read them. I did the same but added an offset of half of the disk size (to search the back of the disk). I found:
Both methods took almost the same time.
Searching the start or the end of the disk did not affect time.
Reducing the range of the addresses did reduce time.
Sorting the addresses did reduce time, but not by much.
This suggests to me that this is truly seeking and not merely reading and skipping (as a stream tends to). The speed is still terrible at this stage and it makes my hard drive sound like a washing machine, but the code was designed for a quick test and has yet to be made pretty. It may still work fine.
Thanks to both EJP and Apangin for the help. Read more in their respective answers.
Edit:
I have since run my code on a Windows 7 machine (I didn't have one originally), and I get a slightly different exception (see below). This was run with admin privileges, and the first piece of code still works under the same conditions.
java.nio.file.FileSystemException: \\.\PhysicalDrive0\: A device attached to the system is not functioning.
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at testapp.TestApp.doStuff(TestApp.java:30)
at testapp.TestApp.main(TestApp.java:24)
Edit 2:
In response to EJP, I have tried:
byte[] bytes = new byte[20];
ByteBuffer bb = ByteBuffer.wrap(bytes);
bb.rewind();
File disk = new File("\\\\.\\PhysicalDrive0\\");
try (FileInputStream fis = new FileInputStream(disk);
ReadableByteChannel rbc = Channels.newChannel(new FileInputStream(disk))) {
System.out.println("Channel created");
int read = rbc.read(bb);
System.out.println("Read " + read + " bytes");
System.out.println("No exceptions! Yay!");
} catch (Exception ex) {
System.out.println("Oh bother: " + ex);
}
When I try this I get the following output:
Channel created
Oh bother: java.io.IOException: The parameter is incorrect
So it appears that I can create a FileChannel or ReadableByteChannel, but I can't use it; that is, the error is simply deferred.

When accessing physical drive without buffering, you can read only complete sectors. This means, if a sector size is 512 bytes, you can read only multiple of 512 bytes. Change your buffer length to 512 or 4096 (whatever your sector size is) and FileChannel will work fine:
ByteBuffer buf = ByteBuffer.allocate(512);
try (RandomAccessFile raf = new RandomAccessFile("\\\\.\\PhysicalDrive0", "r");
FileChannel fc = raf.getChannel()) {
fc.read(buf);
System.out.println("It worked! Read bytes: " + buf.position());
} catch (Exception e) {
e.printStackTrace();
}
See Alignment and File Access Requirements.
Your original FileInputStream code works obviously because of BufferedInputStream which has the default buffer size of 8192. Take it away - and the code will fail with the same exception.

Using NIO your original code only needs to change very slightly.
Path disk = Paths.get("d:\\.");
try (ByteChannel bc = Files.newByteChannel(disk, StandardOpenOption.READ)) {
ByteBuffer buffer = ByteBuffer.allocate(10);
bc.read(buffer);
} catch (Exception e){
e.printStackTrace();
}
Is fine, workable code, but I get an access denied error in both your version and mine.

Run this as administrator. It really does work, as it's only a thin wrapper over java.io:
try (FileInputStream fis = new FileInputStream(disk);
ReadableByteChannel fc = Channels.newChannel(fis))
{
System.out.println("No exceptions! Yay!");
ByteBuffer bb = ByteBuffer.allocate(4096);
int count = fc.read(bb);
System.out.println("read count="+count);
}
catch (Exception ex)
{
System.out.println("Oh bother: "+ex);
ex.printStackTrace();
}
EDIT If you need random access, you're stuck with RandomAccessFile. There's no mapping from that via Channels. But the solution above isn't NIO anyway, just a Java NIO layer over FileInput/OutputStream.

Most efficient merging of 2 text files.

So I have large (around 4 gigs each) txt files in pairs and I need to create a 3rd file which would consist of the 2 files in shuffle mode. The following equation presents it best:
3rdfile = (4 lines from file 1) + (4 lines from file 2) and this is repeated until I hit the end of file 1 (both input files will have the same length - this is by definition). Here is the code I'm using now but this doesn't scale very good on large files. I was wondering if there is a more efficient way to do this - would working with memory mapped file help ? All ideas are welcome.
public static void mergeFastq(String forwardFile, String reverseFile, String outputFile) {
try {
BufferedReader inputReaderForward = new BufferedReader(new FileReader(forwardFile));
BufferedReader inputReaderReverse = new BufferedReader(new FileReader(reverseFile));
PrintWriter outputWriter = new PrintWriter(new FileWriter(outputFile, true));
String forwardLine = null;
System.out.println("Begin merging Fastq files");
int readsMerge = 0;
while ((forwardLine = inputReaderForward.readLine()) != null) {
//append the forward file
outputWriter.println(forwardLine);
outputWriter.println(inputReaderForward.readLine());
outputWriter.println(inputReaderForward.readLine());
outputWriter.println(inputReaderForward.readLine());
//append the reverse file
outputWriter.println(inputReaderReverse.readLine());
outputWriter.println(inputReaderReverse.readLine());
outputWriter.println(inputReaderReverse.readLine());
outputWriter.println(inputReaderReverse.readLine());
readsMerge++;
if(readsMerge % 10000 == 0) {
System.out.println("[" + now() + "] Merged 10000");
readsMerge = 0;
}
}
inputReaderForward.close();
inputReaderReverse.close();
outputWriter.close();
} catch (IOException ex) {
Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, "Error while merging FastQ files", ex);
}
}

Maybe you also want to try to use a BufferedWriter to cut down your file IO operations.
http://download.oracle.com/javase/6/docs/api/java/io/BufferedWriter.html

A simple answer is to use a bigger buffer, which help to reduce to total number of I/O call being made.
Usually, memory mapped IO with FileChannel (see Java NIO) will be used for handling large data file IO. In this case, however, it is not the case, as you need to inspect the file content in order to determine the boundary for every 4 lines.

If performance was the main requirement, then I would code this function in C or C++ instead of Java.
But regardless of language used, what I would do is try to manage memory myself. I would create two large buffers, say 128MB or more each and fill them with data from the two text files. Then you need a 3rd buffer that is twice as big as the previous two. The algorithm will start moving characters one by one from input buffer #1 to destination buffer, and at the same time count EOLs. Once you reach the 4th line you store the current position on that buffer away and repeat the same process with the 2nd input buffer. You continue alternating between the two input buffers, replenishing the buffers when you consume all the data in them. Each time you have to refill the input buffers you can also write the destination buffer and empty it.

Buffer your read and write operations. Buffer needs to be large enough to minimize the read/write operations and still be memory efficient. This is really simple and it works.
void write(InputStream is, OutputStream os) throws IOException {
byte[] buf = new byte[102400]; //optimize the size of buffer to your needs
int num;
while((n = is.read(buf)) != -1){
os.write(buffer, 0, num);
}
}
EDIT:
I just realized that you need to shuffle the lines, so this code will not work for you as is but, the concept still remains the same.