Develop Reference

Java-Shifting Characters from Binary Files

I looked at some previous threads about binary files and I am doing the dataStream like it says, but I am not for sure why mine isn't working as I think I am doing the same thing as threads say I am. My goal is to make a method that takes in a file name that is in .bin format with a shift integer. I will make a new file of the .bin type with the characters shifted. Only capital or lower case letters will be shifted though. I don't know the length of the binary file that is being read in and needs to go through all of the characters. The file will only have 1 line though. I have a method that gives me the number of characters on that line and a method that creates a file. The program I know does create the file correctly. Anyways, what is happening is it creates the file, then gives me an EOF exception about the line: char currentChar=data.readChar();
Here is my code:
private static void cipherShifter(String file, int shift) {
String newFile=file+"_cipher";
createFile(newFile);
int numChar;
try {
FileInputStream stream=new FileInputStream(file);
DataInputStream data=new DataInputStream(stream);
FileOutputStream streamOut=new FileOutputStream(newFile);
DataOutputStream dataOut=new DataOutputStream(streamOut);
numChar=readAllInts(data);
for (int i=0;i<numChar;++i) {
char currentChar=data.readChar();
if (((currentChar>='A')&&(currentChar<='Z'))||((currentChar>='a')&&(currentChar<='z'))) {
currentChar=currentChar+=shift;
dataOut.writeChar(currentChar);
}
else {
dataOut.writeChar(currentChar);
}
}
data.close();
dataOut.flush();
dataOut.close();
} catch(IOException error) {
error.printStackTrace();
}
}
private static void createFile(String fileName) {
File file=new File(fileName);
if (file.exists()) {
//Do nothing
}
else {
try {
file.createNewFile();
} catch (IOException e) {
//Do nothing
}
}
}
private static int readAllInts(DataInputStream din) throws IOException {
int count = 0;
while (true) {
try {
din.readInt(); ++count;
} catch (EOFException e) {
return count;
}
}
}
So the error I do not think should be happening because I do have the correct data type and I am telling it to read just a character. Any help would be great. Thanks in advance.

Based on the description above, your error is reported at the data.readChar() method invocation and not inside the readAllInts method. I simulated the code near your error and got the same Exception on a text file at the same location.
I used the readByte method to read one byte at a time since you are mainly interested in ASCII bytes. I also changed readAllInts to be readAllBytes so I work with total byte count.
private static void cipherShifter(String file, int shift) {
String newFile=file+"_cipher";
createFile(newFile);
int numChar;
try {
FileInputStream stream=new FileInputStream(file);
DataInputStream data=new DataInputStream(stream);
FileOutputStream streamOut=new FileOutputStream(newFile);
DataOutputStream dataOut=new DataOutputStream(streamOut);
numBytes=readAllBytes(data);
stream.close();
data.close();
stream=new FileInputStream(file);
data=new DataInputStream(stream);
for (int i=0;i<numBytes;++i) {
byte currentByte=data.readByte();
if (((currentByte>=65)&&(currentByte<=90))||((currentByte>=97)&&(currentByte<=122))) {
currentByte=currentByte+=shift; //need to ensure no overflow beyond a byte
dataOut.writeByte(currentByte);
}
else {
dataOut.writeByte(currentByte);
}
}
data.close();
dataOut.flush();
dataOut.close();
} catch(IOException error) {
error.printStackTrace();
}
}
private static void createFile(String fileName) {
File file=new File(fileName);
if (file.exists()) {
//Do nothing
}
else {
try {
file.createNewFile();
} catch (IOException e) {
//Do nothing
}
}
}
private static int readAllBytes(DataInputStream din) throws IOException {
int count = 0;
while (true) {
try {
din.readByte(); ++count;
} catch (EOFException e) {
return count;
}
}
}

It looks like you're getting the EOFException because you're passing the DataInputStream object to your readAllInts method, reading through the stream, then trying to read from it again inside your for loop. The problem there is that the pointer that keeps track of where you are in the stream is already near the end of the stream (or at the end of it) when readAllInts returns. I suspect it's near the end, rather than at it since the readChar() method is throwing the EOFException immediately, which it does when it only reads one of the two bytes it expects to be able to read before hitting the EOF.
To solve that problem, you could call data.mark() before passing the reader to the readAllInts method, then calling data.reset() after that method returns; that would repoint the pointer to the beginning of the stream. (This assumes data.markSupported() is true.)
You also have the problem we talked about above that your counter is reading in four bytes at a time, and your character reader is reading in two at a time. Your suggested method of doubling the return value of readAllInts would help (you could also use readChar() instead of readInt().)
You still need to think about how you're going to handle the case of binary files that are odd-numbered bytes long. There are a variety of ways you could handle that one. I'm too beat to write up a code sample tonight, but if you're still stuck tomorrow, add a comment and I'll see what I can do to help.

Reading a list of Files as a Java 8 Stream

I have a (possibly long) list of binary files that I want to read lazily. There will be too many files to load into memory. I'm currently reading them as a MappedByteBuffer with FileChannel.map(), but that probably isn't required. I want the method readBinaryFiles(...) to return a Java 8 Stream so I can lazy load the list of files as I access them.
public List<FileDataMetaData> readBinaryFiles(
List<File> files,
int numDataPoints,
int dataPacketSize )
throws
IOException {
List<FileDataMetaData> fmdList = new ArrayList<FileDataMetaData>();
IOException lastException = null;
for (File f: files) {
try {
FileDataMetaData fmd = readRawFile(f, numDataPoints, dataPacketSize);
fmdList.add(fmd);
} catch (IOException e) {
logger.error("", e);
lastException = e;
}
}
if (null != lastException)
throw lastException;
return fmdList;
}
// The List<DataPacket> returned will be in the same order as in the file.
public FileDataMetaData readRawFile(File file, int numDataPoints, int dataPacketSize) throws IOException {
FileDataMetaData fmd;
FileChannel fileChannel = null;
try {
fileChannel = new RandomAccessFile(file, "r").getChannel();
long fileSz = fileChannel.size();
ByteBuffer bbRead = ByteBuffer.allocate((int) fileSz);
MappedByteBuffer buffer = fileChannel.map(FileChannel.MapMode.READ_ONLY, 0, fileSz);
buffer.get(bbRead.array());
List<DataPacket> dataPacketList = new ArrayList<DataPacket>();
while (bbRead.hasRemaining()) {
int channelId = bbRead.getInt();
long timestamp = bbRead.getLong();
int[] data = new int[numDataPoints];
for (int i=0; i<numDataPoints; i++)
data[i] = bbRead.getInt();
DataPacket dp = new DataPacket(channelId, timestamp, data);
dataPacketList.add(dp);
}
fmd = new FileDataMetaData(file.getCanonicalPath(), fileSz, dataPacketList);
} catch (IOException e) {
logger.error("", e);
throw e;
} finally {
if (null != fileChannel) {
try {
fileChannel.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return fmd;
}
Returning fmdList.Stream() from readBinaryFiles(...) won't accomplish this because the file contents will already have been read into memory, which I won't be able to do.
The other approaches to reading the contents of multiple files as a Stream rely on using Files.lines(), but I need to read binary files.
I'm, open to doing this in Scala or golang if those languages have better support for this use case than Java.
I'd appreciate any pointers on how to read the contents of multiple binary files lazily.

There is no laziness possible for the reading within the a file as you are reading the entire file for constructing an instance of FileDataMetaData. You would need a substantial refactoring of that class to be able to construct an instance of FileDataMetaData without having to read the entire file.
However, there are several things to clean up in that code, even specific to Java 7 rather than Java 8, i.e you don’t need a RandomAccessFile detour to open a channel anymore and there is try-with-resources to ensure proper closing. Note further that you usage of memory mapping makes no sense. When copy the entire contents into a heap ByteBuffer after mapping the file, there is nothing lazy about it. It’s exactly the same what happens, when call read with a heap ByteBuffer on a channel, except that the JRE can reuse buffers in the read case.
In order to allow the system to manage the pages, you have to read from the mapped byte buffer. Depending on the system, this might still not be better than repeatedly reading small chunks into a heap byte buffer.
public FileDataMetaData readRawFile(
File file, int numDataPoints, int dataPacketSize) throws IOException {
try(FileChannel fileChannel=FileChannel.open(file.toPath(), StandardOpenOption.READ)) {
long fileSz = fileChannel.size();
MappedByteBuffer bbRead=fileChannel.map(FileChannel.MapMode.READ_ONLY, 0, fileSz);
List<DataPacket> dataPacketList = new ArrayList<>();
while(bbRead.hasRemaining()) {
int channelId = bbRead.getInt();
long timestamp = bbRead.getLong();
int[] data = new int[numDataPoints];
for (int i=0; i<numDataPoints; i++)
data[i] = bbRead.getInt();
dataPacketList.add(new DataPacket(channelId, timestamp, data));
}
return new FileDataMetaData(file.getCanonicalPath(), fileSz, dataPacketList);
} catch (IOException e) {
logger.error("", e);
throw e;
}
}
Building a Stream based on this method is straight-forward, only the checked exception has to be handled:
public Stream<FileDataMetaData> readBinaryFiles(
List<File> files, int numDataPoints, int dataPacketSize) throws IOException {
return files.stream().map(f -> {
try {
return readRawFile(f, numDataPoints, dataPacketSize);
} catch (IOException e) {
logger.error("", e);
throw new UncheckedIOException(e);
}
});
}

This should be sufficient:
return files.stream().map(f -> readRawFile(f, numDataPoints, dataPacketSize));
…if, that is, you are willing to remove throws IOException from the readRawFile method’s signature. You could have that method catch IOException internally and wrap it in an UncheckedIOException. (The problem with deferred execution is that the exceptions also need to be deferred.)

I don't know how performant this is, but you can use java.io.SequenceInputStream wrapped inside of DataInputStream. This will effectively concatenate your files together. If you create a BufferedInputStream from each file, then the whole thing should be properly buffered.

Building on VGR's comment, I think his basic solution of:
return files.stream().map(f -> readRawFile(f, numDataPoints, dataPacketSize))
is correct, in that it will lazily process the files (and stop if a short-circuiting terminal action is invoked off the result of the map() operation. I would also suggest a slightly different to the implementation of readRawFile that leverages try with resources and InputStream, which will not load the whole file into memory:
public FileDataMetaData readRawFile(File file, int numDataPoints, int dataPacketSize)
throws DataPacketReadException { // <- Custom unchecked exception, nested for class
FileDataMetadata results = null;
try (FileInputStream fileInput = new FileInputStream(file)) {
String filePath = file.getCanonicalPath();
long fileSize = fileInput.getChannel().size()
DataInputStream dataInput = new DataInputStream(new BufferedInputStream(fileInput);
results = new FileDataMetadata(
filePath,
fileSize,
dataPacketsFrom(dataInput, numDataPoints, dataPacketSize, filePath);
}
return results;
}
private List<DataPacket> dataPacketsFrom(DataInputStream dataInput, int numDataPoints, int dataPacketSize, String filePath)
throws DataPacketReadException {
List<DataPacket> packets = new
while (dataInput.available() > 0) {
try {
// Logic to assemble DataPacket
}
catch (EOFException e) {
throw new DataPacketReadException("Unexpected EOF on file: " + filePath, e);
}
catch (IOException e) {
throw new DataPacketReadException("Unexpected I/O exception on file: " + filePath, e);
}
}
return packets;
}
This should reduce the amount of code, and make sure that your files get closed on error.

Initializing In Try/Catch

I have run into quite a snag while writing my app. Here is my issue:
I am trying to initialize the file input stream like so:
FileInputStream fis
fis = openFileInput(selectedFile);
Then put this 1 line later:
byte[] input = new byte[fis.available()];
Problem is both bits of code need try/catch statements and the second block cannot recognize fis because it was initialized within a try/catch. Here is my code:
private void openFile(String selectedFile) {
String value = "";
FileInputStream fis;
try {
fis = openFileInput(selectedFile);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
try {
byte[] input = new byte[fis.available()];
} catch (IOException e) {
e.printStackTrace();
}
What should I do? (Thanks in advance)

The best approach in this situation is not to catch IOException at all.
private void openFile(String selectedFile) throws IOException {
FileInputStream fis = openFileInput(selectedFile);
byte[] input = new byte[fis.available()];
It does not make sense to continue after you got FileNotFoundException

Set FileInputStream fis = null; when you first declare the variable.
You could also run your code like this because IOException will also catch the file not found exception.
String value = "";
FileInputStream fis;
try {
fis = openFileInput(selectedFile);
byte[] input = new byte[fis.available()];
} catch (IOException e) {
e.printStackTrace();
}

Set the FileInputStream to a temporary value. null would be the best option, as in:
FileInputStream fis = null;
The reason for this is because if your try statement throws an error, then the fis will never me initialized. Then you'll have problems. If you don't exit the thing entirely, you should also add the statement after the try/catch blocks that tests if the value is null, just so that the program does not throw a null pointer exception.
So maybe something like:
if(fis == null) {
return; // Which will just end the method.
}
Also might want to put the try/catches together (you should still declare the other stuff outside of the try, at least anything you plan on using directly later on in the code) but it just might be more efficient coding wise), as in:
FileInputStream fis = null;
byte[] input = null;
try {
fis = openFileInput(selectedFile);
input = new byte[fis.available()];
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}

Is there a way to catch exceptions without breaking loop in Java?

In Java, there's a difference between a loop surrounded with a try-catch block if an exception could be thrown inside the while loop, and a statement surrounded by a try-catch block inside a loop.
For instance, the following code snippets are different:
Snippet 1:
try {
for (File file : files) {
FileInputStream fis = new FileInputStream(file);
System.out.println("OK!");
}
}
catch (FileNotFoundException exc) {
System.out.println("Error!");
}
^This code snippet breaks the loop if a FileNotFoundException is thrown. So if a file cannot be read, then the loop breaks and Java will stop reading further files.
Snippet 2:
for (File file : files) {
try {
FileInputStream fis = new FileInputStream(file);
System.out.println("OK!");
}
catch (FileNotFoundException exc) {
System.out.println("Error!");
}
}
^This code snippet does not break the loop if an exception is thrown, if an exception occurs, the code catches the exception and continues to the next element in files. With other words, it won't stop reading the files.
Now I want to read a certain file in a directory (say bananas.xml), and, unregarded if that file is readable or not—the XML file is a metadata file, which might not be required for the program to run—, read the corresponding directory (which is bananas):
File main = new File("/home/MCEmperor/test");
File fruitMeta = new File(main, "bananas.xml");
FileInputStream fruitInputStream = new FileInputStream(fruitMeta); // This code COULD throw a FileNotFoundException
// Do something with the fruitInputStream...
File fruitDir = new File(main, "bananas");
if (fruitDir.exists() && fruitDir.canRead()) {
File[] listBananas = fruitDir.listFiles();
for (File file : listBananas) {
FileInputStream fis = new FileInputStream(file); // This code COULD throws a FileNotFoundException
// Do something with the fis...
}
}
Now two lines in the snippet above may throw a FileNotFoundException and I don't want to break the loop.
Now is there a way to make one try-catch block with catches both lines if an exception is thrown, but without breaking the for-loop?

How about something like this?
FileInputStream fruitInputStream = getFileInputStream(fruitMeta);
...
fis = getFileInputStream(file);
private static FileInputStream getFileInputStream(File file) {
try {
return new FileInputStream(file);
catch(FileNotFoundException e) {
return null;
}
}

exception during file copy in Java

I have a function that copies binary file
public static void copyFile(String Src, String Dst) throws FileNotFoundException, IOException {
File f1 = new File(Src);
File f2 = new File(Dst);
FileInputStream in = new FileInputStream(f1);
FileOutputStream out = new FileOutputStream(f2);
byte[] buf = new byte[1024];
int len;
while ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
}
in.close();
out.close();
}
and the second function
private String copyDriverToSafeLocation(String driverPath) {
String safeDir = System.getProperty("user.home");
String safeLocation = safeDir + "\\my_pkcs11tmp.dll";
try {
Utils.copyFile(driverPath, safeLocation);
return safeLocation;
} catch (Exception ex) {
System.out.println("Exception occured while copying driver: " + ex);
return null;
}
}
The second function is run for every driver found in the system.
The driver file is copied and I am trying to initialize PKCS11 with that driver.
If initialization failed I go to next driver, I copy it to the tmp location and so on.
The initialization is in try/catch block
After the first failure I am no longer able to copy next driver to the standard location.
I get the exception
Exception occured while copying driver: java.io.FileNotFoundException: C:\Users\Norbert\my_pkcs11tmp.dll (The process cannot access the file because it is being used by another process)
How can I avoid the exception and safely copy the driver file?
For those curious why am I trying to copy the driver ... PKCS11 has nasty BUG, which prevents using drivers stored in the location that has "(" in the path ... and this is a case I am facing.
I will appreciate your help.

I would move the try-catch block into the copyFile method. That way you can properly handle closing the InputStreams (which is probably causing the JVM to hold onto the file handle). Something like this:
public static void copyFile(String Src, String Dst) {
try {
File f1 = new File(Src);
File f2 = new File(Dst);
FileInputStream in = new FileInputStream(f1);
FileOutputStream out = new FileOutputStream(f2);
byte[] buf = new byte[1024];
int len;
while ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
}
}
catch(Exception e) {
System.out.println("Exception occured while copying driver: " + ex);
}
finally {
in.close();
out.close();
}
}
Then you can remove the try-catch from the copyDriverToSafeLocation method.

Or there's the Java 7 Way:
public static void copyFile(String src, String dst) throws FileNotFoundException, IOException {
try (FileInputStream in = new FileInputStream(new File(src))) {
try (FileOutputStream out = new FileOutputStream(new File(dst))) {
byte[] buf = new byte[1024];
int len;
while ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
}
}
}
}
Edit: And the Java 7 NIO way.
public static void copyFile(String src, String dst) throws FileNotFoundException, IOException {
copyFile(new File(src), new File(dst));
}
public static void copyFile(File src, File dst) throws FileNotFoundException, IOException {
try (FileInputStream in = new FileInputStream(src)) {
try (FileOutputStream out = new FileOutputStream(dst)) {
copyFile(in, out);
}
}
}
public static void copyFile(FileInputStream in, FileOutputStream out) throws IOException {
FileChannel cin = in.getChannel();
FileChannel cout = out.getChannel();
cin.transferTo(0, cin.size(), cout);
}

If the file is used by an other process and locked, there is no generic solutions to be able to access it. You best chance is to use FileLock but it's plateform-dependant, read the documentation, it's written that the results are "advisory", so be carefull. you can also take a look at the ReentrantReadWriteLock class.

I would choose to go with Apache Commons IO and their FileUtils.copyFile() routine(s).
Standard concise way to copy a file in Java?

I'm not sure why a problem with one file would prevent copying a different file. However, not closing a file when an exception occurs could definitely cause problems. Use try...finally to make sure you call close on every file you open.