Develop Reference

Parsing files over 2.15 GB in Java using Kaitai Struct

I'm parsing large PCAP files in Java using Kaitai-Struct. Whenever the file size exceeds Integer.MAX_VALUE bytes I face an IllegalArgumentException caused by the size limit of the underlying ByteBuffer.
I haven't found references to this issue elsewhere, which leads me to believe that this is not a library limitation but a mistake in the way I'm using it.
Since the problem is caused by trying to map the whole file into the ByteBuffer I'd think that the solution would be mapping only the first region of the file, and as the data is being consumed map again skipping the data already parsed.
As this is done within the Kaitai Struct Runtime library it would mean to write my own class extending fom KatiaiStream and overwrite the auto-generated fromFile(...) method, and this doesn't really seem the right approach.
The auto-generated method to parse from file for the PCAP class is.
public static Pcap fromFile(String fileName) throws IOException {
return new Pcap(new ByteBufferKaitaiStream(fileName));
}
And the ByteBufferKaitaiStream provided by the Kaitai Struct Runtime library is backed by a ByteBuffer.
private final FileChannel fc;
private final ByteBuffer bb;
public ByteBufferKaitaiStream(String fileName) throws IOException {
fc = FileChannel.open(Paths.get(fileName), StandardOpenOption.READ);
bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
}
Which in turn is limitted by the ByteBuffer max size.
Am I missing some obvious workaround? Is it really a limitation of the implementation of Katiati Struct in Java?

There are two separate issues here:
Running Pcap.fromFile() for large files is generally not a very efficient method, as you'll eventually get all files parsed into memory array at once. A example on how to avoid that is given in kaitai_struct/issues/255. The basic idea is that you'd want to have control over how you read every packet, and then dispose of every packet after you've parsed / accounted it somehow.
2GB limit on Java's mmaped files. To mitigate that, you can use alternative RandomAccessFile-based KaitaiStream implementation: RandomAccessFileKaitaiStream — it might be slower, but it should avoid that 2GB problem.

This library provides a ByteBuffer implementation which uses long offset. I haven't tried this approach but looks promising. See section Mapping Files Bigger than 2 GB
http://www.kdgregory.com/index.php?page=java.byteBuffer
public int getInt(long index)
{
return buffer(index).getInt();
}
private ByteBuffer buffer(long index)
{
ByteBuffer buf = _buffers[(int)(index / _segmentSize)];
buf.position((int)(index % _segmentSize));
return buf;
}
public MappedFileBuffer(File file, int segmentSize, boolean readWrite)
throws IOException
{
if (segmentSize > MAX_SEGMENT_SIZE)
throw new IllegalArgumentException(
"segment size too large (max " + MAX_SEGMENT_SIZE + "): " + segmentSize);
_segmentSize = segmentSize;
_fileSize = file.length();
RandomAccessFile mappedFile = null;
try
{
String mode = readWrite ? "rw" : "r";
MapMode mapMode = readWrite ? MapMode.READ_WRITE : MapMode.READ_ONLY;
mappedFile = new RandomAccessFile(file, mode);
FileChannel channel = mappedFile.getChannel();
_buffers = new MappedByteBuffer[(int)(_fileSize / segmentSize) + 1];
int bufIdx = 0;
for (long offset = 0 ; offset < _fileSize ; offset += segmentSize)
{
long remainingFileSize = _fileSize - offset;
long thisSegmentSize = Math.min(2L * segmentSize, remainingFileSize);
_buffers[bufIdx++] = channel.map(mapMode, offset, thisSegmentSize);
}
}
finally
{
// close quietly
if (mappedFile != null)
{
try
{
mappedFile.close();
}
catch (IOException ignored) { /* */ }
}
}
}

Cannot find a image file that exists in java

I have written a function which takes in a BufferedImage and compares it to a pre-existing image in my hard drive checking if they are same or not.
public boolean checkIfSimilarImages(BufferedImage imgA, File B) {
DataBuffer imgAdata = imgA.getData().getDataBuffer();
int sizeA = imgAdata.getSize();
BufferedImage imgB = null;
try {
imgB = ImageIO.read(B);
} catch (IOException ex) {
Logger.getLogger(SupportClass.class.getName()).log(Level.SEVERE, null, ex);
}
DataBuffer imgBdata = imgB.getData().getDataBuffer();
int sizeB = imgBdata.getSize();
if(sizeA == sizeB) {
for(int i = 0; i < sizeA; i++) {
if (imgAdata.getElem(i) != imgBdata.getElem(i)) {
return false;
}
}
}
return true;
}
This throws IOException "Cant read input file". Idk why this is happening. I am calling the function like this...
while(support.checkIfSimilarImages(currentDisplay, new File(pathToOriginalImage)) == false) {
System.out.println("Executing while-loop!");
bot.delay(3000);
currentDisplay = bot.createScreenCapture(captureArea);
}
where,
String pathToOriginalImage = "‪‪‪‪C:\\Users\\Chandrachur\\Desktop\\Home.jpg";
I can see that the path is valid. But as I am testing it for File.exists() or File.canRead() or File.absoluteFile().exists() inside the checkIfSimilarImages function and everything is returning false.
I have researched my question here and tried out these suggestions:
It is not only for this location, I have tried a variety of other locations but in vain. Also it is not a problem where I have hidden file extensions and the actual file might be Home.jpg.jpg .
The only thing that might be is that permissions might be different. I dont really know how to verify this, but there is no reason it should have some permission which is not readable by java. It is just another normal jpg file.
Can it be because I am passing the file object reference into a function so in this process somehow the reference is getting modified or something. I just dont know. I am running out of possibilities to test for...
The whole stack trace is as follows:
javax.imageio.IIOException: Can't read input file!
at javax.imageio.ImageIO.read(ImageIO.java:1301)
at battlesbot.SupportClass.checkIfSimilarImages(SupportClass.java:77)
at battlesbot.AutomatedActions.reachHomeScreen(AutomatedActions.java:72)
at battlesbot.BattlesBot.main(BattlesBot.java:22)
Exception in thread "main" java.lang.NullPointerException
at battlesbot.SupportClass.checkIfSimilarImages(SupportClass.java:81)
at battlesbot.AutomatedActions.reachHomeScreen(AutomatedActions.java:72)
at battlesbot.BattlesBot.main(BattlesBot.java:22)
C:\Users\Chandrachur\AppData\Local\NetBeans\Cache\8.2\executor-snippets\run.xml:53: Java returned: 1
BUILD FAILED (total time: 11 seconds)
I am on Windows 10, IDE is NetBeans.
UPDATE:
Huge thanks to #k5_ . He told me to paste this in path and it worked.
"C:/Users/Chandrachur/Desktop/Home.jpg";
It seems some invisible characters were in the path. But I still don't understand what that means.

Usually this kind of problem lies with access problem or typos in the filename.
In this case there were some invisible unicode characters x202A in the filename. The windows dialog box, the file path was copied from, uses them for direction of writing (left to right).
One way of displaying them would be this loop, it has 4 invisible characters at the start of the String. You would also see them in a debugger.
String x = "‪‪‪‪C:\\Users\\Chandrachur\\Desktop\\Home.jpg";
for(char c : x.toCharArray()) {
System.out.println( c + " " + (int) c);
}

Using multiple ports to send images via two separate pair of Arducam and arduino Uno

I am trying to send images via two ports namely COM5 and COM7.
Following code does the most. Most significant part of the code is captureAndsaveImage method.
The problem is when i use both serial ports; images are getting distorted they feel like they are getting mixed up.
My question: Is it possible to use both port simaltaneously? What should i do such that there is no mixing up?
Don't mind second image's black circle it might have happened due to some signal losses in the second camera.
public class ReadPort {
private static final char[]COMMAND = {'*', 'R', 'D', 'Y', '*'};
private static final int WIDTH = 320; //640;
private static final int HEIGHT = 240; //480;
SerialPort serialPort,serialPort2;
public int[][] rgb2 = new int[WIDTH][HEIGHT];
public static void main(String[] args) {
ReadPort reader= new ReadPort();
}
public ReadPort() {
int[][]rgb = new int[HEIGHT][WIDTH];
try {
serialPort = SerialPort.getCommPort("COM7");
serialPort.openPort();
inputStream = serialPort.getInputStream();
serialPort.setComPortParameters(1000000,
8,
SerialPort.ONE_STOP_BIT,
SerialPort.NO_PARITY);
if(serialPort.isOpen()){
System.out.println("COM5 opened");
}
serialPort2 = SerialPort.getCommPort("COM5");
serialPort2.openPort();
inputStream2 = serialPort2.getInputStream();
serialPort2.setComPortParameters(1000000,
8,
SerialPort.ONE_STOP_BIT,
SerialPort.NO_PARITY);
if(serialPort2.isOpen()){
System.out.println("COM7 opened");
}
int counter = 0;
while(true) {
captureAndsaveImage( inputStream2,counter, rgb, "COM5");
captureAndsaveImage(inputStream, counter, rgb, "COM7");
counter++;
}
} catch (Exception e) {
e.printStackTrace();
}
}
public static void captureAndsaveImage(InputStream inputStream, int counter,int[][] rgb,String name) throws IOException{
while(!isImageStart(inputStream, 0)){};
System.out.print("Found image: " + counter);
for (int y = 0; y < HEIGHT; y++) {
for (int x = 0; x < WIDTH; x++) {
int temp =read(inputStream);
rgb[y][x] = ((temp&0xFF) << 16) | ((temp&0xFF) << 8) | (temp&0xFF);
}
}
BMP bmp = new BMP();
bmp.saveBMP("c:/out/" +name+"images/"+ counter + ".bmp", rgb);
System.out.println(", Saved image:"+name+"images/"+counter+".bmp");
}
private static int read(InputStream inputStream) throws IOException {
int temp = (char) inputStream.read();
//System.out.print(temp);
if (temp == -1) {
throw new IllegalStateException("Exit");
}
return temp;
}
private static boolean isImageStart(InputStream inputStream, int index) throws IOException {
if (index < COMMAND.length) {
if (COMMAND[index] == read(inputStream)) {
return isImageStart(inputStream, ++index);
} else {
return false;
}
}
return true;
}
}
Edit: I used a debug statement like
if(inputStream.available()>0){
System.out.println(inputStream.toString());}
in the captureAndsaveImage method and i got output like
COM5 opened
COM7 opened
Found image:
0com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#7f31245a
, Saved image:COM5images/0.bmp
Found image:
0com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28
, Saved image:COM7images/0.bmp
Found image:
1com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#7f31245a
, Saved image:COM5images/1.bmp
Found image:
1com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28
, Saved image:COM7images/1.bmp
Found image: 2, Saved image:COM5images/2.bmp
Found image:
2com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28
, Saved image:COM7images/2.bmp
Found image:
3com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#7f31245a
, Saved image:COM5images/3.bmp
Found image:
3com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28
, Saved image:COM7images/3.bmp
Found image: 4, Saved image:COM5images/4.bmp
Found image:
4com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28
, Saved image:COM7images/4.bmp
Found image:
5com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#7f31245a
, Saved image:COM5images/5.bmp
Found image:
5com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28
, Saved image:COM7images/5.bmp
Found image: 6, Saved image:COM5images/6.bmp
Found image: 6, Saved image:COM7images/6.bmp
Found image:
7com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#7f31245a
, Saved image:COM5images/7.bmp
Found image:
7com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28
, Saved image:COM7images/7.bmp
Found image: 8, Saved image:COM5images/8.bmp
Found image:
8com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28
, Saved image:COM7images/8.bmp
Found image:
9com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#7f31245a
, Saved image:COM5images/9.bmp
Things i observe is that some lines are like
Found image: 6, Saved image:COM5images/6.bmp
and most of them are
Found image:
5com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28
, Saved image:COM7images/5.bmp
What is the reason? As far as i know com.fazecast.jSerialComm.SerialPort$SerialPortInputStream#6d6f6e28 this is supposed to be address of the inputStream. But why it is not happening in some cases?
(I am beginner in Serial communication.)

Hurray! I have solved my problem. Even though solution isn't elegant.
I put a block of code in the start of captureAndsaveImage method like
while(inputStream.available()>0){
int temp=read(inputStream);
}
And now I am getting cleared image. I have some vague idea how it worked
but i would love if anyone can give logic for these.
Edit: I observe that distorted images were coming in odd frames. So the above code just skips those frames shows even frames which are not mixed up. :/

Your code seems to be very disorganized. For example, where you open COM5, your debug messages says it's opening COM7 and vice versa.
However, the bug causing the issue you raised in your question is with these lines of code:
while(true) {
captureAndsaveImage( inputStream2,counter, rgb, "COM5");
captureAndsaveImage(inputStream, counter, rgb, "COM7");
counter++;
}
As you can see, you're storing the data from both image sources into the same array, rgb. Your code has an rgb2, so I suspect you meant to use one of those with COM5 and the other for COM7, though the array declarations being at different scopes is strange. I would suggest you review your code, and perhaps focus on getting things working with one serial port/data source before introducing a second.
Edit: reading your comment and reviewing your error, I found another bug:
private static boolean isImageStart(InputStream inputStream, int index) throws IOException {
if (index < COMMAND.length) {
if (COMMAND[index] == read(inputStream)) {
return isImageStart(inputStream, ++index);
}
else {
return false;
}
}
return true;
}
Here, isImageStart() could return true if it works out that you missed the starting character in a stream. Essentially, since you recursively call isImageStart, if you start with a stream that doesn't contain the command character, you will run until you reach COMMAND.length, at which point, the next recursive call would skip the if (index < COMMAND.length) and return true. So, if you have a case where you started reading too soon (or too late), isImageStart() will still return true. Then, in CaptureAndSaveImage(), you still continue calling read on the inputstream and likely are reading stale data from the previous stream. Except by that point, the stream may be valid and depending on how fast data is coming in, you will have a mix of the previous image and the one currently being received.

reading UTF-16 produces unexpected results

I use the beaglebuddy Java library in an Android project for reading/writing ID3 tags of mp3 files. I'm having an issue with reading the text that was previously written using the same library and could not find anything related in their docs.
Assume I write the following info:
MP3 mp3 = new MP3(pathToFile);
mp3.setLeadPerformer("Jon Skeet");
mp3.setTitle("A Million Rep");
mp3.save();
Looking at the source code of the library, I see that UTF-16 encoding is explicitly set, internally it calls
protected ID3v23Frame setV23Text(String text, FrameType frameType) {
return this.setV23Text(Encoding.UTF_16, text, frameType);
}
and
protected ID3v23Frame setV23Text(Encoding encoding, String text, FrameType frameType) {
ID3v23FrameBodyTextInformation frameBody = null;
ID3v23Frame frame = this.getV23Frame(frameType);
if(frame == null) {
frame = this.addV23Frame(frameType);
}
frameBody = (ID3v23FrameBodyTextInformation)frame.getBody();
frameBody.setEncoding(encoding);
frameBody.setText(encoding == Encoding.UTF_16?Utility.getUTF16String(text):text);
return frame;
}
At a later point, I read the data and it gives me some weird Chinese characters:
mp3.getLeadPerformer(); // 䄀 䴀椀氀氀椀漀渀 刀攀瀀
mp3.getTitle(); // 䨀漀渀 匀欀攀攀琀
I took a look at the built-in Utility.getUTF16String(String) method:
public static String getUTF16String(String string) {
String text = string;
byte[] bytes = string.getBytes(Encoding.UTF_16.getCharacterSet());
if(bytes.length < 2 || bytes[0] != -2 || bytes[1] != -1) {
byte[] bytez = new byte[bytes.length + 2];
bytes[0] = -2;
bytes[1] = -1;
System.arraycopy(bytes, 0, bytez, 2, bytes.length);
text = new String(bytez, Encoding.UTF_16.getCharacterSet());
}
return text;
}
I'm not quite getting the point of setting the first 2 bytes to -2 and -1 respectively, is this a pattern stating that the string is UTF-16 encoded?
However, I tried to explicitly call this method when reading the data, that seems to be readable, but always prepends some cryptic characters at the start:
Utility.getUTF16String(mp3.getLeadPerformer()); // ��Jon Skeet
Utility.getUTF16String(mp3.getTitle()); // ��A Million Rep
Since the count of those characters seems to be constant, I created a temporary workaround by simply cutting them off.
Fields like "comments" where the author does not explicitly enforce UTF-16 when writing are read without any issues.
I'm really curious about what's going on here and appreciate any suggestions.

StringIndexOutOfBoundsException Java Error

public void onStart() throws Exception
{
// start up code here
}enter code here
public void onExecute() throws Exception
{
// execute code (set executeOnChange flag on inputs)
String tmp = getInASCIIString().getValue();
setOutCSVString(new BStatusString(AsciiToBinary(tmp)));
}
public void onStop() throws Exception
{
// shutdown code here
}
public static String AsciiToBinary(String asciiString) throws Exception
{
String padding = "00000000";
StringBuffer dataString = new StringBuffer();
StringBuffer outCSV = new StringBuffer();
StringTokenizer Values = new StringTokenizer(asciiString,",");
while (Values.hasMoreTokens())
{
String bin = padding + Integer.toBinaryString(Integer.parseInt(Values.nextToken()));
String reversedString = new StringBuffer(bin.substring(bin.length() - 8, bin.length())).reverse().toString();
dataString.append(reversedString);
}
try
{
char[] charArray = dataString.toString().toCharArray();
for (int i = 1; i < charArray.length; i++)
{
if (charArray[i] == '1')
{
outCSV.append(i+"");
outCSV.append(',');
}
}
if (outCSV.toString().length() > 1)
{
return outCSV.toString().substring(0, outCSV.toString().length()-1);
}
else
{
return outCSV.toString();
}
}
catch(StringIndexOutOfBoundsException e)
{
return "";
}
}
We use a Tridium- which uses Java as the backend. This program seems to be randomly and occasionally throwing an error. I'm only limited to the packages that are pre-installed, including: java.util, java.baja.sys, javax.baja.status, javax.baja.util, com.tridium.program
Which is why some of the code is written using the logic/functions that it does. Anyway- I cannot figure out why this is throwing an error. Any thoughts?
java.lang.StringIndexOutOfBoundsException: String index out of range: 15 at java.lang.String.charAt(String.java:658)
Full stack trace:
java.lang.StringIndexOutOfBoundsException: String index out of range: 15
at java.lang.String.charAt(String.java:658)
at com.korsengineering.niagara.conversion.BStatusNumericToStatusBoolean.changed(BStatusNumericToStatusBoolean.java:38)
at com.tridium.sys.schema.ComponentSlotMap.fireComponentEvent(ComponentSlotMap.java:1000)
at com.tridium.sys.schema.ComponentSlotMap.modified(ComponentSlotMap.java:902)
at com.tridium.sys.schema.ComplexSlotMap.modified(ComplexSlotMap.java:1538)
at com.tridium.sys.schema.ComplexSlotMap.setDouble(ComplexSlotMap.java:1254)
at javax.baja.sys.BComplex.setDouble(BComplex.java:666)
at com.tridium.sys.schema.ComplexSlotMap.copyFrom(ComplexSlotMap.java:294)
at javax.baja.sys.BComplex.copyFrom(BComplex.java:246)
at javax.baja.sys.BLink.propagatePropertyToProperty(BLink.java:593)
at javax.baja.sys.BLink.propagate(BLink.java:523)
at com.tridium.sys.engine.SlotKnobs.propagate(SlotKnobs.java:56)
at com.tridium.sys.schema.ComponentSlotMap.modified(ComponentSlotMap.java:899)
at com.tridium.sys.schema.ComplexSlotMap.modified(ComplexSlotMap.java:1538)
at com.tridium.sys.schema.ComplexSlotMap.setDouble(ComplexSlotMap.java:1254)
at javax.baja.sys.BComplex.setDouble(BComplex.java:666)
at javax.baja.status.BStatusNumeric.setValue(BStatusNumeric.java:66)
at com.tridium.kitControl.conversion.BStatusStringToStatusNumeric.calculate(BStatusStringToStatusNumeric.java:161)
at com.tridium.kitControl.conversion.BStatusStringToStatusNumeric.changed(BStatusStringToStatusNumeric.java:155)
at com.tridium.sys.schema.ComponentSlotMap.fireComponentEvent(ComponentSlotMap.java:1000)
at com.tridium.sys.schema.ComponentSlotMap.modified(ComponentSlotMap.java:902)
at com.tridium.sys.schema.ComplexSlotMap.modified(ComplexSlotMap.java:1538)
at com.tridium.sys.schema.ComplexSlotMap.setString(ComplexSlotMap.java:1335)
at javax.baja.sys.BComplex.setString(BComplex.java:668)
at com.tridium.sys.schema.ComplexSlotMap.copyFrom(ComplexSlotMap.java:295)
at javax.baja.sys.BComplex.copyFrom(BComplex.java:246)
at javax.baja.sys.BLink.propagatePropertyToProperty(BLink.java:593)
at javax.baja.sys.BLink.propagate(BLink.java:523)
at com.tridium.sys.engine.SlotKnobs.propagate(SlotKnobs.java:56)
at com.tridium.sys.schema.ComponentSlotMap.modified(ComponentSlotMap.java:899)
at com.tridium.sys.schema.ComplexSlotMap.modified(ComplexSlotMap.java:1538)
at com.tridium.sys.schema.ComplexSlotMap.setString(ComplexSlotMap.java:1335)
at javax.baja.sys.BComplex.setString(BComplex.java:668)
at com.tridium.sys.schema.ComplexSlotMap.copyFrom(ComplexSlotMap.java:295)
at javax.baja.sys.BComplex.copyFrom(BComplex.java:238)
at javax.baja.control.BControlPoint.doExecute(BControlPoint.java:271)
at auto.javax_baja_control_BStringWritable.invoke(AutoGenerated)
at com.tridium.sys.schema.ComponentSlotMap.invoke(ComponentSlotMap.java:1599)
at com.tridium.sys.engine.EngineUtil.doInvoke(EngineUtil.java:49)
at com.tridium.sys.engine.EngineManager.checkAsyncActions(EngineManager.java:364)
at com.tridium.sys.engine.EngineManager.execute(EngineManager.java:209)
at com.tridium.sys.engine.EngineManager$EngineThread.run(EngineManager.java:691)

Something is happening outside your Program Object, once you wire your outCSVString to whatever other wire sheet logic you are linking it to. This is more than likely due to your AsciiToBinary method returning a null string output that the rest of your logic can't deal with.
Wire outCSVString to a StringWritable object that has a StringCov history extension on it, and look for what value the history records at the same timestamp where you see the exception, to make sure your ProgramObject is generating the output you expect.
Regarding your AsciiToBinary method, the Tridium framework is limited in the modules it provides and what you can import, however it does come packaged with Apache Oro Regular Expression Tools Version 2.0.8. Search for "oro" in the Niagara Help file for more information.
In my experience, you will be more assured that outCSVString will always follow a desired format if you use a regular expression with a substitution to build the string, rather than tokenizing and iterating through the string yourself.