Develop Reference

How can I check the Payload size when using Azure Eventhubs and avoid a PayloadSizeExceededException?

So I am getting this exception:
com.microsoft.azure.servicebus.PayloadSizeExceededException: Size of the payload exceeded Maximum message size: 256 kb
I believe the exception is self explanatory, however, I an not sure what to do about it.
private int MAXBYTES = (int) ((1024 * 256) * .8);
for (EHubMessage message : payloads) {
byte[] payloadBytes = message.getPayload().getBytes(StandardCharsets.UTF_8);
EventData sendEvent = new EventData(payloadBytes);
events.add(sendEvent);
byteCount += payloadBytes.length;
if (byteCount > this.MAXBYTES) {
calls.add(ehc.sendASync(events));
logs.append("[Size:").append(events.size()).append(" - ").append(byteCount / 1024).append("kb] ");
events = new LinkedList<EventData>();
byteCount = 0;
pushes++;
}
}
I am counting the bytes and such. I have thought through the UTF-8 thing but I believe that should not matter. UTF-8 can be more than one byte, but it should be counted correctly with the "getBytes".
I could not find a reliable way to get the bytes in a string and I am not even sure how Azure counts the bytes. "payload" is a broad statement. Could include the boilerplate stuff and such.
Any Ideas? It would be great if there was a
EventHubClient.checkPayload(list);
method but there doesn't seem to be. How do you guys check the Payload Size?

Per my experience, I think you need to check the size of the current payload count and a new payload first before you add the new payload into events, as below.
const int MAXBYTES = 1024 * 256; // not necessary to multiply by .8
for (EHubMessage message : payloads) {
byte[] payloadBytes = message.getPayload().getBytes(StandardCharsets.UTF_8);
if (byteCount + payloadBytes.length > this.MAXBYTES) {
calls.add(ehc.sendASync(events));
logs.append("[Size:").append(events.size()).append(" - ").append(byteCount / 1024).append("kb] ");
events = new LinkedList<EventData>();
byteCount = 0;
pushes++;
}
EventData sendEvent = new EventData(payloadBytes);
events.add(sendEvent);
}
If you first added the new event data to count the payload size, it's too late and the data size of events which will be sent that might be exceed the payload limits.

Well, I should have added more of the actual code then I did in the original post. Here is what I came up with:
private int MAXBYTES = (int) ((1024 * 256) * .9);
for (EHubMessage message : payloads) {
byte[] payloadBytes = message.getPayload().getBytes(StandardCharsets.UTF_8);
int propsSize = message.getProps() == null ? 0 : message.getProps().toString().getBytes().length;
int messageSize = payloadBytes.length + propsSize;
if (byteCount + messageSize > this.MAXBYTES) {
calls.add(ehc.sendASync(events));
logs.append("[Size:").append(events.size()).append(" - ").append(byteCount / 1024).append("kb] ");
events = new LinkedList<EventData>();
byteCount = 0;
pushes++;
}
byteCount += messageSize;
EventData sendEvent = new EventData(payloadBytes);
sendEvent.getProperties().putAll(message.getProps());
events.add(sendEvent);
}
if (!events.isEmpty()) {
calls.add(ehc.sendASync(events));
logs.append("[Size:").append(events.size()).append(" - ").append(byteCount / 1024).append("kb]");
pushes++;
}
// lets wait til they are done.
CompletableFuture.allOf(calls.toArray(new CompletableFuture[0])).join();
}
If you notice, I was adding Properties to the EventData but not counting the bytes. The toString() for a Map returns something like:
{markiscool=markiscool}
Again, I am not sure of the boilerplate characters that the Azure api is adding but I am sure it is not much. Notice I still back off the MAXBYTES a bit just in case.
It would still be good to get a "payload size checker" method in the api but I would imagine that it would have to build the payload first to give it back to you. I experimented with having my EHubMessage object figure this out for me, but "getBytes()" on a String actually does some conversion that I don't want to do twice.

SeekableByteChannel.read() always returns 0, InputStream is fine

We have a data file for which we need to generate a CRC. (As a placeholder, I'm using CRC32 while the others figure out what CRC polynomial they actually want.) This code seems like it ought to work:
broken:
Path in = ......;
try (SeekableByteChannel reading =
Files.newByteChannel (in, StandardOpenOption.READ))
{
System.err.println("byte channel is a " + reading.getClass().getName() +
" from " + in + " of size " + reading.size() + " and isopen=" + reading.isOpen());
java.util.zip.CRC32 placeholder = new java.util.zip.CRC32();
ByteBuffer buffer = ByteBuffer.allocate (reasonable_buffer_size);
int bytesread = 0;
int loops = 0;
while ((bytesread = reading.read(buffer)) > 0) {
byte[] raw = buffer.array();
System.err.println("Claims to have read " + bytesread + " bytes, have buffer of size " + raw.length + ", updating CRC");
placeholder.update(raw);
loops++;
buffer.clear();
}
// do stuff with placeholder.getValue()
}
catch (all the things that go wrong with opening files) {
and handle them;
}
The System.err and loops stuff is just for debugging; we don't actually care how many times it takes. The output is:
byte channel is a sun.nio.ch.FileChannelImpl from C:\working\tmp\ls2kst83543216xuxxy8136.tmp of size 7196 and isopen=true
finished after 0 time(s) through the loop
There's no way to run the real code inside a debugger to step through it, but from looking at the source to sun.nio.ch.FileChannelImpl.read() it looks like a 0 is returned if the file magically becomes closed while internal data structures are prepared; the code below is copied from the Java 7 reference implementation, comments added by me:
// sun.nio.ch.FileChannelImpl.java
public int read(ByteBuffer dst) throws IOException {
ensureOpen(); // this throws if file is closed...
if (!readable)
throw new NonReadableChannelException();
synchronized (positionLock) {
int n = 0;
int ti = -1;
Object traceContext = IoTrace.fileReadBegin(path);
try {
begin();
ti = threads.add();
if (!isOpen())
return 0; // ...argh
do {
n = IOUtil.read(fd, dst, -1, nd);
} while (......)
.......
But the debugging code tests isOpen() and gets true. So I don't know what's going wrong.
As the current test data files are tiny, I dropped this in place just to have something working:
works for now:
try {
byte[] scratch = Files.readAllBytes(in);
java.util.zip.CRC32 placeholder = new java.util.zip.CRC32();
placeholder.update(scratch);
// do stuff with placeholder.getValue()
}
I don't want to slurp the entire file into memory for the Real Code, because some of those files can be large. I do note that readAllBytes uses an InputStream in its reference implementation, which has no trouble reading the same file that SeekableByteChannel failed to. So I'll probably rewrite the code to just use input streams instead of byte channels. I'd still like to figure out what's gone wrong in case a future scenario comes up where we need to use byte channels. What am I missing with SeekableByteChannel?

Check that 'reasonable_buffer_size' isn't zero.

Bluetooth Serial Data Transfer Hangs at same points

I am to transfer two files from a Java app to an Arduino of roughly 200k each to be saved to uSD. With a USB connection this is going fine using the RXTX library and simple data parsing on the Arduino. I now want to replace the USB connect with a bluetooth using a HC-06 module.
Without changing anything the transfer works but it takes 12 minutes rather than 2 minutes. The transfer hangs at the same 4 (maybe 50) points of the transfer, seen by percentage progress.
The bottleneck must be in the RXTX library or bluteooth TX but can't find what's happening. Is there anything I can tweak or is this a limitation of a basic bluetooth module?
Here are snippets from code FWIW, nothing out of the ordinary (I think!) :
Java :
int perc = data.length / 100;
for (int i = 0 ; i < data.length ; i++) {
try{
output.write(data[i]);
}
catch(Exception e){
System.err.println(e.toString());
}
if(i % perc == 0) System.out.println(i / perc);
}
Arduino :
boolean waiting = true;
while( lastData + 2000 > millis() || waiting){
while(Serial.available()) {
waiting = false;
if(Serial.available() >= 255) nodes[nodeCount++] = byteCount;
buff[bytInc++] = Serial.read();
if(bytInc == 512) {
bytInc = 0;
in.write(buff,512);
}
lastData = millis();
byteCount++;
}
}
in.write(buff,bytInc);
in.flush();
Any optimisation in the snippets above would be much appreciated. The baud rate is 38400. Any faster and the buffer overloads when writing to uSD.
Arduino board is a Mega with 256 byte serial buffer.
Much appreciated. Have hunted about for relevant posts but can't find anything on bluetooth file transfers like this.

Memory issue when storing images in byteArray

I have an app that needs to access a large number of images very quickly, so I need to load those images into memory in some way. Doing so as bitmaps used over 100MB of RAM, which was completely out of the question, so I opted to read jpg files into memory, storing them inside a byteArray. Then I decode them and write them to the canvas as each is needed. This works pretty well, cutting out the slow disk access, while also respecting memory limits.
However, memory usage seems 'off' to me. I'm storing 450 jpgs with a file size of approximately 33kb each. This totals around 15MB of data. However, the app continually runs at between 35MB and 40MB of RAM as reported by both Eclipse DDMS and Android (on a physical device). I've tried modifying how many jpgs are loaded and the RAM used by the app tends to decrease by around 60-70kb per jpg, indicating that each image is stored twice in RAM. Memory usage does not fluctuate which implies that there is not an actual 'leak' involved.
Here is the relevant loading code:
private byte[][] bitmapArray = new byte[totalFrames][];
for (int x=0; x<totalFrames; x++) {
File file = null;
if (cWidth <= cHeight){
file = new File(directory + "/f"+x+".jpg");
} else {
file = new File(directory + "/f"+x+"-land.jpg");
}
bitmapArray[x] = getBytesFromFile(file);
imagesLoaded = x + 1;
}
public byte[] getBytesFromFile(File file) {
byte[] bytes = null;
try {
InputStream is = new FileInputStream(file);
long length = file.length();
bytes = new byte[(int) length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length && (numRead = is.read(bytes, offset, bytes.length - offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException("Could not completely read file " + file.getName());
}
is.close();
} catch (IOException e) {
//TODO Write your catch method here
}
return bytes;
}
Eventually, they get written to screen like so:
SurfaceHolder holder = getSurfaceHolder();
Canvas c = null;
try {
c = holder.lockCanvas();
if (c != null) {
int canvasWidth = c.getWidth();
int canvasHeight = c.getHeight();
Rect destinationRect = new Rect();
destinationRect.set(0, 0, canvasWidth, canvasHeight);
c.drawBitmap(BitmapFactory.decodeByteArray(bitmapArray[bgcycle], 0, bitmapArray[bgcycle].length), null, destinationRect, null);
}
} finally {
if (c != null)
holder.unlockCanvasAndPost(c);
}
Am I correct that there is some sort of duplication going on here? Or is there just that much overhead involved in storing jpgs in a byteArray like this?

Storing bytes in RAM is very different to storing data on hard drives... There is alot more overhead to it. The references to the objects as well the byte array structures all take up additional memory. There isn't really a single source to all the additional memory but just remember than loading a file into RAM normally takes up 2 ~ 3x more space (from experience, I'm afraid I can't quote any documentation here).
Consider this:
File F = //Some file here (Less than 2 GB please)
FileInputStream fIn = new FileInputStream(F);
ByteArrayOutputStream bOut = new ByteArrayOutputStream(((int)F.length()) + 1);
int r;
byte[] buf = new byte[32 * 1000];
while((r = fIn.read(buf) != -1){
bOut.write(buf, 0, r);
}
//Do a memory measurement at this point. You'll see your using nearly 3x the memory in RAM compared to the file.
//If your actually gonna try this, remember to surround with try-catch and close the streams as appropriate.
Also remember that unused memory is not instantly cleared up. The method getBytesFromFile() may be returning a copy of a byte array which causes memory duplication which may not immediately be garbage collected. If you want to be safe, check the method getBytesFromFile(file) is not leaking any references that should be cleaned up. It won't appear as a memory leak as you only call it a finite number of times.

It might be because your byte array is 2 dimensional, you only need one dimension for loading an image using a byte array, and the second dimension could potentially double the Ram needed as for each byte you would have an empty but still existing byte that you don't use

How is reading an InputStream object from a local file different than from the network (via Amazon S3)?

I didn't think there was a difference between an inputstream object read from a local file vs one from a network source (Amazon S3 in this case) so hopefully someone can enlighten me.
These programs were run on a VM running Centos 6.3.
The test file in both cases are 10MB.
Local file code:
InputStream is = new FileInputStream("/home/anyuser/test.jpg");
int read = 0;
int buf_size = 1024 * 1024 * 2;
byte[] buf = new byte[buf_size];
ByteArrayOutputStream baos = new ByteArrayOutputStream(buf_size);
long t3 = System.currentTimeMillis();
int i = 0;
while ((read = is.read(buf)) != -1) {
baos.write(buf,0,read);
System.out.println("reading for the " + i + "th time");
i++;
}
long t4 = System.currentTimeMillis();
System.out.println("Time to read = " + (t4-t3) + "ms");
The output of this code is this: it reads 5 times, which makes sense since the buffer size read in is 2MB and the file is 10MB.
reading for the 0th time
reading for the 1th time
reading for the 2th time
reading for the 3th time
reading for the 4th time
Time to read = 103ms
Now, we have the same code run with the same 10MB test file, except this time, the source is from Amazon S3. We don't start reading until we finish getting the stream from S3. However, this time, the read loop is running through thousands of times, when it should only read it 5 times.
InputStream is;
long t1 = System.currentTimeMillis();
is = getS3().getFileFromBucket(S3Path,input);
long t2 = System.currentTimeMillis();
System.out.print("Time to get file " + input + " from S3: ");
System.out.println((t2-t1) + "ms");
int read = 0;
int buf_size = 1024*1024*2;
byte[] buf = new byte[buf_size];
ByteArrayOutputStream baos = new ByteArrayOutputStream(buf_size);
long t3 = System.currentTimeMillis();
int i = 0;
while ((read = is.read(buf)) != -1) {
baos.write(buf,0,read);
if ((i % 100) == 0)
System.out.println("reading for the " + i + "th time");
i++;
}
long t4 = System.currentTimeMillis();
System.out.println("Time to read = " + (t4-t3) + "ms");
The output is as follows:
Time to get file test.jpg from S3: 2456ms
reading for the 0th time
reading for the 100th time
reading for the 200th time
reading for the 300th time
reading for the 400th time
reading for the 500th time
reading for the 600th time
reading for the 700th time
reading for the 800th time
reading for the 900th time
reading for the 1000th time
reading for the 1100th time
reading for the 1200th time
reading for the 1300th time
reading for the 1400th time
Time to read = 14471ms
The amount of time taken to read the stream changes from run to run. Sometimes it takes 60 seconds, sometimes 15 seconds. It doesn't get faster than 15 sec. The read loop still loops through 1400+ times on each test run of the program, even though I think it should only be 5 times, like the local file example.
Is this how inputstream works when the source is through the network, even though we had finished getting the file from the network source? Thanks in advance for your help.

I don't think it's specific to java. When you read from the network, the actual read call to the operating system will return a packet of data at a time, no matter how big is the buffer you allocated. If you check the size of the read data (your read variable), it should show the size of the network packet used.
This is one of the reason why people use a separate thread to read from network and avoid blocking by using async i/o technique.

As #imel96 points out, there is nothing in the documentation that guarantees the behaviour you are expecting. You will never read 2MB at a time from a socket, because the socket receive buffer isn't normally that large, quite apart from other factors such as bandwidth.