Develop Reference

how to refer part of an array?

Given an object byte[], when we want to operate with such object often we need pieces of it. In my particular example i get byte[] from wire where first 4 bytes describe lenght of the message then another 4 bytes the type of the message (an integer that maps to concrete protobuf class) then remaining byte[] is actual content of the message... like this
length|type|content
in order to parse this message i have to pass content part to specific class which knows how to parse an instance from it... the problem is that often there are no methods provided so that you could specify from where to where parser shall read the array...
So what we end up doing is copying remaining chuks of that array, which is not effective...
As far as i know in java it is not possible to create another byte[] reference that actually refers to some original bigger byte[] array with just 2 indexes (this was approach with String that led to memory leaks)...
I wonder how do we solve situations like this? I suppose giving up on protobuf just because it does not provide some parseFrom(byte[], int, int) does not make sence... protobuf is just an example, anything could lack that api...
So does this force us to write inefficient code or there is something that can be done? (appart from adding that method)...

Normally you would tackle this kind of thing with streams.
A stream is an abstraction for reading just what you need to process the current block of data. So you can read the correct number of bytes into a byte array and pass it to your parse function.
You ask 'So does this force us to write inefficient code or there is something that can be done?'
Usually you get your data in the form of a stream and then using the technique demonstrated below will be more performant because you skip making one copy. (Two copies instead of three; once by the OS and once by you. You skip making a copy of the total byte array before you start parsing.) If you actually start out with a byte[] but it is constructed by yourself then you may want to change to constructing an object such as { int length, int type, byte[] contentBytes } instead and pass contentBytes to your parse function.
If you really, really have to start out with byte[] then the below technique is just a more convenient way to parse it, it would not be more performant.
So suppose you got a buffer of bytes from somewhere and you want to read the contents of that buffer. First you convert it to a stream:
private static List<Content> read(byte[] buffer) {
try {
ByteArrayInputStream bytesStream = new ByteArrayInputStream(buffer);
return read(bytesStream);
} catch (IOException e) {
e.printStackTrace();
}
}
The above function wraps the byte array with a stream and passes it to the function that does the actual reading.
If you can start out from a stream then obviously you can skip the above step and just pass that stream into the below function directly:
private static List<Content> read(InputStream bytesStream) throws IOException {
List<Content> results = new ArrayList<Content>();
try {
// read the content...
Content content1 = readContent(bytesStream);
results.add(content1);
// I don't know if there's more than one content block but assuming
// that there is, you can just continue reading the stream...
//
// If it's a fixed number of content blocks then just read them one
// after the other... Otherwise make this a loop
Content content2 = readContent(bytesStream);
results.add(content2);
} finally {
bytesStream.close();
}
return results;
}
Since your byte-array contains content you will want to read Content blocks from the stream. Since you have a length and a type field, I am assuming that you have different kinds of content blocks. The next function reads the length and type and passes the processing of the content bytes on to the proper class depending on the read type:
private static Content readContent(InputStream stream) throws IOException {
final int CONTENT_TYPE_A = 10;
final int CONTENT_TYPE_B = 11;
// wrap the InputStream in a DataInputStream because the latter has
// convenience functions to convert bytes to integers, etc.
// Note that DataInputStream handles the stream in a BigEndian way,
// so check that your bytes are in the same byte order. If not you'll
// have to find another stream reader that can convert to ints from
// LittleEndian byte order.
DataInputStream data = new DataInputStream(stream);
int length = data.readInt();
int type = data.readInt();
// I'm assuming that above length field was the number of bytes for the
// content. So, read length number of bytes into a buffer and pass that
// to your `parseFrom(byte[])` function
byte[] contentBytes = new byte[length];
int readCount = data.read(contentBytes, 0, contentBytes.length);
if (readCount < contentBytes.length)
throw new IOException("Unexpected end of stream");
switch (type) {
case CONTENT_TYPE_A:
return ContentTypeA.parseFrom(contentBytes);
case CONTENT_TYPE_B:
return ContentTypeB.parseFrom(contentBytes);
default:
throw new UnsupportedOperationException();
}
}
I have made up the below Content classes. I don't know what protobuf is but it can apparently convert from a byte array to an actual object with its parseFrom(byte[]) function, so take this as pseudocode:
class Content {
// common functionality
}
class ContentTypeA extends Content {
public static ContentTypeA parseFrom(byte[] contentBytes) {
return null; // do the actual parsing of a type A content
}
}
class ContentTypeB extends Content {
public static ContentTypeB parseFrom(byte[] contentBytes) {
return null; // do the actual parsing of a type B content
}
}

In Java, Array is not just section of memory - it is an object, that have some additional fields (at least - length). So you cannot link to part of array - you should:
Use array-copy functions or
Implement and use some algorithm that uses only part of byte array.

The concern seems that there is no way to create a view over an array (e.g., an array equivalent of List#subList()). A workaround might be making your parsing methods take in the reference to the entire array and two indices (or an index and a length) to specify the sub-array the method should work on.
This would not prevent the methods from reading or modifying sections of the array they should not touch. Perhaps an ByteArrayView class could be made to add a little bit of safety if this is a concern:
public class ByteArrayView {
private final byte[] array;
private final int start;
private final int length;
public ByteArrayView(byte[] array, int start, int length) { ... }
public byte[] get(int index) {
if (index < 0 || index >= length) {
throw new ArrayOutOfBoundsExceptionOrSomeOtherRelevantException();
}
return array[start + index];
}
}
But if, on the other hand, performance is a concern, then a method call to get() for fetching each byte is probably undesirable.
The code is for illustration; it's not tested or anything.
EDIT
On a second reading of my own answer, I realized that I should point this out: having a ByteArrayView will copy each byte you read from the original array -- just byte by byte rather than as a chunk. It would be inadequate for the OP's concerns.

Are there any design patterns for high performance file parsing?

I've recently developed my own file parsing class called the BufferedParseStream, and used this to decode PNG images. I've been comparing it's performance against the open source project PNGJ, and have seen that for smaller image sizes, PNGJ can be up to twice as fast as my own implementation. I assume this is associated with the implementation overhead when using the BufferedInputStream, as PNGJ roll their own equivalent instead.
Are there any existing design patterns which guide high performance file parsing, into primitives such as an int, float etc.?
public class BufferedParseStream extends BufferedInputStream {
private final ByteBuffer mByteBuffer;
public BufferedParseStream(final InputStream pInputStream, final int pBufferSize) {
super(pInputStream, pBufferSize);
/* Initialize the ByteBuffer. */
this.mByteBuffer = DataUtils.delegateNative(new byte[8]);
}
private final void buffer(final int pNumBytes) throws IOException {
/* Read the bytes into the ByteStorage. */
this.read(this.getByteBuffer().array(), 0, pNumBytes);
/* Reset the ByteBuffer Location. */
this.getByteBuffer().position(0);
}
public final char parseChar() throws IOException {
/* Read a single byte. */
this.buffer(DataUtils.BYTES_PER_CHAR);
/* Return the corresponding character. */
return this.getByteBuffer().getChar();
}
public final int parseInt() throws IOException {
/* Read four bytes. */
this.buffer(DataUtils.BYTES_PER_INT);
/* Return the corresponding integer. */
return this.getByteBuffer().getInt();
}
public final long parseLong() throws IOException {
/* Read eight bytes. */
this.buffer(DataUtils.BYTES_PER_LONG);
/* Return the corresponding long. */
return this.getByteBuffer().getLong();
}
public final void setParseOrder(final ByteOrder pByteOrder) {
this.getByteBuffer().order(pByteOrder);
}
private final ByteBuffer getByteBuffer() {
return this.mByteBuffer;
}
}

Java nio should be faster than using input streams, your class that you present seems odd to me (might just be me though :)) because it has an extra layer on top of ByteBuffer which I don't think is required.
You should use the byte buffer directly, it has a getInt, getFloat method which you can feed directly in to the required variables.
I think though your performance problems could be in the PNG decoder code as someone else has already mentioned. You should post that for further analysis

Streaming URL Encoder

In my Java app, I'm looking for a streaming version of URLEncoder.encode(String s, String enc). I'd like to stream a large HTTP post request using the "application/x-www-form-urlencoded" content type. Does such a thing exist either in a library, or an open source project? Or is there an easy way to implement it?
This was an early attempt, but is incorrect because it doesn't handle UTF codepoints larger than one byte:
// Incorrect attempt at creating a URLEncoder OutputStream
private class URLEncoderOutputStream extends FilterOutputStream
{
public URLEncoderOutputStream(OutputStream out)
{
super(out);
}
#Override
public void write(int b) throws IOException
{
String s = new String(new byte[] { (byte)b });
String enc = URLEncoder.encode(s, "UTF-8");
out.write(enc.getBytes("UTF-8"));
}
}

The problem is that OutputStreams don't know anything about characters, only bytes. What you really want is a Writer, e.g.
public class URLEncodedWriter extends FilterWriter {
public void write(int c) {
out.write(URLEncoder.encode((char)c, "UTF-8"));
}
... // Same for 2 other write() methods
}

I think the answer is I shouldn't be trying to do this. According to the HTML Specification:
The content type "application/x-www-form-urlencoded" is inefficient for sending large quantities of binary data or text containing non-ASCII characters. The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.
Most servers will reject HTTP headers that exceed a certain length in any case.

DelimiterBasedFrameDecoder, how does it works?

i premiss that i am a newbie of netty; i am trying to create a client to an external server, this server outputs messages that terminates with 0x0d so i decide to use the DelimiterBasedFrameDecoder to handle these messages.
This is just a test of the handler:
public class TestHandler extends DelimiterBasedFrameDecoder {
public TestHandler(){
super(200, true, ChannelBuffers.wrappedBuffer(new byte[] { 0x0d }));
}
#Override
protected Object decode(ChannelHandlerContext ctx, Channel ch,
ChannelBuffer cbuf) throws Exception {
ByteBuffer buf = ByteBuffer.allocate(cbuf.readableBytes());
cbuf.readBytes(buf);
byte[] data = buf.array();
for(byte b : data){
System.out.print(b + " ");
}
System.out.println();
...... (some other code)
}
what i see wrong from this is that it doesn't strip the delimiter as i specified in the constructor; at the end of the byte[] data i always have the 0x0d;
So, just as test, i changed the delimiter in the constructor putting on it a test value like 0x55
super(200, true, ChannelBuffers.wrappedBuffer(new byte[] { 0x55 }));
and it works in the same way, there is not difference from before.
I think i am using it in the wrong way, or i am reading the data in the wrong way.
What is the right way to use this class?
To be clear, in the real code from this handler i create an object from the read data and i return this object from the decode() method, then i have another handler that extends SimpleChannelHandler that get this object ( it is similar to the example in the user guide ).
Thanks for the help
Bye

I think your decode method is not using the "stripBytes" part of the constructor properly.
If you check out the code in DelimiterBasedFrameDecoder of Netty, you will see the following if condition in code, which is missing in your overriden decode method. This is causing the bytes not to be stripped.
if (stripDelimiter) {
frame = buffer.readBytes(minFrameLength);
buffer.skipBytes(minDelimLength);
} else {
frame = buffer.readBytes(minFrameLength + minDelimLength);
}

Writing a string to an OutputStream without using int onebyte

This is a total beginner question, I've spent the past hour searching both stackoverflow and Google, but I haven't found what I'm looking for, hopefully someone here can point me in the right direction.
I'm trying to write a string to an OutputStream, which I will then use to write data to a MySQL database. I've successfully retrieved data from a MySQL (from a .php, implementing JSON and RESTful), so I have some idea of what I'm doing, I think. I'm creating a method which will take a string and return an output stream, and I'm having trouble writing to an output stream, because when I try to initialize one, it creates an anonymous inner class with the write(int oneByte) method. That's not what I want.
private static OutputStream convertStringtoStream(String string) {
byte[] stringByte = string.getBytes();
OutputStream os = new OutputStream() {
#Override
public void write(int oneByte) throws IOException {
/** I'd rather this method be something like
public void write(byte[] bytes), but it requires int oneByte*/
}
};
//return os here
}
As you can see, I want to write to my OutputStream with the buffer, not a single byte. I'm sure this is simple question, but I've not been able to find an answer, or even sample code which does what I want. If someone could point me in the right direction I'd really appreciate it. Thanks.

Your method could look like this, but I'm not sure what it would accomplish. How would you use the returned OutputStream?
private static OutputStream convertStringtoStream(String string) {
byte[] stringByte = string.getBytes();
ByteArrayOutputStream bos = new ByteArrayOutputStream(string.length());
bos.write(stringByte);
return bos;
}
Also, note that using String.getBytes() might get you into trouble in the long run because it uses the system's default encoding. It's better to choose an explicit encoding and use the String.getBytes(Charset) method.

Instead of using the abstract OutputStream class, you might want to use ByteArrayOutputStream which allows you to write a buffer. Even better perhaps would be ObjectOutputStream which would allow you to write string directly since string is serializable. Hope that helps.