I am reading an array of bytes passed in to me (not my choice, but i have to use it this way). I need to get the data to a LinkedBlockingQueue, and ultimately step through the bytes to build one or more (may contain partial messages) xml messages. So my question is this:
What generic should i use for the LBQ type?
what is the most efficient way to get the byte[] to that generic type?
Here is my example code:
parsebytes(byte[] bytes, int length)
{
//assume that i am doing other checks on data
if (length > 0)
{
myThread.putBytes(bytes, length);
}
}
in my thread:
putBytes(byte[] bytes, int length)
{
for (int i = 0; i < length; i++)
{
blockingQueue.put(bytes[i]);
}
}
I also do not want to have to pull off the blocking queue byte-by-byte either. I would rather grab everything that is in the queue and process it.
There is no such thing as a ListBlockingQueue. However, any BlockingQueue<Object> will accept byte[] since Java arrays are objects.
In the absence of other design considerations, the simplest option might be to just stick the arrays into the queue as they arrive, and let the consumer stich them together.
Consider this:
BlockingQueue<byte[]> q = new LinkedBlockingQueue<>();
q.put(new byte[] {1,2,3});
byte[] bytes = q.take();
Related
Hello: I wonder how to write an asynchronous table iterator. Suppose the input table consists of many rows, and when the table is received, it is in serialized format.
When the table is received, the iterator is called to retrieve one row by one row.
It performs the reading and deserialization in the following way:
1) It first reads the integer about the size of the row and deserialize it.
2) Then it reads and deserialize the contents of the row, in which,
a. timestamp is first ready by calling in.readint(),
b. then each key of the row is read and deserialized,
c. then the bitmap string about the non-key columns is read and deserialized.
d. then calls in.readint() to read and deserialize the integer which represents the number of non-key columns, and then it reads and deserialize each non-key column.
3) Finally it reads and deserializes the file end marker, which indicates if the end of the file is reached.
Finally it returns the deserialized row.
Here is the code
enter code here
public Row next() {
/* It first reads the integer about the size of the row and
deserialize it. */
int size = in.readInt();
/*Then it reads and deserialize the contents of the row*/
Row row = Row.deserialize(descriptor, in);
/*Finally it reads and deserializes the file end marker, which
indicates if the end of the file is reached.*/
int signal = in.readInt();
if (signal == FILE.END) {
file_end = true;
return row;
}
return row;
}
public Row deserialize(DataInput in) throws IOException {
/*timestamp is first ready by calling in.readint()*/
long timestamp= in.readLong();
Object[] Key = new Object[KeyColumns().size()];
Map<Column, Object> columns = new HashMap<>();
/*then each key of the row is read and deserialized */
int i = 0;
for (Column<?> col : KeyColumns()) {
Key[i++] = col.type.deserialize(in);
}
/* then the bitmap string about the non-key columns is read and
deserialized. */
int bitstring= in.readInt();
/*then calls in.readint() to read and deserialize the integer which
represents the number of non-key columns, and then it reads and
deserialize each non-key column.*/
i = 0;
for (Column<?> col : rowColumns()) {
if ((bitstring & (1 << i)) != 0){
columns.put(col, col.type.deserialize(in));
}
i++;
}
return new Row(timestamp, Key, columns);
}
To convert this iterator into an asynchronous iterator, I am thinking about using CompletableFuture in Java 8 and decoupling the read from deserialization. That is, using a separate thend to handle the reading, like below
public Row next() {
CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
int size= 0;
try {
size = in.readInt();
} catch (IOException e) {
e.printStackTrace();
}
});
Row row = Row.deserialize(descriptor, in);
int signal = in.readInt();
if (signal == FILE.END) {
file_end = true;
return row;
}
return row;
}
But it seems to me that because the thread which does
“size = in.readInt();” and the main thread which does “Row row = Row.deserialize(descriptor, in);” shares the same stream. They need to happen one after one. Still no parallelism is achieved.
Any better way to implement this asynchronous iterator?
Thanks.
First of all, you have a blocking resource (DataInput) at the heart. So no matter what you do, you will have sync on reading the DataInput.
In Java 8 I would definitely implement this with streams. See the following question:
How to implement a Java stream?
The easiest would be to implement a Spliterator and create a stream with it using StreamSupport.stream(...). In a Spliterator you will primarily only need to implement the tryAdvance method which is basically your "read next row" routine. There you'll need to synchronize reading from DataInput.
Once you have your Stream<Row> you will be able to apply different functions to it using map or forEach etc.
To achieve parallelism you'll need to implement the trySplit method in the Spliterator. Here comes the problem: if you can't read from your DataInput in parallel, splitting won't bring you much. But still I think it would make sense creating a new instance of Spliterator for the same DataInput and synchronize them on reading. Reading will not be parallelized, but further processing may be (in a parallel stream).
Given an object byte[], when we want to operate with such object often we need pieces of it. In my particular example i get byte[] from wire where first 4 bytes describe lenght of the message then another 4 bytes the type of the message (an integer that maps to concrete protobuf class) then remaining byte[] is actual content of the message... like this
length|type|content
in order to parse this message i have to pass content part to specific class which knows how to parse an instance from it... the problem is that often there are no methods provided so that you could specify from where to where parser shall read the array...
So what we end up doing is copying remaining chuks of that array, which is not effective...
As far as i know in java it is not possible to create another byte[] reference that actually refers to some original bigger byte[] array with just 2 indexes (this was approach with String that led to memory leaks)...
I wonder how do we solve situations like this? I suppose giving up on protobuf just because it does not provide some parseFrom(byte[], int, int) does not make sence... protobuf is just an example, anything could lack that api...
So does this force us to write inefficient code or there is something that can be done? (appart from adding that method)...
Normally you would tackle this kind of thing with streams.
A stream is an abstraction for reading just what you need to process the current block of data. So you can read the correct number of bytes into a byte array and pass it to your parse function.
You ask 'So does this force us to write inefficient code or there is something that can be done?'
Usually you get your data in the form of a stream and then using the technique demonstrated below will be more performant because you skip making one copy. (Two copies instead of three; once by the OS and once by you. You skip making a copy of the total byte array before you start parsing.) If you actually start out with a byte[] but it is constructed by yourself then you may want to change to constructing an object such as { int length, int type, byte[] contentBytes } instead and pass contentBytes to your parse function.
If you really, really have to start out with byte[] then the below technique is just a more convenient way to parse it, it would not be more performant.
So suppose you got a buffer of bytes from somewhere and you want to read the contents of that buffer. First you convert it to a stream:
private static List<Content> read(byte[] buffer) {
try {
ByteArrayInputStream bytesStream = new ByteArrayInputStream(buffer);
return read(bytesStream);
} catch (IOException e) {
e.printStackTrace();
}
}
The above function wraps the byte array with a stream and passes it to the function that does the actual reading.
If you can start out from a stream then obviously you can skip the above step and just pass that stream into the below function directly:
private static List<Content> read(InputStream bytesStream) throws IOException {
List<Content> results = new ArrayList<Content>();
try {
// read the content...
Content content1 = readContent(bytesStream);
results.add(content1);
// I don't know if there's more than one content block but assuming
// that there is, you can just continue reading the stream...
//
// If it's a fixed number of content blocks then just read them one
// after the other... Otherwise make this a loop
Content content2 = readContent(bytesStream);
results.add(content2);
} finally {
bytesStream.close();
}
return results;
}
Since your byte-array contains content you will want to read Content blocks from the stream. Since you have a length and a type field, I am assuming that you have different kinds of content blocks. The next function reads the length and type and passes the processing of the content bytes on to the proper class depending on the read type:
private static Content readContent(InputStream stream) throws IOException {
final int CONTENT_TYPE_A = 10;
final int CONTENT_TYPE_B = 11;
// wrap the InputStream in a DataInputStream because the latter has
// convenience functions to convert bytes to integers, etc.
// Note that DataInputStream handles the stream in a BigEndian way,
// so check that your bytes are in the same byte order. If not you'll
// have to find another stream reader that can convert to ints from
// LittleEndian byte order.
DataInputStream data = new DataInputStream(stream);
int length = data.readInt();
int type = data.readInt();
// I'm assuming that above length field was the number of bytes for the
// content. So, read length number of bytes into a buffer and pass that
// to your `parseFrom(byte[])` function
byte[] contentBytes = new byte[length];
int readCount = data.read(contentBytes, 0, contentBytes.length);
if (readCount < contentBytes.length)
throw new IOException("Unexpected end of stream");
switch (type) {
case CONTENT_TYPE_A:
return ContentTypeA.parseFrom(contentBytes);
case CONTENT_TYPE_B:
return ContentTypeB.parseFrom(contentBytes);
default:
throw new UnsupportedOperationException();
}
}
I have made up the below Content classes. I don't know what protobuf is but it can apparently convert from a byte array to an actual object with its parseFrom(byte[]) function, so take this as pseudocode:
class Content {
// common functionality
}
class ContentTypeA extends Content {
public static ContentTypeA parseFrom(byte[] contentBytes) {
return null; // do the actual parsing of a type A content
}
}
class ContentTypeB extends Content {
public static ContentTypeB parseFrom(byte[] contentBytes) {
return null; // do the actual parsing of a type B content
}
}
In Java, Array is not just section of memory - it is an object, that have some additional fields (at least - length). So you cannot link to part of array - you should:
Use array-copy functions or
Implement and use some algorithm that uses only part of byte array.
The concern seems that there is no way to create a view over an array (e.g., an array equivalent of List#subList()). A workaround might be making your parsing methods take in the reference to the entire array and two indices (or an index and a length) to specify the sub-array the method should work on.
This would not prevent the methods from reading or modifying sections of the array they should not touch. Perhaps an ByteArrayView class could be made to add a little bit of safety if this is a concern:
public class ByteArrayView {
private final byte[] array;
private final int start;
private final int length;
public ByteArrayView(byte[] array, int start, int length) { ... }
public byte[] get(int index) {
if (index < 0 || index >= length) {
throw new ArrayOutOfBoundsExceptionOrSomeOtherRelevantException();
}
return array[start + index];
}
}
But if, on the other hand, performance is a concern, then a method call to get() for fetching each byte is probably undesirable.
The code is for illustration; it's not tested or anything.
EDIT
On a second reading of my own answer, I realized that I should point this out: having a ByteArrayView will copy each byte you read from the original array -- just byte by byte rather than as a chunk. It would be inadequate for the OP's concerns.
I came across this query: Create a ByteBuf in Netty 4.0 about conversion from byte[] to ByteBuf and ByteBuffer to ByteBuf. I was curious to know about the conversion the other way:
io.netty.buffer.ByteBuf to java.nio.ByteBuffer
and how to do it efficiently, with minimal/no copying? I did some reading and with some trial and error I found this inefficient way of converting it (with two copies):
// io.netty.handler.codec.http.FullHttpRequest fullHttpRequest;
ByteBuf conByteBuf = fullHttpRequest.content ();
int numReadBytes = conByteBuf.readableBytes ();
conBytes = new byte[numReadBytes];
conByteBuf .readBytes (conBytes); // First Copy
ByteBuffer conByteBuffer = ByteBuffer.allocate (conBytes.length);
conByteBuffer.put (conByteBuf); // Second Copy
My question is, can we avoid one or both the copies and make the internal buffer of ByteBuffer to use the internal buffer of ByteBuf.
Thanks!
You should be able to use ByteBuf.nioBuffers(). Which will return a view of the ByteBuf as an array of ByteBuffer objects.
In most cases this array will only have one element, but in some of the more complicated implementations of ByteBuf there may be multiple underlying ByteBuffer objects and ByteBuf.nioBuffers() can return them as-is instead of merging them as would a call to ByteBuf.nioBuffer().
You can tell ahead of time what the array length will be by using ByteBuf.nioBufferCount()
You can at least use ByteBuffer.wrap() to avoid the second copying.
Not particularly efficient but doing the trick:
public static ByteBuffer toNioBuffer(ByteBuf buffer) {
if (buffer.isDirect()) {
return buffer.nioBuffer();
}
final byte[] bytes = new byte[buffer.readableBytes()];
buffer.getBytes(buffer.readerIndex(), bytes);
return ByteBuffer.wrap(bytes);
}
I'm creating a StringBuilder to collect strings that I periodically flush to a server. If the flush fails, I want to keep the strings to try again next time, although in the mean time I might get additional strings to send which must be added to the StringBuilder.
What I want to know is what the most efficient way to do this would be, as this is being done in an Android app where battery usage and thus CPU usage is a big concern. Does calling StringBuilder's toString() function store the resulting string it returns internally so that a subsequent call doesn't have to do the work of copying all the original strings over? Or if the call fails, should I create a new StringBuilder initialized with the return value from toString()?
Here is the OpenJDK source code for StringBuilder:
public String toString() {
// Create a copy, don't share the array
return new String(value, 0, count);
}
The source for the String constructor with those parameters is:
public String(char value[], int offset, int count) {
if (offset < 0) {
throw new StringIndexOutOfBoundsException(offset);
}
if (count < 0) {
throw new StringIndexOutOfBoundsException(count);
}
// Note: offset or count might be near -1>>>1.
if (offset > value.length - count) {
throw new StringIndexOutOfBoundsException(offset + count);
}
this.offset = 0;
this.count = count;
this.value = Arrays.copyOfRange(value, offset, offset+count);
}
So yes, it does create a new String everytime, and yes, it makes a copy of the char[] everytime.
It's important to note that this is one implementation of toString, and another implementation may obviously be different.
It would be an implementation detail. Since java strings are immutable a correct impl can choose to share or create new strings from StringBuilder.toString() even if it's not needed.
As everyone says, you can test to see if this is indeed a real performance issue for you. If it is one (clunky) workaround is to wrap StringBuilder and cache the resulting string. You can use a dirty flag to indicate the content was modified.
StringBuilder.toString API says that a new String object is allocated and initialized to contain the character sequence currently represented by this object.
I am in the process of prototyping Netty for my project. I am trying to implement a simple Text/String oriented protocol on top of Netty. In my pipeline I am using the following:
public class TextProtocolPipelineFactory implements ChannelPipelineFactory
{
#Override
public ChannelPipeline getPipeline() throws Exception
{
// Create a default pipeline implementation.
ChannelPipeline pipeline = pipeline();
// Add the text line codec combination first,
pipeline.addLast("framer", new DelimiterBasedFrameDecoder(2000000, Delimiters.lineDelimiter()));
pipeline.addLast("decoder", new StringDecoder());
pipeline.addLast("encoder", new StringEncoder());
// and then business logic.
pipeline.addLast("handler", new TextProtocolHandler());
return pipeline;
}
}
I have a DelimiterBasedFrameDecoder, a String Decoder, and a String Encoder in the pipeline.
As a result of this setup my incoming message is split into multiple Strings. This results in multiple invocations of the "messageReceived" method of my handler. This is fine. However , this requires me to accumulate these messages in memory and re-construct the message when the last string packet of the message is received.
My question is, what is the most memory efficient way to "accumulate the strings" and then "re-construct them into the final message". I have 3 options so far. They are:
Use a StringBuilder to accumulate and toString to construct. (This gives the worst memory performance. In fact for large payloads with lots of concurrent users this gives non-acceptable performance)
Accumulate into a ByteArray via a ByteArrayOutputStream and then construct using the byte-array (this gives a much better performance than option 1, but it still hogs quite a bit of memory)
Accumulate into a Dymamic Channel Buffer and use toString(charset) to construct. I have not profiled this setup yet but I am curious how this compares to the above two options. Has anyone solved this issue using the Dynamic Channel Buffer?
I am new to Netty and its possible I may be doing something wrong architecturally. Your input will be greatly appreciated.
Thanks in advance
Sohil
Adding my implementation of a custom FrameDecoder for Norman to review
public final class TextProtocolFrameDecoder extends FrameDecoder
{
public static ChannelBuffer messageDelimiter()
{
return ChannelBuffers.wrappedBuffer(new byte[] {'E','O','F'});
}
#Override
protected Object decode(ChannelHandlerContext ctx, Channel channel,ChannelBuffer buffer)
throws Exception
{
int eofIndex = find(buffer, messageDelimiter());
if(eofIndex != -1)
{
ChannelBuffer frame = buffer.readBytes(buffer.readableBytes());
return frame;
}
return null;
}
private static int find(ChannelBuffer haystack, ChannelBuffer needle) {
for (int i = haystack.readerIndex(); i < haystack.writerIndex(); i ++) {
int haystackIndex = i;
int needleIndex;
for (needleIndex = 0; needleIndex < needle.capacity(); needleIndex ++) {
if (haystack.getByte(haystackIndex) != needle.getByte(needleIndex)) {
break;
} else {
haystackIndex ++;
if (haystackIndex == haystack.writerIndex() &&
needleIndex != needle.capacity() - 1) {
return -1;
}
}
}
if (needleIndex == needle.capacity()) {
// Found the needle from the haystack!
return i - haystack.readerIndex();
}
}
return -1;
}
}
I think you would get the best performance if you would implement your own FrameDecoder. This would allow you to buffer all the data till you really need to dispatch it to the next Handler in the chain. Please refer to the FrameDecoder apidocs.
If you don't want to handle the detect of CRLF by yourself it would also be possible to keep the DelimiterBasedFrameDecoder and just add a custom FrameDecoder behind it to assemble the ChannelBuffers that represent a line of text.
In both cases FrameDecoder will take care to minimize memory copies as much as possible by try to just "wrap" buffers and not copy them each time.
That said if you want to have the best performance go with the first approach, if you want it easy go with the second ;)