Develop Reference

Can ChronicleQueue tailers for two different queues be interleaved?

I have two separate ChronicleQueues that were created by independent threads that monitor web socket streams in a Java application. When I read each queue independently in a separate single-thread program, I can traverse each entire queue as expected - using the following minimal code:
final ExcerptTailer queue1Tailer = queue1.createTailer();
final ExcerptTailer queue2Tailer = queue2.createTailer();
while (true)
{
try( final DocumentContext context = queue1Tailer.readingDocument() )
{
if ( isNull(context.wire()) )
break;
counter1++;
queue1Data = context.wire()
.bytes()
.readObject(Queue1Data.class);
queue1Writer.write(String.format("%d\t%d\t%d%n", counter1, queue1Data.getEventTime(), queue1Data.getEventContent()));
}
}
while (true)
{
try( final DocumentContext context = queue2Tailer.readingDocument() )
{
if ( isNull(context.wire()) )
break;
counter2++;
queue2Data = context.wire()
.bytes()
.readObject(Queue2Data.class);
queue2Writer.write(String.format("%d\t%d\t%d%n", counter2, queue2Data.getEventTime(), queue2Data.getEventContent()));
}
}
In the above, I am able to read all the Queue1Data objects, then all the Queue2Data objects and access values as expected. However, when I try to interleave reading the queues (read an object from one queue, based on a property of Queue1Data object (a time stamp), read Queue2Data objects until the first object that is after the time stamp (the limit variable below), of the active Queue1Data object is found - then do something with it) after only one object from the queue2Tailer is read, an exception is thrown .DecoratedBufferUnderflowException: readCheckOffset0 failed. The simplified code that fails is below (I have tried putting the outer while(true) loop inside and outside the the queue2Tailer try block):
final ExcerptTailer queue1Tailer = queue1Queue.createTailer("label1");
try( final DocumentContext queue1Context = queue1Tailer.readingDocument() )
{
final ExcerptTailer queue2Tailer = queue2Queue.createTailer("label2");
while (true)
{
try( final DocumentContext queue2Context = queue2Tailer.readingDocument() )
{
if ( isNull(queue2Context.wire()) )
{
terminate = true;
break;
}
queue2Data = queue2Context.wire()
.bytes()
.readObject(Queue2Data.class);
while(true)
{
queue1Data = queue1Context.wire()
.bytes()
.readObject(Queue1Data.class); // first read succeeds
if (queue1Data.getFieldValue() > limit) // if this fails the inner loop continues
{ // but the second read fails
// cache a value
break;
}
}
// continue working with queu2Data object and cached values
} // end try block for queue2 tailer
} // end outer while loop
} // end outer try block for queue1 tailer
I have tried as above, and also with both Tailers created at the beginning of the function which does the processing (a private function executed when a button is clicked in a relatively simple Java application). Basically I took the loop which worked independently, and put it inside another loop in the function, expecting no problems. I thinking I am missing something crucial in how tailers are positioned and used to read objects, but I cannot figure out what it is - since the same basic code works when reading queues independently. The use of isNull(context.wire()) to determine when there are no more objects in a queue I got from one of the examples, though I am not sure this is the proper way to determine when there are no more objects in a queue when processing the queue sequentially.
Any suggestions would be appreciated.

You're not writing it correctly in the first instance.
Now, there's hardcore way of achieving what you are trying to achieve (that is, do everything explicitly, on lower level), and use MethodReader/MethodWriter magic rovided by Chronicle.
Hardcore way
Writing
// write first event type
try (DocumentContext dc = queueAppender.writingDocument()) {
dc.wire().writeEventName("first").text("Hello first");
}
// write second event type
try (DocumentContext dc = queueAppender.writingDocument()) {
dc.wire().writeEventName("second").text("Hello second");
}
This will write different types of messages into the same queue, and you will be able to easily distinguish those when reading.
Reading
StringBuilder reusable = new StringBuilder();
while (true) {
try (DocumentContext dc = tailer.readingDocument()) {
if (!dc.isPresent) {
continue;
}
dc.wire().readEventName(reusable);
if ("first".contentEquals(reusable)) {
// handle first
} else if ("second".contentEquals(reusable)) {
// handle second
}
// optionally handle other events
}
}
The Chronicle Way (aka Peter's magic)
This works with any marshallable types, as well as any primitive types and CharSequence subclasses (i.e. Strings), and Bytes. For more details have a read of MethodReader/MethodWriter documentation.
Suppose you have some data classes:
public class FirstDataType implements Marshallable { // alternatively - extends SelfDescribingMarshallable
// data fields...
}
public class SecondDataType implements Marshallable { // alternatively - extends SelfDescribingMarshallable
// data fields...
}
Then, to write those data classes to the queue, you just need to define the interface, like this:
interface EventHandler {
void first(FirstDataType first);
void second(SecondDataType second);
}
Writing
Then, writing data is as simple as:
final EventHandler writer = appender.methodWriterBuilder(EventHandler).get();
// assuming firstDatum and secondDatum are created earlier
writer.first(firstDatum);
writer.second(secondDatum);
What this does is the same as in the hardcore section - it writes event name (which is taken from the method name in method writer, i.e. "first" or "second" correspondingly), and then the actual data object.
Reading
Now, to read those events from the queue, you need to provide an implementation of the above interface, that will handle corresponding event types, e.g.:
// you implement this to read data from the queue
private class MyEventHandler implements EventHandler {
public void first(FirstDataType first) {
// handle first type of events
}
public void second(SecondDataType second) {
// handle second type of events
}
}
And then you read as follows:
EventHandler handler = new MyEventHandler();
MethodReader reader = tailer.methodReader(handler);
while (true) {
reader.readOne(); // readOne returns boolean value which can be used to determine if there's no more data, and pause if appropriate
}
Misc
You don't have to use the same interface for reading and writing. In case you want to only read events of second type, you can define another interface:
interface OnlySecond {
void second(SecondDataType second);
}
Now, if you create a handler implementing this interface and give it to tailer#methodReader() call, the readOne() calls will only process events of second type while skipping all others.
This also works for MethodWriters, i.e. if you have several processes writing different types of data and one process consuming all that data, it is not uncommon to define multiple interfaces for writing data and then single interface extending all others for reading, e.g.:
interface FirstOut {
void first(String first);
}
interface SecondOut {
void second(long second);
}
interface ThirdOut {
void third(ThirdDataType third);
}
interface AllIn extends FirstOut, SecondOut, ThirdOut {
}
(I deliberately used different data types for method parameters to show how it is possible to use various types)

With further testing, I have found that nested loops to read multiple queues which contain data in different POJO classes is possible. The problem with the code in the above question is that queue1Context is obtained once, OUTSIDE the loop that I expected to read queue1Data objects. My fundamental misconception was that DocumentContext objects managed stepping through objects in a queue, whereas actually ExcerptTailer objects manage stepping (maintaining indices) when reading a queue sequentially.
In case it might help someone else just getting started with ChronicleQueues, the inner loop in the original question should be:
while(true)
{
try (final DocumentContext queue1Context = queue1Tailer() )
{
queue1Data = queue1Context.wire()
.bytes()
.readObject(Queue1Data.class); // first read succeeds
if (queue1Data.getFieldValue() > limit) // if this fails the inner loop continues as expected
{ // and second and subsequent reads now succeed
// cache a value
break;
}
}
}
And of course the outer-most try block containing queue1Context (in the original code) should be removed.

java instance member corruption

I have a Java 1.6 multi-threaded application (5-7 threads, most idle), which has a strange behavior.
The flow involves updating a device, with a 4-byte ID.
I hold the ID in a private byte-array. When the update succeeds, after ~4 seconds, the device sends a STATUS message, in which I compare it's ID with the one I'm holding, and clear the private bite-array and disable an error-timer.
All the work is done in a singleton class instance.
The strange behavior:
I print the value of the private byte-array from a method, which is called periodically.
Within the 4 seconds waiting for the STATUS message, the log displays a different ID (not garbage, but a 4-byte ID of a different object). Checking the value with a breakpoint shows this invalid value (meaning it's not a log error).
But, yet, when the STATUS message arrives, I compare the ID with the one I'l holding, and they match!
I moved the private member into a synchronized getter/setter, adding a log of the change, which doesn't catch the issue.
Here's a pseudo-code of my setter/getter and periodic status + the disturbing log:
public class Manager {
private volatile byte[] readerID = null;
public synchronized void setReaderID(byte[] readerID) {
this.readerID = readerID;
logger.debug("readerID = {}", StringUtilities.binaryToAscii(this.readerID));
}
public synchronized byte[] getReaderID() {
if (this.readerID == null)
return null;
return Arrays.copyOf(this.readerID, this.readerID.length);
}
/* Called every second */
public void periodicStatus() {
logger.debug("readerID = {}", StringUtilities.binaryToAscii(getReaderID()));
}
}
13:53:46,103|ad-5|INFO |Manager|readerUpdateFinish(): Received firmware install finish for reader 000189D0 at slot 0
13:53:46,103|ad-5|DEBUG|Manager|setReaderID(): readerID = 000189D0
13:53:46,103|ad-5|DEBUG|Manager|readerUpdateFinish(): triggered reader firmware timer, 1526986426103, 000189D0
13:53:46,408|ad-5|DEBUG|Manager|periodicStatus(): readerID = E69EAD03 // <- where's the setter???
13:53:50,030|ad-5|INFO |Manager|readerStatus(): Received status information for reader 000189D0 at slot 0
13:53:50,031|ad-5|DEBUG|Manager|setReaderID(): readerID = null
13:53:50,031|ad-5|DEBUG|Manager|readerStatus(): timer cleared, null
Any ideas?

As Progman pointed out, readerID is passed by reference. It arrives from the transport-layer, held there as an instance-member, and is updated by the following incoming message, with the new ID, which is being displayed.

Converting a Java byte reader to an InputStream

Consider a generic byte reader implementing the following simple API to read an unspecified number of bytes from a data structure that is otherwise inaccessible:
public interface ByteReader
{
public byte[] read() throws IOException; // Returns null only at EOF
}
How could the above be efficiently converted to a standard Java InputStream, so that an application using all methods defined by the InputStream class, works as expected?
A simple solution would be subclassing InputStream to
Call the read() method of the ByteReader as much as needed by the read(...) methods of the InputStream
Buffer the bytes retrieved in a byte[] array
Return part of the byte array as expected, e.g., 1 byte at a time whenever the InputStream read() method is called.
However, this requires more work to be efficient (e.g., for avoiding multiple byte array allocations). Also, for the application to scale to large input sizes, reading everything into memory and then processing is not an option.
Any ideas or open source implementations that could be used?

Create multiple ByteArrayInputStream instances around the returned arrays and use them in a stream that provides for concatenation. You could for instance use SequenceInputStream for this.
Trick is to implement a Enumeration<ByteArrayInputStream> that is can use the ByteReader class.
EDIT: I've implemented this answer, but it is probably better to create your own InputStream instance instead. Unfortunately, this solution does not let you handle IOException gracefully.
final Enumeration<ByteArrayInputStream> basEnum = new Enumeration<ByteArrayInputStream>() {
ByteArrayInputStream baos;
boolean ended;
#Override
public boolean hasMoreElements() {
if (ended) {
return false;
}
if (baos == null) {
getNextBA();
if (ended) {
return false;
}
}
return true;
}
#Override
public ByteArrayInputStream nextElement() {
if (ended) {
throw new NoSuchElementException();
}
if (baos.available() != 0) {
return baos;
}
getNextBA();
return baos;
}
private void getNextBA() {
byte[] next;
try {
next = byteReader.read();
} catch (IOException e) {
throw new IllegalStateException("Issues reading byte arrays");
}
if (next == null) {
ended = true;
return;
}
this.baos = new ByteArrayInputStream(next);
}
};
SequenceInputStream sis = new SequenceInputStream(basEnum);

I assume, by your use of "convert", that a replacement is acceptable.
The easiest way to do this is to just use a ByteArrayInputStream, which already provides all the features you are looking for (but must wrap an existing array), or to use any of the other already provided InputStream for reading data from various sources.
It seems like you may be running the risk of reinventing wheels here. If possible, I would consider scrapping your ByteReader interface entirely, and instead going with one of these options:
Replace with ByteInputStream.
Use the various other InputStream classes (depending on the source of the data).
Extend InputStream with your custom implementation.
I'd stick to the existing InputStream class everywhere. I have no idea how your code is structured but you could, for example, add a getInputStream() method to your current data sources, and have them return an appropriate already-existing InputStream (or a custom subclass if necessary).
By the way, I recommend avoiding the term Reader in your own IO classes, as Reader is already heavily used in the Java SDK to indicate stream readers that operate on encoded character data (as opposed to InputStream which generally operates on raw byte data).

Assembling a Netty Message in the Handler

I am in the process of prototyping Netty for my project. I am trying to implement a simple Text/String oriented protocol on top of Netty. In my pipeline I am using the following:
public class TextProtocolPipelineFactory implements ChannelPipelineFactory
{
#Override
public ChannelPipeline getPipeline() throws Exception
{
// Create a default pipeline implementation.
ChannelPipeline pipeline = pipeline();
// Add the text line codec combination first,
pipeline.addLast("framer", new DelimiterBasedFrameDecoder(2000000, Delimiters.lineDelimiter()));
pipeline.addLast("decoder", new StringDecoder());
pipeline.addLast("encoder", new StringEncoder());
// and then business logic.
pipeline.addLast("handler", new TextProtocolHandler());
return pipeline;
}
}
I have a DelimiterBasedFrameDecoder, a String Decoder, and a String Encoder in the pipeline.
As a result of this setup my incoming message is split into multiple Strings. This results in multiple invocations of the "messageReceived" method of my handler. This is fine. However , this requires me to accumulate these messages in memory and re-construct the message when the last string packet of the message is received.
My question is, what is the most memory efficient way to "accumulate the strings" and then "re-construct them into the final message". I have 3 options so far. They are:
Use a StringBuilder to accumulate and toString to construct. (This gives the worst memory performance. In fact for large payloads with lots of concurrent users this gives non-acceptable performance)
Accumulate into a ByteArray via a ByteArrayOutputStream and then construct using the byte-array (this gives a much better performance than option 1, but it still hogs quite a bit of memory)
Accumulate into a Dymamic Channel Buffer and use toString(charset) to construct. I have not profiled this setup yet but I am curious how this compares to the above two options. Has anyone solved this issue using the Dynamic Channel Buffer?
I am new to Netty and its possible I may be doing something wrong architecturally. Your input will be greatly appreciated.
Thanks in advance
Sohil
Adding my implementation of a custom FrameDecoder for Norman to review
public final class TextProtocolFrameDecoder extends FrameDecoder
{
public static ChannelBuffer messageDelimiter()
{
return ChannelBuffers.wrappedBuffer(new byte[] {'E','O','F'});
}
#Override
protected Object decode(ChannelHandlerContext ctx, Channel channel,ChannelBuffer buffer)
throws Exception
{
int eofIndex = find(buffer, messageDelimiter());
if(eofIndex != -1)
{
ChannelBuffer frame = buffer.readBytes(buffer.readableBytes());
return frame;
}
return null;
}
private static int find(ChannelBuffer haystack, ChannelBuffer needle) {
for (int i = haystack.readerIndex(); i < haystack.writerIndex(); i ++) {
int haystackIndex = i;
int needleIndex;
for (needleIndex = 0; needleIndex < needle.capacity(); needleIndex ++) {
if (haystack.getByte(haystackIndex) != needle.getByte(needleIndex)) {
break;
} else {
haystackIndex ++;
if (haystackIndex == haystack.writerIndex() &&
needleIndex != needle.capacity() - 1) {
return -1;
}
}
}
if (needleIndex == needle.capacity()) {
// Found the needle from the haystack!
return i - haystack.readerIndex();
}
}
return -1;
}
}

I think you would get the best performance if you would implement your own FrameDecoder. This would allow you to buffer all the data till you really need to dispatch it to the next Handler in the chain. Please refer to the FrameDecoder apidocs.
If you don't want to handle the detect of CRLF by yourself it would also be possible to keep the DelimiterBasedFrameDecoder and just add a custom FrameDecoder behind it to assemble the ChannelBuffers that represent a line of text.
In both cases FrameDecoder will take care to minimize memory copies as much as possible by try to just "wrap" buffers and not copy them each time.
That said if you want to have the best performance go with the first approach, if you want it easy go with the second ;)

DelimiterBasedFrameDecoder, how does it works?

i premiss that i am a newbie of netty; i am trying to create a client to an external server, this server outputs messages that terminates with 0x0d so i decide to use the DelimiterBasedFrameDecoder to handle these messages.
This is just a test of the handler:
public class TestHandler extends DelimiterBasedFrameDecoder {
public TestHandler(){
super(200, true, ChannelBuffers.wrappedBuffer(new byte[] { 0x0d }));
}
#Override
protected Object decode(ChannelHandlerContext ctx, Channel ch,
ChannelBuffer cbuf) throws Exception {
ByteBuffer buf = ByteBuffer.allocate(cbuf.readableBytes());
cbuf.readBytes(buf);
byte[] data = buf.array();
for(byte b : data){
System.out.print(b + " ");
}
System.out.println();
...... (some other code)
}
what i see wrong from this is that it doesn't strip the delimiter as i specified in the constructor; at the end of the byte[] data i always have the 0x0d;
So, just as test, i changed the delimiter in the constructor putting on it a test value like 0x55
super(200, true, ChannelBuffers.wrappedBuffer(new byte[] { 0x55 }));
and it works in the same way, there is not difference from before.
I think i am using it in the wrong way, or i am reading the data in the wrong way.
What is the right way to use this class?
To be clear, in the real code from this handler i create an object from the read data and i return this object from the decode() method, then i have another handler that extends SimpleChannelHandler that get this object ( it is similar to the example in the user guide ).
Thanks for the help
Bye

I think your decode method is not using the "stripBytes" part of the constructor properly.
If you check out the code in DelimiterBasedFrameDecoder of Netty, you will see the following if condition in code, which is missing in your overriden decode method. This is causing the bytes not to be stripped.
if (stripDelimiter) {
frame = buffer.readBytes(minFrameLength);
buffer.skipBytes(minDelimLength);
} else {
frame = buffer.readBytes(minFrameLength + minDelimLength);
}