Develop Reference

Creating device driver packets from Java

Now that I have some spare time on my hands, I decided to create a Java program to connect my XBee (i.e. zigbee) chips to my new SmartThings hub. I found a nice tutorial on doing this by creating the packets by hand (https://nzfalco.jimdofree.com/electronic-projects/xbee-to-smartthings/). My next task is to create a set of Java routines to create, send, receive, and access the required packets (i.e. a sequence of bytes).
Having done similar in C for other projects, my first thought was to simple create a class with the packet structure and send it. Something like this:
class DeviceAnnounce {
public byte frameId;
public byte addr64[];
public byte addr16[];
public byte capability;
};
Problem is there does not appear to be a way to cast this "structure" to an array of bytes to send to the device.
Next I thought, we have a serialize capability built into the Java runtime. So I added Serializable to the class and used the writeObject() method to convert the instance into a byte stream. Problem here is that writeObject() converts not only your bytes, but includes the definition of the object in the encoding. Works great for reading and writing object to disk, but it's not creating the packet I need to send to the xbee device.
I finally coded it the hard way, explicitly adding a method to my class that creates the byte array.
class DeviceAnnounce {
public DeviceAnnounce(byte frameId, byte[] addr64, byte[] addr16, byte capability) {
super();
this.frameId = frameId;
this.addr64 = addr64;
this.addr16 = addr16;
this.capability = capability;
}
public byte frameId;
public byte addr64[];
public byte addr16[];
public byte capability;
byte[] getBytes() throws IOException {
byte[] data=new byte[12];
data[0]=frameId;
data[1]=addr64[7];
data[2]=addr64[6];
data[3]=addr64[5];
data[4]=addr64[4];
data[5]=addr64[3];
data[6]=addr64[2];
data[7]=addr64[1];
data[8]=addr64[0];
data[9]=addr16[1];
data[10]=addr16[0];
data[11]=capability;
return data;
}
#Override
public String toString() {
return "DeviceAnnounce [frameId=" + frameId + ", addr64=" + HexUtils.prettyHexString(addr64) + ", addr16="
+ HexUtils.prettyHexString(addr16) + ", capability=" + capability + "]";
}
}
It works, but I keep thinking there must be a better way. Now the 64 dollar (or maybe bit) question. Is there a way to convert a POJO into a simple byte stream/array?

To build a block of bytes for transmitting, I recommend using the built-in ByteBuffer, which e.g. has helpers for 16-, 32-, and 64-bit integers in big- or little-endian.
You should then store the values as you use them, e.g.
public byte frameId;
public long addr64;
public short addr16;
public byte capability;
byte[] getBytes() throws IOException {
ByteBuffer buf = ByteBuffer.allocate(12)
.order(ByteOrder.BIG_ENDIAN/*Network Byte Order*/);
buf.put(frameId);
buf.putLong(addr64);
buf.putShort(addr16);
buf.put(capability);
return buf.array(); // or return the ByteBuffer itself
}

JavaSsist: toBytecode() throws exception java.lang.RuntimeException: remaper.by.moofMonkey.Main class is frozen but him isn't frozen

If I calling toBytecode() method in my context it throws
java.lang.RuntimeException: remaper.by.moofMonkey.Main class is frozen
at javassist.CtClassType.checkModify(CtClassType.java:515)
at javassist.CtClass.getClassFile(CtClass.java:524)
at com.moofMonkey.Main.writeFile(Main.java:340)
at com.moofMonkey.Main.saveClasses(Main.java:324)
at com.moofMonkey.Main.main(Main.java:309)
My context:
.....
for (CtClass cl : modClasses) {
cl.stopPruning(true);
writeFile(cl, "./ModifiedClasses"); //cl.writeFile("./ModifiedClasses");
cl.stopPruning(false);
}
.....
public static void writeFile(CtClass cl, String directoryName) throws Throwable {
System.out.println(">> " + cl.getName());
byte[] bc = cl.toBytecode();
String s = cl.getClassFile().getSourceFile();
int index = new String(bc).indexOf(s);
for(int i = 0; i < s.length(); i++) //KILL SOURCEFILE (c) moofMonkey
bc[index + i] = '-';
DataOutputStream out = cl.makeFileOutput(directoryName);
out.write(bc);
out.flush();
out.close();
}
BUT... But. If I calling analog of writeFile() - cl.writeFile() - all works!
I can do this:
1. Save File
2. Read bytes from him
3. Dp what I need
4. Save File

Having a look into the javadoc of CtClass reveals
Once this method is called, further modifications are not possible any more.
If you change the call order to
String s = cl.getClassFile().getSourceFile();
byte[] bc = cl.toBytecode();
you can call toBytecode.

The exception isn't coming where you call toBytecode but in the next source line, where you call getClassFile. The documentation says that you aren't allowed to call this on a frozen class.
There's a method called getClassFile2 which seems intended to work around this problem:
Returns a class file for this class (read only). Normal applications do not need calling this method. Use getClassFile().
The ClassFile object obtained by this method is read only. Changes to this object might not be reflected on a class file generated by toBytecode(), toClass(), etc.
This method is available even if isFrozen() is true. However, if the class is frozen, it might be also pruned.
The first paragraph suggests that if there's some way to restructure your code so it doesn't need to get a class file for a frozen class, that might be better (or at least better-thought-of by the creators of Javassist).

how to refer part of an array?

Given an object byte[], when we want to operate with such object often we need pieces of it. In my particular example i get byte[] from wire where first 4 bytes describe lenght of the message then another 4 bytes the type of the message (an integer that maps to concrete protobuf class) then remaining byte[] is actual content of the message... like this
length|type|content
in order to parse this message i have to pass content part to specific class which knows how to parse an instance from it... the problem is that often there are no methods provided so that you could specify from where to where parser shall read the array...
So what we end up doing is copying remaining chuks of that array, which is not effective...
As far as i know in java it is not possible to create another byte[] reference that actually refers to some original bigger byte[] array with just 2 indexes (this was approach with String that led to memory leaks)...
I wonder how do we solve situations like this? I suppose giving up on protobuf just because it does not provide some parseFrom(byte[], int, int) does not make sence... protobuf is just an example, anything could lack that api...
So does this force us to write inefficient code or there is something that can be done? (appart from adding that method)...

Normally you would tackle this kind of thing with streams.
A stream is an abstraction for reading just what you need to process the current block of data. So you can read the correct number of bytes into a byte array and pass it to your parse function.
You ask 'So does this force us to write inefficient code or there is something that can be done?'
Usually you get your data in the form of a stream and then using the technique demonstrated below will be more performant because you skip making one copy. (Two copies instead of three; once by the OS and once by you. You skip making a copy of the total byte array before you start parsing.) If you actually start out with a byte[] but it is constructed by yourself then you may want to change to constructing an object such as { int length, int type, byte[] contentBytes } instead and pass contentBytes to your parse function.
If you really, really have to start out with byte[] then the below technique is just a more convenient way to parse it, it would not be more performant.
So suppose you got a buffer of bytes from somewhere and you want to read the contents of that buffer. First you convert it to a stream:
private static List<Content> read(byte[] buffer) {
try {
ByteArrayInputStream bytesStream = new ByteArrayInputStream(buffer);
return read(bytesStream);
} catch (IOException e) {
e.printStackTrace();
}
}
The above function wraps the byte array with a stream and passes it to the function that does the actual reading.
If you can start out from a stream then obviously you can skip the above step and just pass that stream into the below function directly:
private static List<Content> read(InputStream bytesStream) throws IOException {
List<Content> results = new ArrayList<Content>();
try {
// read the content...
Content content1 = readContent(bytesStream);
results.add(content1);
// I don't know if there's more than one content block but assuming
// that there is, you can just continue reading the stream...
//
// If it's a fixed number of content blocks then just read them one
// after the other... Otherwise make this a loop
Content content2 = readContent(bytesStream);
results.add(content2);
} finally {
bytesStream.close();
}
return results;
}
Since your byte-array contains content you will want to read Content blocks from the stream. Since you have a length and a type field, I am assuming that you have different kinds of content blocks. The next function reads the length and type and passes the processing of the content bytes on to the proper class depending on the read type:
private static Content readContent(InputStream stream) throws IOException {
final int CONTENT_TYPE_A = 10;
final int CONTENT_TYPE_B = 11;
// wrap the InputStream in a DataInputStream because the latter has
// convenience functions to convert bytes to integers, etc.
// Note that DataInputStream handles the stream in a BigEndian way,
// so check that your bytes are in the same byte order. If not you'll
// have to find another stream reader that can convert to ints from
// LittleEndian byte order.
DataInputStream data = new DataInputStream(stream);
int length = data.readInt();
int type = data.readInt();
// I'm assuming that above length field was the number of bytes for the
// content. So, read length number of bytes into a buffer and pass that
// to your `parseFrom(byte[])` function
byte[] contentBytes = new byte[length];
int readCount = data.read(contentBytes, 0, contentBytes.length);
if (readCount < contentBytes.length)
throw new IOException("Unexpected end of stream");
switch (type) {
case CONTENT_TYPE_A:
return ContentTypeA.parseFrom(contentBytes);
case CONTENT_TYPE_B:
return ContentTypeB.parseFrom(contentBytes);
default:
throw new UnsupportedOperationException();
}
}
I have made up the below Content classes. I don't know what protobuf is but it can apparently convert from a byte array to an actual object with its parseFrom(byte[]) function, so take this as pseudocode:
class Content {
// common functionality
}
class ContentTypeA extends Content {
public static ContentTypeA parseFrom(byte[] contentBytes) {
return null; // do the actual parsing of a type A content
}
}
class ContentTypeB extends Content {
public static ContentTypeB parseFrom(byte[] contentBytes) {
return null; // do the actual parsing of a type B content
}
}

In Java, Array is not just section of memory - it is an object, that have some additional fields (at least - length). So you cannot link to part of array - you should:
Use array-copy functions or
Implement and use some algorithm that uses only part of byte array.

The concern seems that there is no way to create a view over an array (e.g., an array equivalent of List#subList()). A workaround might be making your parsing methods take in the reference to the entire array and two indices (or an index and a length) to specify the sub-array the method should work on.
This would not prevent the methods from reading or modifying sections of the array they should not touch. Perhaps an ByteArrayView class could be made to add a little bit of safety if this is a concern:
public class ByteArrayView {
private final byte[] array;
private final int start;
private final int length;
public ByteArrayView(byte[] array, int start, int length) { ... }
public byte[] get(int index) {
if (index < 0 || index >= length) {
throw new ArrayOutOfBoundsExceptionOrSomeOtherRelevantException();
}
return array[start + index];
}
}
But if, on the other hand, performance is a concern, then a method call to get() for fetching each byte is probably undesirable.
The code is for illustration; it's not tested or anything.
EDIT
On a second reading of my own answer, I realized that I should point this out: having a ByteArrayView will copy each byte you read from the original array -- just byte by byte rather than as a chunk. It would be inadequate for the OP's concerns.

Netty: Add and Remove FrameDecoder dynamically from a Pipeline, Protocol encapsulation

I've been working with Netty 3.3.1-Final for 3 weeks now.
My Protocol has 3 steps and each step needs a different FrameDecoder:
Read arguments
Transfer some data
Mutual close of the data pipe
I've been through a lot of "blocking" issues that I could not understand. It finally appears to me, reading the org.jboss.netty.example.portunification example that I had some buffer issue when trying to dynamically change my FrameDecoder: the buffer of one FrameDecoder was (probably) not empty when changing for the next one...
Is there a way to do that easily in Netty? Do I have to change my Protocol? Do I need to write one big FrameDecoder and manage a state?
If so, how to avoid code duplication between different protocols with common sub parts (for instance "reading arguments")?
Today I came to the idea of a FrameDecoderUnifier (code below) with the purpose of a way to hot add and remove some FrameDecoder, what do you think?
Thanks for your help!
Renaud
----------- FrameDecoderUnifier class --------------
/**
* This FrameDecoder is able to forward the unused bytes from one decoder to the next one. It provides
* a safe way to replace a FrameDecoder inside a Pipeline.
* It is not safe to just add and remove FrameDecoder dynamically from a Pipeline because there is a risk
* of unread bytes inside the buffer of the FrameDecoder you wan't to remove.
*/
public class FrameDecoderUnifier extends FrameDecoder {
private final Method frameDecoderDecodeMethod;
volatile boolean skip = false;
LastFrameEventHandler eventHandler;
LinkedList<Entry> entries;
Entry entry = null;
public FrameDecoderUnifier(LastFrameEventHandler eventHandler) {
this.eventHandler = eventHandler;
this.entries = new LinkedList<Entry>();
try {
this.frameDecoderDecodeMethod = FrameDecoder.class.getMethod("decode", ChannelHandlerContext.class, Channel.class, ChannelBuffer.class);
} catch (NoSuchMethodException ex) {
throw new RuntimeException(ex);
} catch (SecurityException ex) {
throw new RuntimeException(ex);
}
}
public void addLast(FrameDecoder decoder, LastFrameIdentifier identifier) {
entries.addLast(new Entry(decoder, identifier));
}
private Object callDecode(FrameDecoder decoder, ChannelHandlerContext ctx, Channel channel, ChannelBuffer buffer) throws Exception {
return frameDecoderDecodeMethod.invoke(decoder, ctx, channel, buffer);
}
#Override
protected Object decode(ChannelHandlerContext ctx, Channel channel, ChannelBuffer buffer) throws Exception {
if (entry == null && !entries.isEmpty()) {
entry = entries.getFirst();
}
if (entry == null) {
return buffer; //No framing, no decoding
}
//Perform the decode operation
Object obj = callDecode(entry.getDecoder(), ctx, channel, buffer);
if (obj != null && entry.getIdentifier().isLastFrame(obj)) {
//Fire event
eventHandler.lastObjectDecoded(entry.getDecoder(), obj);
entry = null;
}
return obj;
}
/**
* You can use this interface to take some action when the current decoder is changed for the next one.
* This can be useful to change some upper Handler in the pipeline.
*/
public interface LastFrameEventHandler {
public void lastObjectDecoded(FrameDecoder decoder, Object obj);
}
public interface LastFrameIdentifier {
/**
* True if after this frame, we should disable this decoder.
* #param obj
* #return
*/
public abstract boolean isLastFrame(Object decodedObj);
}
private class Entry {
FrameDecoder decoder;
LastFrameIdentifier identifier;
public Entry(FrameDecoder decoder, LastFrameIdentifier identifier) {
this.decoder = decoder;
this.identifier = identifier;
}
public FrameDecoder getDecoder() {
return decoder;
}
public LastFrameIdentifier getIdentifier() {
return identifier;
}
}
}

I have had similar problems, in that removing a frame decoder from a pipeline does not seem to prevent it from being called, and there isn't an obvious way to make the decoder to behave as if it wasn't in the chain: Netty insists that the decode() reads at least one byte so you can't simply return the incoming ChannelBuffer, whereas returning null stops the processing of incoming data until the next packet arrives, stalling the protocol decoding process.
Firstly: the Netty 3.7 docs for FrameDecoder does in fact has a section "Replacing a decoder with another decoder in a pipeline". It says:
It is not possible to achieve this simply by calling
ChannelPipeline#replace()
Instead, it suggests passing the data on by returning an array wrapping the decoded first packet and the rest of the data received.
return new Object[] { firstMessage, buf.readBytes(buf.readableBytes()) };
Importantly, "unfolding" must have been enabled prior to this, but this part is easy to miss and isn't explained. The best clue I could find was Netty issue 132, which evidently gave rise to the "unfold" flag on FrameDecoders. If true, the decoder will unpack such arrays into objects in a way which is transparent to downstream handlers. A peep at the source code seems to confirm this is what "unfolding" means.
Secondly: there seems to be an even simpler way, since the example also shows how to pass data on down the pipeline unchanged. For example, after doing its job, my sync packet FrameDecoder sets an internal flag and removes itself from the pipeline, returning the decoded object as normal. Any subsequent invocations when the flag is set then simply pass the data on like so:
protected Object decode(ChannelHandlerContext ctx,
Channel channel, ChannelBuffer cbuf) throws Exception {
// Close the door on more than one sync packet being decoded
if (m_received) {
// Pass on the data to the next handler in the pipeline.
// Note we can't just return cbuf as-is, we must drain it
// and return a new one. Otherwise Netty will detect that
// no bytes were read and throw an IllegalStateException.
return cbuf.readBytes(cbuf.readableBytes());
}
// Handle the framing
ChannelBuffer decoded = (ChannelBuffer) super.decode(ctx, channel, cbuf);
if (decoded == null) {
return null;
}
// Remove ourselves from the pipeline now
ctx.getPipeline().remove(this);
m_received = true;
// Can we assume an array backed ChannelBuffer?
// I have only hints that we can't, so let's copy the bytes out.
byte[] sequence = new byte[magicSequence.length];
decoded.readBytes(sequence);
// We got the magic sequence? Return the appropriate SyncMsg
return new SyncMsg(Arrays.equals(sequence, magicSequence));
}
A decoder derived from LengthFieldBasedFrameDecoder remains downstream and handles all subsequent data framing. Works for me, so far.

I think, having a frame decoder which switch internal decoders based on some state and dynamically adding/removing upper layer handlers should be avoided because
Difficult to understand/debug the code
Handlers are not having well defined responsibilities (That's why you are removing/adding handlers right? One handler should handle one or more (related) types of protocol messages, not many handlers same type of messages)
Ideally frame decoder only extract the protocol frame, not to decode the frame based on state (here frame decoder can have internal chain of decoders to decoder the frame and fire a MessageEvent with decoded message, above handlers can react to decoded messages).
UPDATE: Here I have considered a protocol where each messages can have a unique tag/identifier with end of the message is clearly marked (for example Tag Length Value frame format)

Writing a string to an OutputStream without using int onebyte

This is a total beginner question, I've spent the past hour searching both stackoverflow and Google, but I haven't found what I'm looking for, hopefully someone here can point me in the right direction.
I'm trying to write a string to an OutputStream, which I will then use to write data to a MySQL database. I've successfully retrieved data from a MySQL (from a .php, implementing JSON and RESTful), so I have some idea of what I'm doing, I think. I'm creating a method which will take a string and return an output stream, and I'm having trouble writing to an output stream, because when I try to initialize one, it creates an anonymous inner class with the write(int oneByte) method. That's not what I want.
private static OutputStream convertStringtoStream(String string) {
byte[] stringByte = string.getBytes();
OutputStream os = new OutputStream() {
#Override
public void write(int oneByte) throws IOException {
/** I'd rather this method be something like
public void write(byte[] bytes), but it requires int oneByte*/
}
};
//return os here
}
As you can see, I want to write to my OutputStream with the buffer, not a single byte. I'm sure this is simple question, but I've not been able to find an answer, or even sample code which does what I want. If someone could point me in the right direction I'd really appreciate it. Thanks.

Your method could look like this, but I'm not sure what it would accomplish. How would you use the returned OutputStream?
private static OutputStream convertStringtoStream(String string) {
byte[] stringByte = string.getBytes();
ByteArrayOutputStream bos = new ByteArrayOutputStream(string.length());
bos.write(stringByte);
return bos;
}
Also, note that using String.getBytes() might get you into trouble in the long run because it uses the system's default encoding. It's better to choose an explicit encoding and use the String.getBytes(Charset) method.

Instead of using the abstract OutputStream class, you might want to use ByteArrayOutputStream which allows you to write a buffer. Even better perhaps would be ObjectOutputStream which would allow you to write string directly since string is serializable. Hope that helps.