I have implemented a TCP socket using JAVA and Netty where I am able to receive Strings from clients.
Now I also have reqirements to accept XML based strings and not quite sure which kind of delimiter I could use. At the moment I have the following implementation.
pipeline.addLast(new DelimiterBasedFrameDecoder(8192, Delimiters.lineDelimiter()));
With this the XML is read in line by line which is obvious. Now how can I tell Netty to read the whole XML string in? Am looking for some sort of xmlDelimiter.
Could someone help me out?
Thanks
This is a protocol issue that both sender and receiver has to agree upon. A real life example is that of NETCONF (used to configure network devices). NETCONF uses XML for both protocol operations and data encoding, and needed a mechanism for delimiting individual XML messages. The first version of the specification defined the string ]]>]]> as the delimiter. A sequence of XML messages could looke like so:
<rpc message-id="1"> ... </rpc>
]]>]]>
<rpc message-id="2"> ... </rpc>
]]>]]>
It was later realized that this string may legally exist in XML attributes and comments, why a later version of the spec defines a totally different framing mechanism based on length encoded chunks, much like the HTTP chunking mechanism.
If you have control of the clients and what they produce, a simple delimiter based approach like the original NETCONF solution should work fine.
Related
I saw this description in the Oracle website:
"Since TCP by its nature is a stream based protocol, in order to reuse an existing connection, the HTTP protocol has to have a way to indicate the end of the previous response and the beginning of the next one. Thus, it is required that all messages on the connection MUST have a self-defined message length (i.e., one not defined by closure of the connection). Self demarcation is achieved by either setting the Content-Length header, or in the case of chunked transfer encoded entity body, each chunk starts with a size, and the response body ends with a special last chunk."
See Oracle doc
I don't know how to implement, can someone give me an example of Java implementation ?
If you are trying to implement "self-demarcation" in the same way as HTTP does it:
the HTTP 1.1 specification defines how it works,
the source code of (say) the Apache HTTP libraries are an example of its implementation.
In fact, it is advisable NOT to try and implement this (HTTP) yourself from scratch. Use an existing implementation.
On the other hand, if you simply want to implement your own ad-hoc self-demarcation scheme, it is really easy to do.
The sender figures out the size of the message, in bytes or characters or some other unit that makes sense.
The sender sends a the message size, followed by the message itself.
At the other end:
The receiver reads the message size, and then reads the requisite number of bytes, characters, to form the message body.
An alternative is to for the sender to send the message followed by a special end-of-message marker. To make this work, either you need to guarantee that no message will contain the end-of-message marker, or you need to use some sort of escaping mechanism.
Implementing these schemes is simple Java programming.
What makes a connection reusable
That is answered by the text that you quoted in your Question.
I created the following Thrift Object:
struct Student{
1: string id;
2: string firstName;
3: string lastName
}
Now I would like to read this object from JSON. According to this post this is possible
So I wrote the following code:
String json = "{\"id\":\"aaa\",\"firstName\":\"Danny\",\"lastName\":\"Lesnik\"}";
StudentThriftObject s = new StudentThriftObject();
byte[] jsonAsByte = json.getBytes("UTF-8");
TMemoryBuffer memBuffer = new TMemoryBuffer(jsonAsByte.length);
memBuffer.write(jsonAsByte);
TProtocol proto = new TJSONProtocol(memBuffer);
s.read(proto);
What I'm getting is the following exception:
Exception in thread "main" org.apache.thrift.protocol.TProtocolException: Unexpected character:i
at org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:322)
at org.apache.thrift.protocol.TJSONProtocol.readJSONInteger(TJSONProtocol.java:698)
at org.apache.thrift.protocol.TJSONProtocol.readFieldBegin(TJSONProtocol.java:837)
at com.vanilla.thrift.example.entities.StudentThriftObject$StudentThriftObjectStandardScheme.read(StudentThriftObject.java:486)
at com.vanilla.thrift.example.entities.StudentThriftObject$StudentThriftObjectStandardScheme.read(StudentThriftObject.java:479)
at com.vanilla.thrift.example.entities.StudentThriftObject.read(StudentThriftObject.java:413)
at com.vanilla.thrift.controller.Main.main(Main.java:24)
Am I missing something?
You are missing the fact, that Thrift's JSON is different from yours. The field names are not written, instead the assigned field ID numbers are written (and expected). Here's an example for Thrift's JSON protocol:
[1,"MyService",2,1,{"1":{"rec":{"1":{"str":"Error: Process() failed"}}}}]
In other words, Thrift is not intended to parse any kind of JSON. It supports a very specific JSON format as one of the possible transports.
However, depending on what the origin of your JSON data is, Thrift can possibly still help you out, if you are able to use it on both sides. In that case, write an IDL to describe the data structures, feed it to the Thrift compiler and integrate both the generated code and the neccessary parts of the library with your projects.
If the origin of the JSON lies outside of your reach, or if the JSON format cannot be changed for some reason, you need to find another way.
Format and semantics are different beasts
To some extent, the whole issue can be compared with XML: There is one general XML syntax, which tells us how we have to fomat things so any standard conformant XML processor can read them.
But knowing the rules of XML is only half the answer, if we get a certain XML file from someone. Even if our XML parser can read the file successfully, because it is well-formed XML, we need to know the semantics of the data to really make use of what's within that file: Is it a customer data record? Or is it a SOAP envelope? Maybe a configuration file?
That is where DTDs or XML Schema come into play, they exist to describe the contents of the XML data. Without knowing the logical structure you are lost, because there are myriads of possible ways to express things in XML. And exactly the same is true with JSON, except that JSON schema descriptions are less commonly used.
"So you mean, we need just a way to tell Thrift how the JSON is organized?"
No, because the purpose and idea behind Thrift is to have a framework to de/serialize things and/or implement RPC servers and clients as efficiently as possible. It is not intended to have a general purpose file parser. Instead, Thrift reads and speaks only its own set of formats, which are plugged into the architecture as protocols: Thrift Binary, Thrift JSON, Thrift Compact, and a few more.
What you could do: In addition to what I said at in the first section of my answer, you may consider writing your own custom Thrift protocol implementation to support your particular JSON format of choice. It is not that hard, and worth a try.
I have two applications, one that sends UDP messages using Camel with the Netty component, and one that receives UDP messages in Java NIO with DatagramChannel.
When receiving the data, I've noticed that there's an extra 29 bytes prepended to the front of my message. Netty Camel prints out the outgoing bytes and it looks fine, but when I do a packet.getData() as soon as the message comes in on the other side, it has extra stuff on the front (and it's always the same bytes).
Is Camel or Netty wrapping the packet before sending it?
[edit] Additional information:
-Camel is printing the log statement, not Netty
-the bytes prepended to the message change when the content of the message changes (only two bytes are changed)
I know this question is pretty old now, but I hit exactly this problem and it took me a while to find the solution. So here it is...
Basically the problem boils down to confusion of what camel-netty will do when you're telling it to send something sufficiently low-level like a byte[] in a UDP packet. I expect that like me the OP assumed they were setting raw data, but camel-netty uses Java Object Serialization by default - resulting in those extra "random" bytes appearing before the expected data.
The solution is to change the encoder/decoder used by the endpoint(s) in question. There are various built-in alternatives, but you can subclass them if you need something more... weird. Either way, the process is:
1) Add the "encoder=#myEncoder" and "decoder=#myDecoder" options as appropriate on to the endpoint URIs. e.g.
String destinationUri = "netty:udp://localhost:4242"
+ "?sync=false&encoder=#myEncoder";
2) Add a mapping from "myEncoder" to an instance of your new Encoder class to a Camel Registry. Same for myDecoder. Then use that registry when constructing the CamelContext. e.g.
SimpleRegistry registry = new SimpleRegistry();
registry.put("myEncoder", new StringEncoder());
registry.put("myDecoder", new StringDecoder());
CamelContext camelContext = new CamelContext(registry);
Obviously the real trick is in finding or making an Encoder/Decoder that suits your needs. A blog post at znetdevelopment really helped me, though it goes one step further and puts the custom Encoders in a custom Pipeline (I ignored that bit).
I'm new to Netty and Java, and I'm trying to build up a simple Netty server that reads XML from a separate client (The client will keep sending me XMLs with a fixed format). For each XML, I need to do some processing.
I've looked at examples at echo-client/server and Object echo-client/server, and trying to decide which one I should model after, I'm not exactly sure if I should use a ChannelInboundByteHandlerAdapter or a ChannelInboundMessageAdapter?
And are there utility packages in Netty 3.6.x that handle demarshalling XMLs? How should I handle converting the raw data to XML?
Also, I don't know how stable 4.0.0.Beta is, since this app is not too complicated I wonder if it's just OK to use the 3.6.x.Final as it's probably more stable.
Thanks a bunch!!
Create XMLDecoder, XMLEncoder class(extends OneToOneEncoder) in Server program.
And add ChannelPipeline as keys "decoder", "encoder".
In decode of XMLDecoder class, Convert received xml to custom class using JAXB.
If a server send to client using xml, convert response custom class to xml string in encode method of XmlEncoder.
sorry, I have limited English proficiency.
I'm writing an application server that will receive SIP and DNS messages from the network.
When I receive a message from the network, I understand from the documentation that at first, I get a ChannelBuffer. I would like to determine which kind of message has been received (SIP or DNS) and to decode it.
To determine the message type, I can dedicate port to each type of message, but I would be interested to know if there exist another solution for that. My question is more about how to decode the ChannelBuffer.
Is there a ChannelHandler provided by Netty to decode SIP or DNS messages? If not, what would be the right place in the type hierarchy to write my custom ChannelHandler?
To illustrate my question, let's take as example the HttpRequestDecoder, the hierarchy is:
java.lang.Object
org.jboss.netty.channel.SimpleChannelUpstreamHandler
org.jboss.netty.handler.codec.frame.FrameDecoder
org.jboss.netty.handler.codec.replay.ReplayingDecoder<HttpMessageDecoder.State>
org.jboss.netty.handler.codec.http.HttpMessageDecoder
org.jboss.netty.handler.codec.http.HttpRequestDecoder
Also, do I need to use two different ChannelHandler for decoding and encoding, or is there a possibility to use a single ChannelHandler for both?
Thanks
If you really have a requirement for port unification (an example here), i.e. receiving different protocols on the same port, then you would have to detect the protocol in a handler and take appropriate actions. Could be as simple as inserting different handlers in the pipe line.
However, I find it very improbable that SIP and DNS would share the same port, hence no need to complicate matters.
I haven't seen a SIP decoder/encoder for Netty, but depending on what you want to do with the message, the HTTP decoder is a a very good starting point (and could be made simpler since chunking is not supported in SIP).
I would strongly recommend not to try to combine DNS and SIP decoding in one handler (or any other combination for that matter). Keep the handlers as simple and coherent as possible. Combine handlers instead, if needed.