I need to make a byte array in which I will have header values initially and my actual message will come after the header values.
My header values will have - data center which is a string, client_id which is integer, pool_id which is also integer and data_count is also an integer.
And my actual message which will come after header values is - hello world
In my case, my header length may grow so I need to initialize that as a variable so that I can increase it later on as needed.
I am little bit confuse in how to use Byte Array here. How can I represent this in a byte array in network byte order so that c++ program can decode this out properly on ubuntu 12.04 machine?
You can use Protocol Buffers to represent the messages (header and content). It will handle the transformations between languages and different platforms. Also, it is providing room for further expansion and support for multiple message versions.
For your example you can define the message format like (eg. messageModel.proto):
package common;
option java_package = "my.java.package";
option java_outer_classname = "MessageProto";
message MyMessage {
optional string dataCenter = 1 [default = DEFAULT_DC];
optional int64 clientId = 2;
optional int64 poolId = 3;
optional int64 dataCount = 4;
optional string body = 5;
}
Then using the protoc compile like:
protoc -I src/java/ --java_out=src/java/ messageModel.proto
You will generate the transport objects and the utility classes to marshal them from one endpoint to another (representing different messages even). Please check the java tutorial for more details.
To create a MyMessage from java you will be able to do something like:
MessageProto.MyMessage.Builder mb = MessageProto.MyMessage.newBuilder();
mb.setDataCenter("aDC");
mb.setClientId(12);
mb.setPoolId(14);
mb.setDataCount(2);
mb.setbody("hello world");
MessageProto.MyMessage message = mb.build();
To transform the message into a byte array, you will use: message.toByteArray()
If C++/C is your destination you will need to generate (from the same model) the C builders and objects too. And to decode the message you will do something like:
MessageProto.MyMessage message = MessageProto.MyMessage.parseFrom(buffer);
Where buffer will represent the received content.
If this is only a homework assignment then you can serialize your header and body message using
a DataOutputStream, but I would suggest investigating Protocol Buffers as well.
Try using a DataOutputStream that is targeted to a ByteArrayOutputStream. When you're done with writing the message to the DataOutputStream, you can obtain the constructed byte array from the ByteArrayOutputStream.
Like this:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(baos);
dos.writeInt(client_id);
dos.writeUTF(data_center);
// etc...
byte[] message = baos.toByteArray();
Protocol Buffers are also a good option, if you want more flexibility and higher performance. It depends on what you want to get out of this application; if it needs higher performance, or whether it's a one-off throwaway app or something that you expect to grow and be maintained in the longer future. DataOutputStream and DataInputStream are simple to use and you can start right away, you need to invest a bit more of your time to learn Protocol Buffers.
Related
I want to determine JSON response payload size for REST API using Java. To describe further I have to design a bulk API that is exporting large data from application, pagination and security is in place but I want to limit the individual page size to ensure the service continuity and performance output.
your help in this regards is appreciated Thanks in advance!
You're likely to need to actually encode it to JSON to be able to calculate that. The data model may not accurately indicate exactly what properties are exposed via JSON, what they're called, or how they're encoded.
There are a few other questions on SO asking roughly the same thing, see
is there an easy way to estimate size of a json object? and
Calculate size in bytes of JSON payload including it in the JSON payload in PHP
You don't say how you're using JSON, but it's fairly likely it's using Jackson, and perhaps Spring, so something like this ought to put you in the basic ballpark
ObjectMapper om = new ObjectMapper();
Object example = ... // the object you want to find the JSON size for
String jsonString = om.writeValueAsString(example); // get the JSON string
int characterCount = json.length(); // count the number of characters
byte[] bytes = om.writeValueAsBytes(example); // get the JSON as UTF-8 bytes
int byteCount = bytes.length; // count the number of bytes
The difference between characterCount and byteCount may not be obvious, especially if you're not used to dealing with Unicode code-points. HTTP Content-Length indicates the number of octets (bytes), so byteCount would be most accurate, see What's the "Content-Length" field in HTTP header?.
The Server should set the Content-Length header in the response (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length). This is the size of the payload in bytes.
So, the client can read the content length like this:
HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
connection.connect();
int length = connection.getContentLength();
I have asked this question https://stackoverflow.com/questions/32735189/sending-files-from-java-server-to-unity3d-c-sharp-client but I saw that it isn't an optimal solution to send files between Java and C# via built-in operations, because I also need also other messages, not only the file content.
Therefore, I tried using Protobuf, because it is fast and can serialize/deserialize objects platform independent. My .proto file is the following:
message File{
optional int32 fileSize = 1;
optional string fileName = 2;
optional bytes fileContent = 3;
}
So, I set the values for each variable in the generated .java file:
file.setFileSize(fileSize);
file.setFileName(fileName);
file.setFileContent(ByteString.copyFrom(fileContent, 0, fileContent.length);
I saw many tutorials about how to write the objects to a file and read from it. However, I can't find any example about how to send a file from server socket to client socket.
My intention is to serialize the object (file size, file name and file content) on the java server and to send these information to the C# client. So, the file can be deserialized and stored on the client side.
In my example code above, the server read the bytes of the file (image file) and write it to the output stream, so that the client can read and write the bytes to disk through input stream. I want to achieve the same thing with serialization of my generated .proto file.
Can anyone provide me an example or give me a hint how to do that?
As described in the documentation, protobuf does not take care of where a message start and stops, so when using a stream socket like TCP you'll have to do that yourself.
From the doc:
[...] If you want to write multiple messages to a single file or stream, it is up to you to keep track of where one message ends and the next begins. The Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer. [...]
Length-prefixing is a good candidate. Depending on what language you're writing, there are libraries that does length-prefixing for e.g. TCP that you can use, or you can define it yourself.
An example representation of the buffer on the wire might beof the format might be (beginning of buffer to the left):
[buf_length|serialized_buffer2]
So you code to pack the the buffer before sending might look something like (this is in javascript with node.js):
function pack(message) {
var packet = new Buffer(message.length + 2);
packet.writeIntBE(message.length, 0, 2);
message.copy(packet, 2);
return packet;
}
To read you would have to do the opposite:
client.on('data', function (data) {
dataBuffer = Buffer.concat([dataBuffer, data]);
var dataLen = dataBuffer.readIntBE(0, 2);
while(dataBuffer.length >= dataLen) {
// Message length excluding length prefix of 2 bytes
var msgLen = dataBuffer.readIntBE(0, 2);
var thisMsg = new Buffer(dataBuffer.slice(2, msgLen + 2));
//do something with the msg here
// Remove processed message from buffer
dataBuffer = dataBuffer.slice(msgLen + 2);
}
});
You should also be aware of that when sending multiple protobufs on a TCP socket, they are likely to be buffered for network optimizations (concatenated) and sent together. Meaning some sort of delimiter is needed anyway.
I am currently using google's protocol buffers. It works painlessly between Java and C#, however I am running into problems trying to achieve the same use with Obj-c.
The Java WS returns a byte[]. The code that uses the protocol buffer API is simple enough:
productGroup.toByteArray();
I am able to recreate the object with Skeet's C# port, using:
byte[] result = searchWebService.SearchProductsProtocolBuffer(search);
ProductProtoGroup products = ProductProtoGroup.ParseFrom(result);
However, on the obj-c side, I am struggling to work with the return value.
The NSString I receive from the same web service RPC is this:
CmYKEzgwMDAwMUFELTEzMjUyNzk5MTQySUZPT0QgJiBCRV...
I'm not quite sure what to do with this, because I don't know what it is, save it was generated from an array of bytes. I tried parsing it directly to NSData using
NSData* data = [returnValue dataUsingEncoding:NSUTF8StringEncoding];
but on [ProductProtoGroup parseFromData:data];, I get an InvalidProtocolBuffer
I've checked on the Java side what byte string/hexadecimal representations of the original byte[], and it doesn't match the string I receive from the ws.
Hexadecimal is 0-F. Perhaps each byte was converted to a char? No, that doesn't match.
Any help would be appreciated.
Dane
With a fresher mind and some helpful comments, I finally got it.
I was wondering how the web service automagically sent a byte[] in Java, then reconstructed it in C# later, and also how to store this message later.
Turns out the string,
CmYKEzgwMDAwMUFELTEzMjUyNzk5MTQySUZPT0QgJiBCRV...
was indeed what was being sent in the soap envelopes. So inbetween creation of a byte[] in Java and transmission, something was happening. Looking at the message handler in C#,
[System.Web.Services.Protocols.SoapDocumentMethodAttribute("",
RequestNamespace="x", ResponseNamespace="x",
Use=System.Web.Services.Description.SoapBindingUse.Literal,
ParameterStyle=System.Web.Services.Protocols.SoapParameterStyle.Wrapped)]
[return: System.Xml.Serialization.XmlElementAttribute("return",
Form=System.Xml.Schema.XmlSchemaForm.Unqualified,
DataType="base64Binary", IsNullable=true)]
public byte[] searchProductProtocolBuffer([System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)] string arg0)
{
object[] results = this.Invoke("searchProductProtocolBuffer", new object[] {
arg0});
return ((byte[])(results[0]));
}
So base64Binary!, which I admittedly am meeting for the first time. Poking around, Skeet says it is the safest way:
How can I safely convert a byte array into a string and back?
So, knowing the encoding, the solution becomes straight forward. Using the algorithm presented in an answer to this question: How do I do base64 encoding on iphone-sdk?, my final code becomes:
NSString* returnValue = [WebServiceUtil processStringReturnValue:value];
NSData* data = [Encoding base64DataFromString:returnValue];
ProductProtoGroup* products = [ProductProtoGroup parseFromData:data];
I know very little about Objective C, but an NSString is no byte array. Have you tried converting the NSString to char* using -[NSString UTF8String]?
i have a problem when reading special charatters from oracle database (use JDBC driver and glassfish tooplink).
I store on database the name "GRØNLÅEN KJÆTIL" through WebService and, on database, the data are store correctly.
But when i read this String, print on log file and convert this in byte array whit this code:
int pos = 0;
byte[] msg=new byte[1024];
String F = "F" + passenger.getName();
logger.debug("Add " + F + " " + F.length());
msg = addStringToArrayBytePlusSeparator(msg, F,pos);
..............
private byte[] addStringToArrayBytePlusSeparator(byte[] arrDest,String strToAdd,int destPosition)
{
System.arraycopy(strToAdd.getBytes(Charset.forName("ISO-8859-1")), 0, arrDest, destPosition, strToAdd.getBytes().length);
arrDest = addSeparator(arrDest,destPosition+strToAdd.getBytes().length,1);
return arrDest;
}
1) In the log file there is:"Add FGRÃNLÃ " (the name isn't correct and the F.length() are not printed).
2) The code throw:
java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at it.edea.ebooking.business.chi.control.VingCardImpl.addStringToArrayBytePlusSeparator(Test.java:225).
Any solution?
Tanks
You're calling strToAdd.getBytes() without specifying the character encoding, within the System.arraycopy call - that will be using the system default encoding, which may well not be ISO-8859-1. You should be consistent in which encoding you use. Frankly I'd also suggest that you use UTF-8 rather than ISO-8859-1 if you have the choice, but that's a different matter.
Why are you dealing with byte arrays anyway at this point? Why not just use strings?
Also note that your addStringToArrayBytePlusSeparator method doesn't give any indication of how many bytes it's copied, which means the caller won't have any idea what to do with it afterwards. If you must use byte arrays like this, I'd suggest making addStringToArrayBytePlusSeparator return either the new "end of logical array" or the number of bytes copied. For example:
private static final Charset ISO_8859_1 = Charset.forName("ISO-8859-1");
/**
* (Insert fuller description here.)
* Returns the number of bytes written to the array
*/
private static int addStringToArrayBytePlusSeparator(byte[] arrDest,
String strToAdd,
int destPosition)
{
byte[] encodedText = ISO_8859_1.getBytes(strToAdd);
// TODO: Verify that there's enough space in the array
System.arraycopy(encodedText, 0, arrDest, destPosition, encodedText.length);
return encodedText.length;
}
Encoding/Decoding problems are hard. In every process step you have to do the correct encoding/decoding. So,
familiarize yourself with the difference of bytes (inputstream) and Characters (Readers, Strings)
Choose in which character encoding you want to store your data in the database, and in which character encoding you want to expose your webservice. Make sure when you load initial data in the database it's in the right encoding
connect with the right database properties. mysql requires an addition to the connection url:?useUnicode=true&characterEncoding=UTF-8 when using UTF-8, I don't know about oracle.
if you print/debug at a certain step and it looks ok, you can't be sure you did it right. The logger can write with the wrong encoding (sometimes making something look ok, while in fact it's broken). Your terminal might not handle strange byte encodings correct. The same holds for command-line database clients. Your data might wrongly be stored, but your wrongly configured terminal interprets/shows the data as correct.
In XML, it's not only the stream encoding that matters, but also the xml-encoding attribute.
I am using ByteBuffer to transfer data with java nio. A same message can be sent to multiple receivers. The message format is "message header + message content". A staright way is to allocate a new byte buffer for each receiver. This is not effiecient.
My question is whether there is similar java function for ByteBuffer to pointer funciton in C/C++. So I can use one buffer to hold message content and concate with different headers. In this way, it is efficiency.
thanks.
In Java your can use a GatheringByteChannel (which you most probably are dealing with). It allows to have one static buffer containing the header and an individual buffer for each client holding the varying contents. For some material to get started you might want to check out this blog post:
http://javaol.wordpress.com/2011/05/06/java-nio-scatter-gather/
I use a single ByteBuffer to send to multiple receivers.
ByteBuffer bb = ByteBuffer.allocateDirect(LARGE_BUFFER);
bb.clear();
bb.position(START_OF_CONTENT /* 1024 */);
appendContentTo(bb);
int endOfContent = bb.position();
bb.limit(endOfContent);
for(Connection conn: connections) {
bb.position(START_OF_CONTENT);
/* prepend header BEFORE the position and move the position back */
conn.prependHeader(bb);
conn.write(bb);
}
This way, you can use the same ByteBuffer for every connection. There is only ever one copy of the content.
An example of what conn.prependHeader() might look like
public void prependHeader(ByteBuffer bb) {
// bb starts at the start of the content.
int pos = bb.position();
// it would be better if a byte[] wasn't required. This is just an example
byte[] header = getHeaderAsBytes();
bb.position(bb.position()-header.length);
bb.put(header);
// bb starts at the start of the header.
bb.position(bb.position()-header.length);
}