Develop Reference

Send binary file from Java Server to C# Unity3d Client with Protocol Buffer

I have asked this question https://stackoverflow.com/questions/32735189/sending-files-from-java-server-to-unity3d-c-sharp-client but I saw that it isn't an optimal solution to send files between Java and C# via built-in operations, because I also need also other messages, not only the file content.
Therefore, I tried using Protobuf, because it is fast and can serialize/deserialize objects platform independent. My .proto file is the following:
message File{
optional int32 fileSize = 1;
optional string fileName = 2;
optional bytes fileContent = 3;
}
So, I set the values for each variable in the generated .java file:
file.setFileSize(fileSize);
file.setFileName(fileName);
file.setFileContent(ByteString.copyFrom(fileContent, 0, fileContent.length);
I saw many tutorials about how to write the objects to a file and read from it. However, I can't find any example about how to send a file from server socket to client socket.
My intention is to serialize the object (file size, file name and file content) on the java server and to send these information to the C# client. So, the file can be deserialized and stored on the client side.
In my example code above, the server read the bytes of the file (image file) and write it to the output stream, so that the client can read and write the bytes to disk through input stream. I want to achieve the same thing with serialization of my generated .proto file.
Can anyone provide me an example or give me a hint how to do that?

As described in the documentation, protobuf does not take care of where a message start and stops, so when using a stream socket like TCP you'll have to do that yourself.
From the doc:
[...] If you want to write multiple messages to a single file or stream, it is up to you to keep track of where one message ends and the next begins. The Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer. [...]
Length-prefixing is a good candidate. Depending on what language you're writing, there are libraries that does length-prefixing for e.g. TCP that you can use, or you can define it yourself.
An example representation of the buffer on the wire might beof the format might be (beginning of buffer to the left):
[buf_length|serialized_buffer2]
So you code to pack the the buffer before sending might look something like (this is in javascript with node.js):
function pack(message) {
var packet = new Buffer(message.length + 2);
packet.writeIntBE(message.length, 0, 2);
message.copy(packet, 2);
return packet;
}
To read you would have to do the opposite:
client.on('data', function (data) {
dataBuffer = Buffer.concat([dataBuffer, data]);
var dataLen = dataBuffer.readIntBE(0, 2);
while(dataBuffer.length >= dataLen) {
// Message length excluding length prefix of 2 bytes
var msgLen = dataBuffer.readIntBE(0, 2);
var thisMsg = new Buffer(dataBuffer.slice(2, msgLen + 2));
//do something with the msg here
// Remove processed message from buffer
dataBuffer = dataBuffer.slice(msgLen + 2);
}
});
You should also be aware of that when sending multiple protobufs on a TCP socket, they are likely to be buffered for network optimizations (concatenated) and sent together. Meaning some sort of delimiter is needed anyway.

How to represent header value and actual message in Byte Array Java?

I need to make a byte array in which I will have header values initially and my actual message will come after the header values.
My header values will have - data center which is a string, client_id which is integer, pool_id which is also integer and data_count is also an integer.
And my actual message which will come after header values is - hello world
In my case, my header length may grow so I need to initialize that as a variable so that I can increase it later on as needed.
I am little bit confuse in how to use Byte Array here. How can I represent this in a byte array in network byte order so that c++ program can decode this out properly on ubuntu 12.04 machine?

You can use Protocol Buffers to represent the messages (header and content). It will handle the transformations between languages and different platforms. Also, it is providing room for further expansion and support for multiple message versions.
For your example you can define the message format like (eg. messageModel.proto):
package common;
option java_package = "my.java.package";
option java_outer_classname = "MessageProto";
message MyMessage {
optional string dataCenter = 1 [default = DEFAULT_DC];
optional int64 clientId = 2;
optional int64 poolId = 3;
optional int64 dataCount = 4;
optional string body = 5;
}
Then using the protoc compile like:
protoc -I src/java/ --java_out=src/java/ messageModel.proto
You will generate the transport objects and the utility classes to marshal them from one endpoint to another (representing different messages even). Please check the java tutorial for more details.
To create a MyMessage from java you will be able to do something like:
MessageProto.MyMessage.Builder mb = MessageProto.MyMessage.newBuilder();
mb.setDataCenter("aDC");
mb.setClientId(12);
mb.setPoolId(14);
mb.setDataCount(2);
mb.setbody("hello world");
MessageProto.MyMessage message = mb.build();
To transform the message into a byte array, you will use: message.toByteArray()
If C++/C is your destination you will need to generate (from the same model) the C builders and objects too. And to decode the message you will do something like:
MessageProto.MyMessage message = MessageProto.MyMessage.parseFrom(buffer);
Where buffer will represent the received content.
If this is only a homework assignment then you can serialize your header and body message using
a DataOutputStream, but I would suggest investigating Protocol Buffers as well.

Try using a DataOutputStream that is targeted to a ByteArrayOutputStream. When you're done with writing the message to the DataOutputStream, you can obtain the constructed byte array from the ByteArrayOutputStream.
Like this:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(baos);
dos.writeInt(client_id);
dos.writeUTF(data_center);
// etc...
byte[] message = baos.toByteArray();
Protocol Buffers are also a good option, if you want more flexibility and higher performance. It depends on what you want to get out of this application; if it needs higher performance, or whether it's a one-off throwaway app or something that you expect to grow and be maintained in the longer future. DataOutputStream and DataInputStream are simple to use and you can start right away, you need to invest a bit more of your time to learn Protocol Buffers.

Sending images through a socket using qt and read it using java

I'm trying to send an image upload in a Qt server trough the socket and visualize it in a client created using Java. Until now I have only transferred strings to communicate on both sides, and tried different examples for sending images but with no results.
The code I used to transfer the image in qt is:
QImage image;
image.load("../punton.png");
qDebug()<<"Image loaded";
QByteArray ban; // Construct a QByteArray object
QBuffer buffer(&ban); // Construct a QBuffer object using the QbyteArray
image.save(&buffer, "PNG"); // Save the QImage data into the QBuffer
socket->write(ban);
In the other end the code to read in Java is:
BufferedInputStream in = new BufferedInputStream(socket.getInputStream(),1);
File f = new File("C:\\Users\\CLOUDMOTO\\Desktop\\JAVA\\image.png");
System.out.println("Receiving...");
FileOutputStream fout = new FileOutputStream(f);
byte[] by = new byte[1];
for(int len; (len = in.read(by)) > 0;){
fout.write(by, 0, len);
System.out.println("Done!");
}
The process in Java gets stuck until I close the Qt server and after that the file generated is corrupt.
I'll appreciate any help because it's neccessary for me to do this and I'm new to programming with both languages.
Also I've used the following commands that and the receiving process now ends and show a message, but the file is corrupt.
socket->write(ban+"-1");
socket->close(); in qt.
And in java:
System.out.println(by);
String received = new String(by, 0, by.length, "ISO8859_1");
System.out.println(received);
System.out.println("Done!");

You cannot transport file over socket in such simple way. You are not giving the receiver any clue, what number of bytes is coming. Read javadoc for InputStream.read() carefully. Your receiver is in endless loop because it is waiting for next byte until the stream is closed. So you have partially fixed that by calling socket->close() at the sender side. Ideally, you need to write the length of ban into the socket before the buffer, read that length at receiver side and then receive only that amount of bytes. Also flush and close the receiver stream before trying to read the received file.
I have absolutely no idea what you wanted to achieve with socket->write(ban+"-1"). Your logged output starts with %PNG which is correct. I can see there "-1" at the end, which means that you added characters to the image binary file, hence you corrupted it. Why so?
And no, 1x1 PNG does not have size of 1 byte. It does not have even 4 bytes (red,green,blue,alpha). PNG needs some things like header and control checksum. Have a look at the size of the file on filesystem. This is your required by size.

transfer jpeg from c++ to android (java)

I am struggling with the transfer of a simple jpeg file inside an ID3v2 tag from c++ over a TCP socket to java (Android). The library "taglib" offers to extract this file and I am able to save the jpeg as a new file.
The send function looks like this
char *parameter_full = new char[f3->picture().size()+2];
sprintf(parameter_full,"%s\n\0",f3->picture().data());
// send
result = send(c,parameter_full,strlen(parameter_full),0);
delete[] parameter_full;
where
f3->picture().data() returns a pointer to the internal data structure (it returns char*) and
f3->picture().size() returns the size of the array.
Then Android receives it with
String imageString = inFromServer.readLine();
byte[] imageBytes = imageString.getBytes();
Bitmap cover = BitmapFactory.decodeByteArray(imageBytes,0,imageBytes.length);
But somehow decodeByteArray always returns null. My idea is that Java doesn't receive the image correctly because imageString only consists of 4 characters...while the extracted jpeg file has a size of 12.7 KB.
But what has gone wrong?
Martin

You shouldn't use string functions on byte data because 0 values are taken as string terminators. Try looking into memcpy on the C++ side if you need to copy the char* and also the byte[] read functions for InputStream on the Java side.

What could lead to the creation of false EOF in a GZip compressed data stream

We are streaming data between a server (written in .Net running on Windows) to a client (written in Java running on Ubuntu) in batches. The data is in XML format. Occasionally the Java client throws an unexpected EOF while trying decompress the stream. The message content always varies and is user driven. The response from the client is also compressed using GZip. This never fails and seems to be rock solid. The response from the client is controlled by the system.
Is there a chance that some arrangement of characters or some special characters are creating false EOF markers? Could it be white-space related? Is GZip suitable for compressing XML?
I am assuming that the code to read and write from the input/output streams works because we only occasionally gets this exception and when we inspect the user data at the time there seems to be special characters (which is why I asked the question) such as the '#' sign.
Any ideas?
UPDATE:
The actual code as requested. I thought it wasn't this due to the fact that I had been to a couple of sites to get help on this issue and they all more or less had the same code. Some sites mentioned appended GZip. Something to do with GZip creating multiple segments?
public String receive() throws IOException {
byte[] buffer = new byte[8192];
ByteArrayOutputStream baos = new ByteArrayOutputStream(8192);
do {
int nrBytes = in.read(buffer);
if (nrBytes > 0) {
baos.write(buffer, 0, nrBytes);
}
} while (in.available() > 0);
return compressor.decompress(baos.toByteArray());
}
public String decompress(byte[] data) throws IOException {
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
ByteArrayInputStream in = new ByteArrayInputStream(data);
try {
GZIPInputStream inflater = new GZIPInputStream(in);
byte[] byteBuffer = new byte[8192];
int r;
while((r = inflater.read(byteBuffer)) > 0 ) {
buffer.write(byteBuffer, 0, r);
}
} catch (IOException e) {
log.error("Could not decompress stream", e);
throw e;
}
return new String(buffer.toByteArray());
}
At first I thought there must be something wrong with the way that I am reading in the stream and I thought perhaps I am not looping properly. I then generated a ton of data to be streamed and checked that it was looping. Also the fact they it happens so seldom and so far has not been reproducable lead me to believe that it was the content rather than the scenario. But at this point I am totally baffled and for all I know it is the code.
Thanks again everyone.
Update 2:
As requested the .Net code:
Dim DataToCompress = Encoding.UTF8.GetBytes(Data)
Dim CompressedData = Compress(DataToCompress)
To get the raw data into bytes. And then it gets compressed
Private Function Compress(ByVal Data As Byte()) As Byte()
Try
Using MS = New MemoryStream()
Using Compression = New GZipStream(MS, CompressionMode.Compress)
Compression.Write(Data, 0, Data.Length)
Compression.Flush()
Compression.Close()
Return MS.ToArray()
End Using
End Using
Catch ex As Exception
Log.Error("Error trying to compress data", ex)
Throw
End Try
End Function
Update 3: Also added more java code. the in variable is the InputStream return from socket.getInputStream()

It certainly shouldn't be due to the data involved - the streams deal with binary data, so that shouldn't make any odds at all.
However, without seeing your code, it's hard to say for sure. My first port of call would be to check anywhere that you're using InputStream.read() - check that you're using the return value correctly, rather than assuming a single call to read() will fill the buffer.
If you could provide some code, that would help a lot...

I would suspect that for some reason the data is altered underway, by treating it as text, not as binary, so it may either be \n conversions or a codepage alteration.
How is the gzipped stream transferred between the two systems?

It is not pssible. EOF in TCP is delivered as an out of band FIN segment, not via the data.