Develop Reference

Sending Protocol Buffer encoded message from Python Server to Java Client

I'm writing a little server that uses protocol buffer to encode some data.
TCP Socket is opened between Android Client and Python Server
Android Client sends string for processing as normal newline delimited utf-8.
Python Server does some processing to generate a response, which gives an Array of Int Arrays: [[int]]. This is encoded in the protocol buffer file:
syntax = "proto2";
package tts;
message SentenceContainer {
repeated Sentence sentence = 1;
}
message Sentence {
repeated uint32 phonemeSymbol = 1;
}
It gets loaded into this structure and sent as follows...
container = ttsSentences_pb2.SentenceContainer()
for sentence in input_sentences:
phonemes = container.sentence.add()
# Add all the phonemes to the phoneme list
phonemes.phonemeSymbol.extend(processor.text_to_sequence(sentence))
payload = container.SerializeToString()
client.send(payload)
Android Client receives Protocol Buffer encoded message and tries to decode.
This is where I'm stuck...
# I get the InputStream when the TCP connection is first opened
bufferIn = socket.getInputStream();
TtsSentences.SentenceContainer sentences = TtsSentences.SentenceContainer.parseDelimitedFrom(bufferIn);
When receiving the message the client gets this exception:
E/TCP: Server Error
com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:164)
at com.google.protobuf.GeneratedMessageLite.parsePartialDelimitedFrom(GeneratedMessageLite.java:1527)
at com.google.protobuf.GeneratedMessageLite.parseDelimitedFrom(GeneratedMessageLite.java:1496)
at com.tensorspeech.tensorflowtts.TtsSentences$SentenceContainer.parseDelimitedFrom(TtsSentences.java:221)
at com.tensorspeech.tensorflowtts.network.PersistentTcpClient.run(PersistentTcpClient.java:100)
at com.tensorspeech.tensorflowtts.MainActivity.lambda$onCreate$0$MainActivity(MainActivity.java:71)
at com.tensorspeech.tensorflowtts.-$$Lambda$MainActivity$NTUE8bAusaoF3UGkWb7-Jt806BY.run(Unknown Source:2)
at java.lang.Thread.run(Thread.java:919)
I already know this problem is caused because Protocol buffer is not self delimiting, but I'm not sure how I'm supposed to properly delimit it. I've tried adding a newline client.send(payload + b'\n'), and adding in the PB size in bytes to the beginning of the payload client.send(container.ByteSize().to_bytes(2, 'little') + payload), but am not sure how to proceed.
It's a shame there's no documentation on how to use Protocol Buffer over TCP Sockets in Java...

OK, I worked this out...
In the case where you have a short-lived connection, the socket closing would signify the end of the payload, so no extra logic is required.
In my case, I have a long-lived connection, so closing the socket to signify the end of the payload wouldn't work.
With a Java Client & Server, you could get around this by using:
MessageLite.writeDelimitedTo(OutputStream)
then on the recipient side:
MessageLite.parseDelimitedFrom(InputStream).
Easy enough...
But in the Python API, there is no writeDelimitedTo() function. So instead we must recreate what writeDelimitedTo() is doing. Fortunately, it's simple. It simply adds a _VarintBytes equal to the payload size to the beginning of the message!
client, _ = socket.accept()
payload = your_PB_item.SerializeToString()
size = payload.ByteSize()
client.send(_VarintBytes(size) + payload)
Then on the Java recipient side...
bufferIn = socket.getInputStream();
yourPbItem message;
if ((message = yourPbItem.parseDelimitedFrom(bufferIn)) != null) {
// Do stuff :)
}
This way, your protocol buffer library knows exactly how many bytes to read, and then to stop caring about the InputStream, rather than sitting listening indefinitely.

How to detect end of Input Stream from Socket using BufferedInputStream?

I'm trying to read and input from a Socket in Java and then proccess it by calling a function. I must read by byte and not by text because reading as text give some erros because of the requests coming from client. So I decided to use the class BufferedInputStream. The problem is that the end of a request is not being detected. I guess that this is because the stream is not ending just being hanged or something...
According to the documentation, the read() function returns a byte read from the stream or -1 if the stream has ended. So, my code reads byte by byte, joins it in a string, and then, when the stream has ended, sends the string to be proccessed (or at least, was supposed to do this). Here is the code:
BufferedInputStream reader = new BufferedInputStream(socket.getInputStream());
int dt;
String cmd = "";
while( (dt = reader.read()) >= 0){
cmd += (char)dt == '\n' || (char)dt == '\r' ? ' ' : (char)dt; // add each byte read to the string
System.out.println(cmd);
}
processaInput(cmd);
The problem is that the function processaInput is never being called like if the program was stuck in the loop. But even more stranger is that is not stuck on the loop because the system.out.println stops being called after the stream has ended. By the way, this while is being ran in a Thread (run() function), don't know if this can mess up so I added this last info here. Anyway, what am I missing here? Thanks.
Edit: There is no Client side code because I'm using the app POSTMAN from google to test it, so it is just sending HTTP request to my program.

You are conflating 'end of request' with 'end of stream'.
'End of stream' on a socket means the peer has closed the connection. No end of stream, no peer close. If you plan to write a response back down this socket, it is therefore incorrect to try to read it to end of stream first.
You are trying to read a single request, in which case you need something in your application protocol to tell you when you have it all: lines; a length word prefix; a self-describing protocol like XML; STX/ETX; type-length-value; ...
In this case if the protocol is HTTP you need to implement the Content-length header: recognise it, read to the end of the headers, then read exactly that many bytes from the stream, e.g. with DataInputStream.readFully().

How do I recognize EOF in Java Sockets?

I want to recognize end of data stream in Java Sockets. When I run the code below, it just stuck and keeps running (it stucks at value 10).
I also want the program to download binary files, but the last byte is always distinct, so I don't know how to stop the while (pragmatically).
String host = "example.com";
String path = "/";
Socket connection = new Socket(host, 80);
PrintWriter out = new PrintWriter(connection.getOutputStream());
out.write("GET "+ path +" HTTP/1.1\r\nHost: "+ host +"\r\n\r\n");
out.flush();
int dataBuffer;
while ((dataBuffer = connection.getInputStream().read()) != -1)
System.out.println(dataBuffer);
out.close();
Thanks for any hints.

Actually your code is not correct.
In HTTP 1.0 each connection is closed and as a result the client could detect when an input has ended.
In HTTP 1.1 with persistent connections, the underlying TCP connection remains open, so a client can detect when an input ends with 1 of the following 2 ways:
1) The HTTP Server puts a Content-Length header indicating the size of the response. This can be used by the client to understand when the reponse has been fully read.
2)The response is send in Chunked-Encoding meaning that it comes in chunks prefixed with the size of each chunk. The client using this information can construct the response from the chunks received by the server.
You should be using an HTTP Client library since implementing a generic HTTP client is not trivial (at all I may say).
To be specific in your code posted you should have followed one of the above approaches.
Additionally you should read in lines, since HTTP is a line terminated protocol.
I.e. something like:
BufferedReader in =new BufferedReader(new InputStreamReader( Connection.getInputStream() ) );
String s=null;
while ( (s=in.readLine()) != null) {
//Read HTTP header
if (s.isEmpty()) break;//No more headers
}
}
By sending a Connection: close as suggested by khachik, gets the job done (since the closing of the connection helps detect the end of input) but the performance gets worse because for each request you start a new connection.
It depends of course on what you are trying to do (if you care or not)

You should use existing libraries for HTTP. See here.
Your code works as expected. The server doesn't close the connection, and dataBuffer never becomes -1. This happens because connections are kept alive in HTTP 1.1 by default. Use HTTP 1.0, or put Connection: close header in your request.
For example:
out.write("GET "+ path +" HTTP/1.1\r\nHost: "+ host +"\r\nConnection: close\r\n\r\n");
out.flush();
int dataBuffer;
while ((dataBuffer = connection.getInputStream().read()) != -1)
System.out.print((char)dataBuffer);
out.close();

Utilizing persistence with Apache HttpClient?

My goal is to connect to a server and then maintain the connection. The server keeps pushing me some data whenever it has any. I wrote the following but it works only the first time. The second time onwards, it gives me an exception saying that the get.getResponseBodyAsStream() is null. I was thinking that Apache's HTTPClient keeps the connection alive by default so what I understand is that I need a blocking call somewhere. Can someone help me out here?
GetMethod get = new GetMethod(url);
String nextLine;
String responseBody = "";
BufferedReader input;
try {
httpClient.executeMethod(get);
while(true) {
try {
input = new BufferedReader(new InputStreamReader(get.getResponseBodyAsStream()));
while ((nextLine = input.readLine()) != null)
responseBody += nextLine;
System.out.println(responseBody);
} catch (IOException ex) {
ex.printStackTrace();
}
}
} catch (Exception e) {
e.printStackTrace();
}
Actually at the end of the day, I am trying to get a persistent connection to the server (I will handle possible errors later) so that I can keep receiving updates from my server. Any pointers on this would be great.

I haven't looked in great detail or tested code, but I think repeatedly opening up a reader on the response is probably a bad idea. I'd take and move the input = line up outside the loop, for starters.

in my opinion HttpClient library is meant for client pull situations. i recommend you to look at comet which supports server push

You cannot do it like this. When you have read the "body" of the response, that is it. To get more information, the client has to send a new request. That is the way that the HTTP protocol works.
If you want to stream multiple chunks of data in a single HTTP response, then you are going to need to do the chunking and unchunking yourself. There a variety of approaches you could use, depending on the nature of the data. For example:
If the data is XML or JSON, send a stream of XML documents / JSON objects an have the receiver separate the stream into documents / objects before sending them to the parser.
Invent your own light-weight "packetization" where you precede each chunk with a start marker and a byte count.
The other alternative is to use multiple GET requests, but try to configure things so that the underlying TCP/IP connection stays open between requests; see HTTP Persistent Connections.
EDIT
Actually, I need to send only one GET request and keep waiting for status messages from the server.
The HTTP status code is transmitted in the first line of the HTTP response message. There can be only one per HTTP response, and (obviously) there can be only one response per HTTP request. Therefore what you are trying to do is impossible using normal HTTP status codes and request/reply messages.
Please review the alternatives that I suggested above. The bullet-pointed alternatives can be tweaked to allow you to include some kind of status in each chunk of data. And the last one (sending multiple requests) solves the problem already.
EDIT 2
To be more particular, it seems that keeping the connection alive is done transparently
That is correct.
... so all I need is a way to get notified when there is some data present that can be consumed.
Assuming that you are not prepared to send multiple GET requests (which is clearly the simplest solution!!!), then your code might look like this:
while (true) {
String header = input.readLine(); // format "status:linecount"
if (header == null) {
break;
}
String[] parts = header.split(":");
String status = parts[0];
StringBuilder sb = new StringBuilder();
int lineCount = Integer.parseInt(parts[1]);
for (int i = 0; i < lineCount; i++) {
String line = input.readLine();
if (line == null) {
throw new Exception("Ooops!");
}
sb.append(line).append('\n');
}
System.out.println("Got status = " + status + " body = " + body);
}
But if you are only sending status codes or if the rest of each data chunk can be shoe-horned onto the same line, you can simplify this further.
If you are trying to implement this so that your main thread doesn't have to wait (block) on reading from the input stream, then either use NIO, or use a separate thread to read from the input stream.

Java socket programming - stream get stuck

I am currently working on a simple proxy server, which receives http request from browser, process it, then forward it to the desire web server.
I try to get the request from the input stream of the socket connected by the browser, everything is fine except that the stream get stuck after receiving the last block of data.
My code is in fact very simple, as shown below:
ServerSocket servSocket = new ServerSocket(8282);
Socket workSocket = servSocket.accept();
InputStream inStream = workSocket.getInputStream();
byte[] buffer = new byte[1024];
int numberRead = 0;
while ((numberRead = inStream.read(buffer, 0, 1024)) != -1){
System.out.println(new String(buffer));
}
The loop simply cannot exit, even the request reception is finished.
Is there any method to workaround this problem?
Thanks in advance for any advice.

As in InputStream javadoc the method will block until the data is available or the EOF is encountered. So, the other side of Socket needs to close it - then the inStream.read() call will return.
Another method is to send the size of message you want to read first, so you know ahead how many bytes you have to read. Or you can use BufferedReader to read from socket in line-wise way. BufferedReader has a method readLine() which returns every time a line is read, which should work for you as HTTP protocol packages are nice divided into lines.

It will cycle until the connection is closed, and the client is probably waiting for HTTP response from you and doesn't close it.

The browser is waiting for a response before it closes the connection.
Your read-method on the other hand will block until the stream/connection is closed or new data is received.

Not a direct solution according to your current code.
As HTTP is a line based protocol, you might want to use a Buffered Reader and call readLine() on it.

The when a http request comes in it will always be concluded with a blank line, for example:
GET /someFile.html HTTP/1.1
Host: www.asdf.com
After sending that request the client connection will then wait for a response from the server before closing the connection. So if you want to parse the request from the user you are probably better off using a BufferedReader and reading full lines until you reach a lines of text that is blank line.