Sending Protocol Buffer encoded message from Python Server to Java Client - java

I'm writing a little server that uses protocol buffer to encode some data.
TCP Socket is opened between Android Client and Python Server
Android Client sends string for processing as normal newline delimited utf-8.
Python Server does some processing to generate a response, which gives an Array of Int Arrays: [[int]]. This is encoded in the protocol buffer file:
syntax = "proto2";
package tts;
message SentenceContainer {
repeated Sentence sentence = 1;
}
message Sentence {
repeated uint32 phonemeSymbol = 1;
}
It gets loaded into this structure and sent as follows...
container = ttsSentences_pb2.SentenceContainer()
for sentence in input_sentences:
phonemes = container.sentence.add()
# Add all the phonemes to the phoneme list
phonemes.phonemeSymbol.extend(processor.text_to_sequence(sentence))
payload = container.SerializeToString()
client.send(payload)
Android Client receives Protocol Buffer encoded message and tries to decode.
This is where I'm stuck...
# I get the InputStream when the TCP connection is first opened
bufferIn = socket.getInputStream();
TtsSentences.SentenceContainer sentences = TtsSentences.SentenceContainer.parseDelimitedFrom(bufferIn);
When receiving the message the client gets this exception:
E/TCP: Server Error
com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:164)
at com.google.protobuf.GeneratedMessageLite.parsePartialDelimitedFrom(GeneratedMessageLite.java:1527)
at com.google.protobuf.GeneratedMessageLite.parseDelimitedFrom(GeneratedMessageLite.java:1496)
at com.tensorspeech.tensorflowtts.TtsSentences$SentenceContainer.parseDelimitedFrom(TtsSentences.java:221)
at com.tensorspeech.tensorflowtts.network.PersistentTcpClient.run(PersistentTcpClient.java:100)
at com.tensorspeech.tensorflowtts.MainActivity.lambda$onCreate$0$MainActivity(MainActivity.java:71)
at com.tensorspeech.tensorflowtts.-$$Lambda$MainActivity$NTUE8bAusaoF3UGkWb7-Jt806BY.run(Unknown Source:2)
at java.lang.Thread.run(Thread.java:919)
I already know this problem is caused because Protocol buffer is not self delimiting, but I'm not sure how I'm supposed to properly delimit it. I've tried adding a newline client.send(payload + b'\n'), and adding in the PB size in bytes to the beginning of the payload client.send(container.ByteSize().to_bytes(2, 'little') + payload), but am not sure how to proceed.
It's a shame there's no documentation on how to use Protocol Buffer over TCP Sockets in Java...

OK, I worked this out...
In the case where you have a short-lived connection, the socket closing would signify the end of the payload, so no extra logic is required.
In my case, I have a long-lived connection, so closing the socket to signify the end of the payload wouldn't work.
With a Java Client & Server, you could get around this by using:
MessageLite.writeDelimitedTo(OutputStream)
then on the recipient side:
MessageLite.parseDelimitedFrom(InputStream).
Easy enough...
But in the Python API, there is no writeDelimitedTo() function. So instead we must recreate what writeDelimitedTo() is doing. Fortunately, it's simple. It simply adds a _VarintBytes equal to the payload size to the beginning of the message!
client, _ = socket.accept()
payload = your_PB_item.SerializeToString()
size = payload.ByteSize()
client.send(_VarintBytes(size) + payload)
Then on the Java recipient side...
bufferIn = socket.getInputStream();
yourPbItem message;
if ((message = yourPbItem.parseDelimitedFrom(bufferIn)) != null) {
// Do stuff :)
}
This way, your protocol buffer library knows exactly how many bytes to read, and then to stop caring about the InputStream, rather than sitting listening indefinitely.

Related

Server isn't forwarding the message to the client

I'm a novice in Java socket communication, I'm trying to connect 3 clients to a server without the use of threads as the number of clients connected to the server will as be 3. I have written a condition in the server to tokenize the received input from the client and send write to the respective stream. But the message isn't reaching the server so I'm not sure whether the client isn't writing to the stream or the server is unable to receive the data. I'm trying to establish a cryptographic protocol between the connected clients for demonstration purposes.
I was using BufferedReader and PrintStream earlier since I was testing with keyboard input, then I changed to data streams still doesn't seem to work
Here my code at Server.java to forward messages, the control doesn't appear to come to this loop at all(not sure).
while(true){
String recvd=cin2.readLine();
System.out.println("At server: "+recvd);
StringTokenizer st = new StringTokenizer(recvd);
String MsgToSend=st.nextToken();
String recipient=st.nextToken();
if(recipient.equalsIgnoreCase("client1")){
cout2.writeUTF(MsgToSend);
}
else if(recipient.equalsIgnoreCase("client2")){
cout.writeUTF(MsgToSend);
}
else if(recipient.equalsIgnoreCase("client3")){
cout3.writeUTF(MsgToSend);
}
}
Here is my client-side code,
while(i==0){
String s1="A";
String s2="B";
String s3="client1";
String toSend=String.join(" ",s1+s2,s3);
System.out.println("toSend :"+toSend);
sout.writeUTF(toSend);
sout.flush();
i++;
}
Receiving Client,
while (true){
s=sin.readLine();
System.out.print("Server : "+s+"\n");
}
Not sure whether the client is unable to write or the server is unable to read. Any suggestions or solutions to correct the code?
The socket navigation logic in your server file seems to be strange as you are manually creating a string with keyword client1 and then tokenizing it for navigation. By this approach your flow would get static and establishment of a protocol gets difficult.
Please see:
You are sending the data to client 1 by forming the string AB client1, then your server file's first line is using client 2 socket reading. (Sending data from client1 socket to client1 socket, not right).
Please change that to client1 socket reading, use prinln() and readline() to send and read socket data and then check if your flow is fine!
Also, for the protocol flow, you have to send the data to the other two sockets and not to the one that is sending.
For eg: if you are sending the data from client 1 to client 2, as per your approach, you have to for a string AB client2 and send to the server, tokenize it and navigate the flow to client 2.
Hope this helps!
Thanks,
Areed

Odd behavior reading SSL socket Java

I am trying to write a simple echo server using SSL. The first line that goes to the server is echoed exactly. When I send a second line, only the first character is echoed. The client works off of a buffered reader's read line from stdin. If I hit CR again the rest of the message comes through. The server seems to be sending all of the data. Here are output from client and server:
CLIENT:
Sending to server at 192.168.0.161
on port 9999
4 seasoNS
echo:4 seasoNS
are really good
echo:a
echo:re really good
SERVER:
server listening on 9999
has cr/lf
4 seasoNS
size to send: 10
has cr/lf
are really good
size to send: 16
exiting...
Here is the client loop:
try {
BufferedReader consoleBufferedReader = getConsoleReader();
sslsocket = getSecSocket(strAddress, port);
BufferedWriter sslBufferedWriter = getSslBufferedWriter(sslsocket);
InputStream srvrStream = sslsocket.getInputStream();
String outMsg;
while ((outMsg = consoleBufferedReader.readLine()) != null) {
byte[] srvrData = new byte[1024];
sslBufferedWriter.write(outMsg);
sslBufferedWriter.newLine();
sslBufferedWriter.flush();
int sz = srvrStream.read(srvrData);
String echoStr = new String(srvrData, 0, sz);
System.out.println("echo:" + echoStr);
}
} catch (Exception exception) {
exception.printStackTrace();
}
This problem seemed so odd that I was hoping there was something obvious that I was missing.
What you're seeing is perfectly normal.
The assumption you're making that you're going to read the whole buffer in one go is wrong:
int sz = srvrStream.read(srvrData);
Instead, you need to keep looping until you get the delimiter of your choice (possibly a new line in your case).
This applies to plain TCP connections as well as SSL/TLS connections in general. This is why application protocols must have delimiters or content length (for example, HTTP has a double new line to end its headers and uses Content-Length or chunked transfer encoding to tell the other party when the entity ends).
In practice, you might not see when your assumption doesn't work for such a small example.
However, the JSSE splits the records it sends into 1/n-1 on purpose to mitigate the BEAST attack. (OpenSSL would send 0/n.)
Hence, the problem is more immediately noticeable in this case.
Again, this is not an SSL/TLS or Java problem, the way to fix this is to treat the input you read as a stream and not to assume the size of buffers you read on one end will match the size of the buffers used to send that data from the other end.

Java server Socket, reading content from DataInputStream

So I have set up a basic client/server connection and I am trying to send a message to one another on connection, I got the client to receive the message from the server, but the server doesn't recieve the clients message. Here is my current code for reading the sent data from the client:
ServerThread.socket = new ServerSocket(5556);
Socket client = ServerThread.socket.accept();
DataInputStream in = new DataInputStream
(
new BufferedInputStream(client.getInputStream())
);
String s = in.readUTF();
System.out.println("Client: " + s);
Using that it doesn't print out anything, Not even just 'Client: '
Here is my code for my client connection, and sending the message. Note: I wrote this part in VB:
client = New TcpClient()
client.Connect("myiphere", 5556)
Dim stream As NetworkStream = client.GetStream()
Dim sendBytes As [Byte]() = Encoding.ASCII.GetBytes("Hello server")
stream.Write(sendBytes, 0, sendBytes.Length)
Is there any reason why the data isn't being recieved? Or why it is being delayed? I have tried surronding the Java portion of the code with a try catch block but no error is emitted.
Any help will be appreciated.
UTFs in a DataInputStream are prepended with 0 and the length.
I haven't done much with VB, so I don't know if there are other errors, but try:
stream.Write(0, sendBytes.Length, sendBytes)
I shouldn't suggest code in a language I don't know. If you want to read it with readUTF, you'll have to send a 0 byte and a byte equal to the length of the string before you send your text bytes.
Edit:
You really might not want to use DataInputStream at all, though. It's intended for storing binary streams. If you're receiving text, try this on the Java side:
BufferedReader in = new BufferedReader(
new InputStreamReader(
client.getInputStream()
)
);
String s = in.readLine();
If you're not sending text, just create a BufferedInputStream as you did and read the bytes off of it.
As maybeWeCouldStealAVan pointed out, readUTF expects two bytes indicating how many more bytes of content there are. See http://docs.oracle.com/javase/6/docs/api/java/io/DataInput.html#readUTF() for details.
However, his/her solution using InputStreamReader doesn't work because InputStreamReader is expecting UTF-16 input (two bytes per character), but your VB client is sending ascii. I would suggest making your VB client send UTF-16 if you can (then using maybeWeCouldStealAVan's java code). If you can't do that (sorry, I don't know what encodings VB allows), then just write the extra two bytes needed to make readUTF work.

How do I recognize EOF in Java Sockets?

I want to recognize end of data stream in Java Sockets. When I run the code below, it just stuck and keeps running (it stucks at value 10).
I also want the program to download binary files, but the last byte is always distinct, so I don't know how to stop the while (pragmatically).
String host = "example.com";
String path = "/";
Socket connection = new Socket(host, 80);
PrintWriter out = new PrintWriter(connection.getOutputStream());
out.write("GET "+ path +" HTTP/1.1\r\nHost: "+ host +"\r\n\r\n");
out.flush();
int dataBuffer;
while ((dataBuffer = connection.getInputStream().read()) != -1)
System.out.println(dataBuffer);
out.close();
Thanks for any hints.
Actually your code is not correct.
In HTTP 1.0 each connection is closed and as a result the client could detect when an input has ended.
In HTTP 1.1 with persistent connections, the underlying TCP connection remains open, so a client can detect when an input ends with 1 of the following 2 ways:
1) The HTTP Server puts a Content-Length header indicating the size of the response. This can be used by the client to understand when the reponse has been fully read.
2)The response is send in Chunked-Encoding meaning that it comes in chunks prefixed with the size of each chunk. The client using this information can construct the response from the chunks received by the server.
You should be using an HTTP Client library since implementing a generic HTTP client is not trivial (at all I may say).
To be specific in your code posted you should have followed one of the above approaches.
Additionally you should read in lines, since HTTP is a line terminated protocol.
I.e. something like:
BufferedReader in =new BufferedReader(new InputStreamReader( Connection.getInputStream() ) );
String s=null;
while ( (s=in.readLine()) != null) {
//Read HTTP header
if (s.isEmpty()) break;//No more headers
}
}
By sending a Connection: close as suggested by khachik, gets the job done (since the closing of the connection helps detect the end of input) but the performance gets worse because for each request you start a new connection.
It depends of course on what you are trying to do (if you care or not)
You should use existing libraries for HTTP. See here.
Your code works as expected. The server doesn't close the connection, and dataBuffer never becomes -1. This happens because connections are kept alive in HTTP 1.1 by default. Use HTTP 1.0, or put Connection: close header in your request.
For example:
out.write("GET "+ path +" HTTP/1.1\r\nHost: "+ host +"\r\nConnection: close\r\n\r\n");
out.flush();
int dataBuffer;
while ((dataBuffer = connection.getInputStream().read()) != -1)
System.out.print((char)dataBuffer);
out.close();

Server not receiving bytes written to a socket by Java app

I have the following Java socket client app, that sends same string to socket server:
import java.net.*;
import java.io.*;
public class ServerClient {
public static void main(String[] args) throws IOException {
System.out.println("Starting a socket server client...");
Socket client = new Socket("XXX.X.XXX.XX", 12001);
BufferedOutputStream stream = new BufferedOutputStream(client.getOutputStream());
String message = "ABC";
BufferedReader inputReader = new BufferedReader(new InputStreamReader(System.in));
String input = null;
while ( true ) {
System.out.print("Would you like to send a message to Server? ");
input = inputReader.readLine();
if ( !input.equals("Y") ) break;
System.out.println("Message to send: " + message);
System.out.println("Message length is: " + message.length());
byte[] messageBytes = message.getBytes("US-ASCII");
stream.write(messageBytes, 0, messageBytes.length);
stream.flush();
}
System.out.println("Shutting down socket server client...");
stream.close();
client.close();
inputReader.close();
}
}
The first time message is sent, server receives the message; however, every subsequent time I'm trying to send this message, server is not receiving anything. Message simply disappears. I am writing to the socket successfully (no exceptions) but nothing is coming on the other side of the pipe (or so I'm told).
I do not have access to the server app, logs or code, so I'm wondering if there is any approach you can recommend to figure out why server is not receiving subsequent messages. Any ideas would be greatly appreciated!
Clarification:
New lines are not expected by the server; otherwise, how would it even receive message the first time? As a trial and error, I did try sending '\n' and "\r\n" and 0x00 characters at the end of the string - all without any luck.
I thought flushing was an issue, so I tried various outputstream classes (PrintStream, PrintWriter, FilterOutputStream), but was still running into same exact issues. Then, if "flushing" is an issue, how is it working the first time?
Other tests:
1 - use a network sniffer to see what is realy hapening on the network
2 - use some program like TCP Test Tool to send data to the server and simulate your program. (netcat can also be used, but it sends a newline after each line)
Remember:
TCP is stream oriented. not message oriented.
One write on the client could take several reads on the server to .. read
Multiple writes on the client could get read by the server in one read
You'll hardly see the above scenarios in a test application on a local network, you will see them very quick in a production environemnt, or when you start to really speed up the sending/receiving.
Following this, if you are sending messages you need a delimiter, or some other way of indicating 'here's one message', e.g. defining the protocol to be 'the first byte is the length of the following message'.
And you'd need to check the receiving end wether it read a partial message, a whole message, and any combination thereof (e.e.g one read might have read 3 and a half message..).
A quick solution for your test app, write lines. That is, a string followed by a newline character. A bufferedreader's ReadLine() could then take care of the reassembly for you on the receiving end.
It works correctly here... but I am missing a carriage return or some other end of message after sending the message.
Hard to write more without knowing what the server expects (protocol)...
Maybe you should try something like
String message = "ABC\n";

Categories

Resources