I'm having a socket problem. This problem occurs when I'm running the server and client on the same PC i.e. using "localhost" parameter. But problem is not seen when different PCs are being used.
Client sends a file with these codes:
output_local.write(buffer, 0, bytesRead);
output_local.flush();
And after that in another method I'm sending a command with these:
outputStream.write(string);
outputStream.flush();
Server appends the command to the end of the file. So it thinks it hasn't received the command from the client yet. do you have an idea what might causing this problem? How can I solve the defect? below is the file receive method at the server:
while (true) {
try {
bytesReceived = input.read(buffer);
} catch (IOException ex) {
Logger.getLogger(Server.class.getName()).log(Level.SEVERE, null, ex);
System.out.println("exception occured");
break;
}
System.out.println("received:" + bytesReceived);
try {
/* Write to the file */
wr.write(buffer, 0, bytesReceived);
} catch (IOException ex) {
Logger.getLogger(Server.class.getName()).log(Level.SEVERE, null, ex);
}
total_byte = total_byte + bytesReceived;
if (total_byte >= filesizeInt) {
break;
}
}
If you want message-like support, you need a create a protocol to clarify what you're going to send and receive.
In TCP, you can't rely on separate "packets" being received separately (e.g., sending 4 chunks of 10 bytes may be received as 1 chunk of 40, or of 2 chunks of 20, or one chunk of 39 and one chunk of 1). TCP guarantees in order delivery, but not any particular 'packetization' of your data.
So, for example, if you're sending a string you need to first send the string length then its bytes. The logic in pseudocode would be something like:
Client:
Send the command indicator
Send the payload length
Send the payload
Server:
Read the command indicator
Read the payload length
Loop reading payload until the complete length has been read
The defect is that you're treating a stream-based protocol (TCP) as if it were a message-oriented protocol. It's not. You should assume that this can happen.
If you need to break your stream into individual messages, you should use either delimiters or (preferably IMO) a length prefix for each message. You should also then anticipate that any read you issue may not receive as much data as you've asked for - in other words, not only can messages be combined if you're not careful, but they can easily be split.
I mentioned that I prefer length-prefixing to delimiters. Pros and cons:
The benefit of using a message delimiter is that you don't need to know the message size before you start sending.
The benefits of using a length prefix are:
The code for reading the message doesn't need to care about the data within the message at all - it only needs to know how long it is. You read the message length, you read the message data (looping round until you've read it all) and then you pass the message on for process. Simple.
You don't need to worry about "escaping" the delimiter if you want it to appear within a normal message.
As TCP is a stream oriented connection, this behaviour is normal if the writer writes faster than the reader reads, or than the TCP stack sends packets.
You should add a separator to separate the parts of the streams, e.g. by using a length field for sub packets, or by using separators such as newline (\n, char code 10).
Another option could be to use UDP (or even SCTP), but that depends on the task to be fulfilled.
Related
I am using Protocol Buffers for Swift (latest from CocoaPods) and Google's official Java Protocol buffer client (version 2.6.0) to to pass messages between a Java server (ServerSocket) and a Swift iOS app (GCDAsyncSocket).
Most messages (several hundred per second; I am streaming audio as float arrays, among other things) flow just fine. Occasionally, however, a message from client to server won't parse. The Java code throws a
com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero)
On both ends I am sending a 4-byte Big-Endian integer representing the number of bytes to follow, then the raw protobuf message. On both ends I am receiving the number of bytes to follow, blocking until I get that many bytes, and then attempting to parse.
There are no errors observed in the Java->Swift direction, only Swift->Java.
The vast majority of messages are fine. The problem appears to increase in frequency with the number of messages being processed.
In Java each client has a thread talking to it and a thread listening to it. The listener thread pulls messages off the wire and puts them into LinkedBlockingQueues per client. The talking thread pulls message off the LinkedBlockingQueue for that client, serializes them, and send them to that client's output stream.
// Take a messageBuilder, serialize and transmit it
func transmit(messageBuilder: Message_.Builder) {
do {
messageBuilder.src = self.networkID;
let data = try messageBuilder.build().data()
var dataLength = CFSwapInt32HostToBig(UInt32(data.length))
self.socket.writeData(NSData(bytes: &dataLength, length: 4), withTimeout: 1, tag: 0)
self.socket.writeData(data, withTimeout: 1, tag: 0)
} catch let error as NSError {
NSLog("Failed to transmit.")
NSLog(error.localizedDescription)
}
}
Java receiving side:
public void run() {
while (true) {
try {
byte[] lengthField = new byte[4];
try {
ghost.in.readFully(lengthField, 0, 4);
} catch (EOFException e) {
e.printStackTrace();
ghost.shutdown();
return;
}
Integer bytesToRead = ByteBuffer.wrap(lengthField).order(ByteOrder.BIG_ENDIAN).getInt();
byte[] wireMessage = new byte[bytesToRead];
in.readFully(wireMessage, 0, bytesToRead);
HauntMessaging.Message message = HauntMessaging.Message.parseFrom(wireMessage);
// do something with the message
} catch (IOException e) {
e.printStackTrace();
ghost.shutdown();
return;
}
}
}
Any ideas?
To debug protocol buffer messages:
capture the packet in Wireshark
right-click on the packet subsection that contains only the protobuf message and copy the hex stream
use a hex editor to save the hex stream to a file
protoc ‒‒decode_raw < file and match the output tags and data to the tags in your .proto file
Due to the exception message, Protocol message contained an invalid tag (zero), I suspect that Swift failed to build the protobuf message and sent an empty message.
Got it!
The two calls consecutive calls to socket.writeData were not necessarily atomic, but called from several threads. They were getting interleaved, so that first it wrote a length and then it wrote a different length (and/or somebody else's message).
Surrounding those two calls in a dispatch_async block for a DISPATCH_QUEUE_SERIAL fixed the problem.
I am trying to write a simple echo server using SSL. The first line that goes to the server is echoed exactly. When I send a second line, only the first character is echoed. The client works off of a buffered reader's read line from stdin. If I hit CR again the rest of the message comes through. The server seems to be sending all of the data. Here are output from client and server:
CLIENT:
Sending to server at 192.168.0.161
on port 9999
4 seasoNS
echo:4 seasoNS
are really good
echo:a
echo:re really good
SERVER:
server listening on 9999
has cr/lf
4 seasoNS
size to send: 10
has cr/lf
are really good
size to send: 16
exiting...
Here is the client loop:
try {
BufferedReader consoleBufferedReader = getConsoleReader();
sslsocket = getSecSocket(strAddress, port);
BufferedWriter sslBufferedWriter = getSslBufferedWriter(sslsocket);
InputStream srvrStream = sslsocket.getInputStream();
String outMsg;
while ((outMsg = consoleBufferedReader.readLine()) != null) {
byte[] srvrData = new byte[1024];
sslBufferedWriter.write(outMsg);
sslBufferedWriter.newLine();
sslBufferedWriter.flush();
int sz = srvrStream.read(srvrData);
String echoStr = new String(srvrData, 0, sz);
System.out.println("echo:" + echoStr);
}
} catch (Exception exception) {
exception.printStackTrace();
}
This problem seemed so odd that I was hoping there was something obvious that I was missing.
What you're seeing is perfectly normal.
The assumption you're making that you're going to read the whole buffer in one go is wrong:
int sz = srvrStream.read(srvrData);
Instead, you need to keep looping until you get the delimiter of your choice (possibly a new line in your case).
This applies to plain TCP connections as well as SSL/TLS connections in general. This is why application protocols must have delimiters or content length (for example, HTTP has a double new line to end its headers and uses Content-Length or chunked transfer encoding to tell the other party when the entity ends).
In practice, you might not see when your assumption doesn't work for such a small example.
However, the JSSE splits the records it sends into 1/n-1 on purpose to mitigate the BEAST attack. (OpenSSL would send 0/n.)
Hence, the problem is more immediately noticeable in this case.
Again, this is not an SSL/TLS or Java problem, the way to fix this is to treat the input you read as a stream and not to assume the size of buffers you read on one end will match the size of the buffers used to send that data from the other end.
using this code:
Java Server side:
...
out = new PrintWriter(this.client.getOutputStream(), true);
...
public void sendMsg(String msg) {
out.println(msg);
//out.flush(); // we don't flush manually because there is auto flush true
}
C# Client side:
while(connected) {
int lData = myStream.Read(myBuffer, 0, client.ReceiveBufferSize);
String myString = Encoding.UTF8.GetString(myBuffer);
myString = myString.Substring(0, lData);
myString = myString.Substring(0, myString.Length-2);
addToQueue(myString);
}
variable myString have many messages that server should send them one by one like
hello \r\t hello \r\t ...
they should come separately like
hello \r\t
hello \r\t ...
which means when i wait one by one they come instantly all of them in a row, how can i make it to send one by one in separate flush.
Note I send 30~ messages in a row in one second (1s), i want them separate.
TCP supports a stream of bytes. This means you have no control how the data arrives regardless of how you send it. (Other than it will comes as bytes) You should rethink your protocol if you depend on it coming in any particular manner.
You can reduce the amount of bunching of data but all this does is reduce latency at the cost of throughput and should never be relied upon. This can be reduce (but not eliminated) by turning off Nagle and reducing co-alessing setting in your TCP driver if you can change these.
i want them separate.
You can want it but TCP does not support messages as you would want them.
The solution in you case is for your reader to match your writers protocol. You send lines so you should read lines at a time, e.g. BufferedReader.readLine(), not blocks of whatever data happens to be in the buffer.
I have a Java Client which sends UTF-8 strings to a C# TCP-Server, I'm using a DataOutputStream to send the strings. The code looks like this:
public void sendUTF8String(String ar) {
if (socket.isConnected()) {
try {
dataOutputStream.write(ar.getBytes(Charset.forName("UTF-8")));
dataOutputStream.flush();
} catch (IOException e) {
handleException(e);
}
}
}
The problem is that flush doesn't seem to work right. If I send two Strings close to each other, the server receives only one message with both strings. The whole thing works if I do a Thread.sleep(1000) between calls, this is obviously not a solution.
What am I missing?
flush() doesn't guarantee that a data packet gets shipped off. Your TCP/IP stack is free to bundle your data for maximum efficiency. Worse, there are probably a bunch of other TCP/IP stacks between you and your destination, and they are free to do the same.
I think you shouldn't rely on packet bundling. Insert a logical terminator/divider in your data and you will be on the safe side.
You shouldn't worry about how the data is broken up into packets.
You should include the length of the string in your messages, and then on the receiving end you would read the length first. So for example to send you would do
byte[] arbytes = ar.getBytes(Charset.forName("UTF-8"));
output.writeInt(arbytes.length)
output.write(arbytes)
and then in your reader you do
byte[] arbytes = new byte[input.readInt()];
for(int i = 0; i < len; i++){
arbytes[i] = input.read();
}
//convert bytes back to string.
You can't just call input.read(arbytes) because the read function doesn't necessarily read the entire length of the array. You can do a loop where you read a chunk at a time but the code for that is a bit more complex.
Anyway, you get the idea.
Also, if you really want to control what goes in what packets, you can use Datagram Sockets, but if you do that then delivery of the packet is not guaranteed.
Socket send a stream of data, not messages.
You shouldn't rely on the packets you receive being the same size as they are sent.
Packets can be grouped together as you have seen but they can also be broken up.
Use #Chad Okere's suggestion on how to ensure you get blocks the same was they are sent.
However in your case, you can just use
dataOutputStream.writeUTF(ar); // sends a string as UTF-8
and
String text = dataInputStream.readUTF(); // reads a string as UTF-8
I have the following Java socket client app, that sends same string to socket server:
import java.net.*;
import java.io.*;
public class ServerClient {
public static void main(String[] args) throws IOException {
System.out.println("Starting a socket server client...");
Socket client = new Socket("XXX.X.XXX.XX", 12001);
BufferedOutputStream stream = new BufferedOutputStream(client.getOutputStream());
String message = "ABC";
BufferedReader inputReader = new BufferedReader(new InputStreamReader(System.in));
String input = null;
while ( true ) {
System.out.print("Would you like to send a message to Server? ");
input = inputReader.readLine();
if ( !input.equals("Y") ) break;
System.out.println("Message to send: " + message);
System.out.println("Message length is: " + message.length());
byte[] messageBytes = message.getBytes("US-ASCII");
stream.write(messageBytes, 0, messageBytes.length);
stream.flush();
}
System.out.println("Shutting down socket server client...");
stream.close();
client.close();
inputReader.close();
}
}
The first time message is sent, server receives the message; however, every subsequent time I'm trying to send this message, server is not receiving anything. Message simply disappears. I am writing to the socket successfully (no exceptions) but nothing is coming on the other side of the pipe (or so I'm told).
I do not have access to the server app, logs or code, so I'm wondering if there is any approach you can recommend to figure out why server is not receiving subsequent messages. Any ideas would be greatly appreciated!
Clarification:
New lines are not expected by the server; otherwise, how would it even receive message the first time? As a trial and error, I did try sending '\n' and "\r\n" and 0x00 characters at the end of the string - all without any luck.
I thought flushing was an issue, so I tried various outputstream classes (PrintStream, PrintWriter, FilterOutputStream), but was still running into same exact issues. Then, if "flushing" is an issue, how is it working the first time?
Other tests:
1 - use a network sniffer to see what is realy hapening on the network
2 - use some program like TCP Test Tool to send data to the server and simulate your program. (netcat can also be used, but it sends a newline after each line)
Remember:
TCP is stream oriented. not message oriented.
One write on the client could take several reads on the server to .. read
Multiple writes on the client could get read by the server in one read
You'll hardly see the above scenarios in a test application on a local network, you will see them very quick in a production environemnt, or when you start to really speed up the sending/receiving.
Following this, if you are sending messages you need a delimiter, or some other way of indicating 'here's one message', e.g. defining the protocol to be 'the first byte is the length of the following message'.
And you'd need to check the receiving end wether it read a partial message, a whole message, and any combination thereof (e.e.g one read might have read 3 and a half message..).
A quick solution for your test app, write lines. That is, a string followed by a newline character. A bufferedreader's ReadLine() could then take care of the reassembly for you on the receiving end.
It works correctly here... but I am missing a carriage return or some other end of message after sending the message.
Hard to write more without knowing what the server expects (protocol)...
Maybe you should try something like
String message = "ABC\n";