Im working on Instant messenger using java 1.6. IM uses multithreading - main thread, receiving, and ping. For tcp/ip communication I used SocketChannel. And it seems there is a problem with receiving bigger packages from server. Server instead of one sends a couple of packages and thats where the problem begins. Every first 8 bytes is telling what is the type of package and how big it is. This is how I managed reading:
public void run(){
while(true){
try{
Headbuffer.clear();
bytes = readChannel.read(Headbuffer); //ReadableByteChannel
Headbuffer.flip();
if(bytes != -1){
int head = Headbuffer.getInt();
int size = Headbuffer.getInt();
System.out.println("received pkg: 0x" + Integer.toHexString(head)+" with size "+ size+" bytes);
switch(head){
case incoming.Pkg1: ReadWelcome(); break;
case incoming.Pkg2: ReadLoginFail();break;
case incoming.Pkg3: ReadLoginOk();break;
case incoming.Pkg4: ReadUserList();break;
case incoming.Pkg5: ReadUserData();break;
case incoming.Pkg6: ReadMessage();break;
case incoming.Pkg7: ReadTypingNotify();break;
case incoming.Pkg8: ReadListStatus();break;
case incoming.Pkg9: ChangeStatus();break;
}
}
}catch(Exception e){
e.printStackTrace();
}
}
}
And during the tests everything was fine until i logged on my account and import my buddylist. I send request to server for statuses and he send me back about 10 out of 80 contacts. So I came up with something like this:
public synchronized void readInStatus(ByteBuffer headBuffer){
byteArray.add(headBuffer); //Store every buffer in ArrayList
int buddies = MainController.controler.getContacts().getSize();
while(buddies>0){
readStuff();
readDescription();
--buddies;
}
}
and each readStuff() and readDescription() are checking each parameter size with remaining bytes in the buffer:
if(byteArray.get(current).remaining() >= 4){
uin = byteArray.get(current).getInt();
}else{
byteArray.add(Receiver.receiver.read());
current = current +1;
uin = byteArray.get(current).getInt();
}
and Receiver.receiver.read() is:
public ByteBuffer read(){
try {
ByteBuffer bb = ByteBuffer.allocate(40000);
bb.order(ByteOrder.LITTLE_ENDIAN);
bytes = readChannel.read(bb);
bb.flip();
return bb;
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
So application is lunched, logged and then sends contacts. Server send me back just a piece of my list. But in the method readInStatus(ByteBuffer headBuffer) I try to force the rest of the list. And now the fun part - after some time it gets to the Receiver.receiver.read() and on bytes = readChannel.read(bb) it just stops and I dont know why , no errors no nothing even after some time and Im out of the ideas. Im fighting with this whole week and i dont get anywhere near the solution. I will appreciate any suggestions. Thanks.
Thanks for response. Yes, I'm using blocking SocketChannel, I tried non-blocking but it goes wild and out of control so I skipped the idea. About the bytes I expect - this is kind of weird, because its giving me size only once in head but its size of the first part not the whole package, other parts is not containing header bytes at all. I can't predict how much bytes it would be, the reason is - descriptions with 255 bytes capacity. This is exactly why I've created variable buddies in : public synchronized void readInStatus(ByteBuffer headBuffer)
wich is basically length of my buddy list and before reading each field I'm checking if there is enough bytes left if its not, I do read(). But last field before description is integer with the length of the incoming description. But its impossible to determine how long package is, until some processing is done. #robert do you think I should try again switching to non-blocking SocketChannel in that situation ?
The problem is most likely that you are sending fewer bytes than you are trying to read. You might have missed writing something, written things in the wrong order, misread a size field or something like that.
I think I'd attack this problem by adding tracing code to count and log the number of bytes read and written, notional packect sizes and so on. Then run, and compare the traces to see where things start to get out of sync.
If you are using a blocking SocketChannel, then read will block until the buffer is filled or the server delivers end of stream. For a server with connection keep-alive, the server does not send end of stream - it will simply stop sending data, and the read will hang indefinitely or until timeout.
You could either:
(i) try using a non-blocking SocketChannel, repeatedly reading until the read delivers 0 bytes (but beware 0 bytes does not necessarily mean end of stream - it could mean an interruption) or
(ii) if you have to use the blocking version, and you know how many bytes you were expecting from the server e.g. from a header, when the number of bytes left to read is less than buffer.capacity(), move position and/or limit on the buffer so as to leave only the required space in the buffer before the read. I am working this solution now. If it works for you, please let me know!
So far as I can work out, if you have to use a blocking SocketChannel and you do not know how many bytes you are expecting, and the server does not send end of stream, there is no solution.
Related
I'm developing a client (Java)/server(C++) application using TCP sockets.
The protocol I used is composed of Messages beginning by 2 bytes defining the type of what will be the content of the Message.
So basically, the receiving thread waits for data to be received in a loop. But I want to use a timeout with the socket to be notified that the other host takes too long to send data.
receivingSocket.setSoTimeout(durationInMilliseconds);
DataInputStream in = new DataInputStream(receivingSocket.getInputStream());
boolean success = false;
short value = 0;
do {
try {
value = in.readShort();// will throw a SocketTimeoutException in case of timeout, without 2 bytes available from the socket
success = true;
} catch (SocketTimeoutException e) {
/// do something if it happens to often. Otherwise go on with the loop
}
} catch (IOException e) {
/// abort connection in case of other problem
}
} while (!success)
Now, what happens if the receiving thread calls in.readShort() at a point where the socket has got only one byte available in its buffer ? Does this byte remain on the socket's stack ? Or is it lost ? In the first case, I could read it next time I call in.readShort(), otherwise it seems lost for good...
readShort() here is an example, my question stands also for readInt(), ...
Thanks for your help,
It isn't specified. I believe the way the implementation works is that the half data is lost, but in any case there's nothing written that says anything else, so you just have to assume the worst.
However in practice this is very unlikely to happen, provided you observe common sense at the sender.
I have the following Bluetooth Server class that extends a Thread to listen for incoming stream:
class SeverThread extends Thread{
bytes[] inBuffer = new bytes[1024];
int bytes;
while(true){
try{
bytes = inputStream.read(inBuffer);
}catch(IOException e){
// handle exception
}
System.out.println(new String(inBuffer));
}
}
The problem with this implementation:
When client first sends "Sanjeev" , output is "Sanjeev". Cool. After that, the client sends smk, the output now is smkjeev. Note that first 3 letters - san - have become smk. Now when the client sends hell, the output is helleev. If the client again sends "Sanjeev" then output is "Sanjeev" , because length of received string is same as the used space inside the buffer.
This is the expected behavior based on above code. But it's an unintended bug. I want the outputs to be Sanjeev , smk and hello respectively. The problem is apparent. The same buffer in memory is being used to take all the incoming words.
Any suggestions on how to fix this? Possible approaches:
Clear or refresh the buffer for every loop inside while(true). I don't know how to do this.
Move the declaration of inBuffer inside the while loop. But this will continuously allocate memory, a big wastage.
Thanks.
How about this ?
bytes = inputStream.read(inBuffer);
inBuffer[bytes] = '\0';
I have developed a client-server chat using the Sockets and it works great, but when I try to transmit data with Deflate compression it doesn't work: the output is "empty" (actually it's not empty, but I'll explain below).
The compression/decompression part is 100% working (I have already tested it), so the problem must be elsewhere in the transmission/receiving part.
I send the message from the client to the server using these methods:
// streamOut is an instance of DataOutputStream
// message is a String
if (zip) { // zip is a boolean variable: true means that compression is active
streamOut.write(Zip.compress(message)); // Zip.compress(String) returns a byte[] array of the compressed "message"
} else {
// if compression isn't active, the client sends the not compressed message to the server (and this works great)
streamOut.writeUTF(message);
}
streamOut.flush();
And I receive the message from the client to the server using these other methods:
// streamIn is an instace of DataInputStream
if (server.zip) { // same as before: true = compression is active
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[512];
int n;
while ((n = streamIn.read(buf)) > 0) {
bos.write(buf, 0, n);
}
byte[] output = bos.toByteArray();
System.out.println("output: " + Zip.decompress(output)); // Zip.decompress(byte[]) returns a String of decompressed byte[] array received
} else {
System.out.println("output: " + streamIn.readUTF()); // this works great
}
Debugging a little bit my program, I've discovered that the while loop never ends, so:
byte[] output = bos.toByteArray();
System.out.println("output: " + Zip.decompress(output));
is never called.
If I put those 2 lines of code in the while loop (after bos.write()), then all works fine (it prints the message sent from the client)! But I don't think that's the solution, because the byte[] array received may vary in size. Because of this I assumed that the problem is in the receiving part (the client is actually able to send data).
So my problem became the while loop in the receiving part. I tried with:
while ((n = streamIn.read(buf)) != -1) {
and even with the condition != 0, but it's the same as before: the loop never ends, so the output part is never called.
-1 is only returned when the socket is closed or broken. You could close the socket after sending your zipped content, and your code would start working. But I suspect you want to keep the socket open for more (future) chat messages. So you need some other way of letting the client know when a discrete message has been fully transmitted. Like Patrick suggested, you could transmit the message length before each zipped payload.
You might be able to leverage something in the deflate format itself, though. I think it has a last-block-in-stream marker. If you're using java.util.zip.Inflater have a look at Inflater.finished().
The read function will not return a -1 until the stream is closed. What you can do is calculate the number of bytes that should be sent from the server to the client, and then read that number of bytes on the client side.
Calculating the number of bytes is as easy as sending the length of the byte array returned from the Zip.compress function before the actual message, and then use the readInt function to get that number.
Using this algorithm makes sure that you read the correct number of bytes before decompressing, so even if the client actually reads 0 bytes it will continue to read until it receives all bytes it wants. You can do a streamIn.read(buf, 0, Math.min(bytesLeft, buf.length)) to only read as many bytes you want.
Your problem is the way you are working with stream. You must send some meta-data so your client know what to expect as data. Idealy you are creating a protocol/state machine to read the stream. For your example, as a quick and dirt solution, send something like data size or a termination sequence or something.
Example of solution:
Server: send the "data size" before the compressed data
Client: wait for the "data size" bytes. Now loop till read is equal or greater "data size" value. Something like:
while( streamIn.ready() && dataRead < dataExpected)
{
dataRead += streamIn.read(buf);
}
Of course you need to read the dataExpected before, with a similar code.
Tip: You could also use UDP if you dont mind having the possibility to lose data. Its easier to program with datagrams...
I'm attempting to send a .proto message from an iPhone application to a Java server via a socket connection. However so far I'm running into an issue when it comes to the server receiving the data; it only seems to process it after the client connection has been terminated. This points to me that the data is getting sent, but the server is keeping its inputstream open and waiting for more data. Would anyone know how I might go about solving this? The current code (or at least the relevant parts) is as follows:
iPhone:
Person *person = [[[[Person builder] setId:1] setName:#"Bob"] build];
RequestWrapper *request = [[[RequestWrapper builder] setPerson:person] build];
NSData *data = [request data];
AsyncSocket *socket = [[AsyncSocket alloc] initWithDelegate:self];
if (![socket connectToHost:#"192.168.0.6" onPort:6666 error:nil]){
[self updateLabel:#"Problem connecting to socket!"];
} else {
[self updateLabel:#"Sending data to server..."];
[socket writeData:data withTimeout:-1 tag:0];
[self updateLabel:#"Data sent, disconnecting"];
//[socket disconnect];
}
Java:
try {
RequestWrapper wrapper = RequestWrapper.parseFrom(socket.getInputStream());
Person person = wrapper.getPerson();
if (person != null) {
System.out.println("Persons name is " + person.getName());
socket.close();
}
On running this, it seems to hang on the line where the RequestWrapper is processing the inputStream. I did try replacing the socket writedata method with:
[request writeToOutputStream:[socket getCFWriteStream]];
(here I'm calling the gpb to write to the output stream, instead of writing the data generated to the output stream)
Which I thought might work, however I get an error claiming that the "Protocol message contained an invalid tag (zero)". I'm fairly certain that it doesn't contain an invalid tag as the message works when sending it via the writedata method.
Any help on the matter would be greatly appreciated!
Cheers!
Dan
(EDIT: I should mention, I am using the metasyntactic gpb code; and the cocoaasyncsocket implementation)
Maybe you need to flush the output socket on the iPhone side, its possible that the data is sitting in an OS/library buffer and is not being written until the connection is closed (which causes an implicit flush).
EDIT
It looks like the api doesn't support flushing (i guess flushing isn't a very async thing to do) but you can subscribe to the didWriteDataWithTag event: from the headers
/**
* Called when a socket has completed writing the requested data. Not called if there is an error.
**/
- (void)onSocket:(AsyncSocket *)sock didWriteDataWithTag:(long)tag;
I would subscribe to this event and then in the event handler call
[self updateLabel:#"Data sent, disconnecting"];
[socket disconnect];
this way you only show the Data Sent label when it is actually sent.
(in the interest of full disclosure i have no idea how to program in objective-c :) good luck)
Finally I managed to solve it!! It's embarrassing how simple it was once I stopped caring about the GPB way of doing it. The C++ code I used is:
NSData *data = [wrapper data];
int s = [wrapper serializedSize];
NSData *size = [NSData dataWithBytes:&s length:1];
[sock writeData:size withTimeout:-1 tag:1];
[sock writeData:data withTimeout:-1 tag:1];
and then on the Java end I just kept the
RequestWrapper.parseDelimitedFrom(socket.getInputStream())
line and it works a treat! All I end up doing is sending the size of the data before the data itself and the GPB method on the Java end works out the rest!
One major problem I had was actually converting the size of the data to send, from an int to NSData, and having it send across the network correctly. The way I was advised of doing it was
NSData *size = [NSData dataWithBytes:&s length:sizeof(s)];
However whenever I sent that across as data, it would seem to send the first byte, along with 3 "0" bytes. This caused havoc with GPB because if it receives a 0 byte at any point it throws an exception thinking the code is corrupt (my guess). Seeing as I never looked at the actual bytes coming across and analysing them until trying to do it a different way today, I am a bit gutted as I could have figured out that this was the issue a while ago. After some experimenting around with it, I gathered that the 'sizeof' method was the problem so I removed it. Currently I have just put a '1' instead of the actual size, which
seems to only return 1 byte when sending the data file across the network; although I'm not sure thats going to be an 'ideal' solution (although the message size should only be in 1 byte anyway) - if anyone could advise me why this sizeof() is causing an issue, it would be appreciated :)
Points should really go to Luke Steffen who helped me with this on the cocoaasyncsocket google group despite my idiocy - so again, thanks Luke!
I have a Java Client which sends UTF-8 strings to a C# TCP-Server, I'm using a DataOutputStream to send the strings. The code looks like this:
public void sendUTF8String(String ar) {
if (socket.isConnected()) {
try {
dataOutputStream.write(ar.getBytes(Charset.forName("UTF-8")));
dataOutputStream.flush();
} catch (IOException e) {
handleException(e);
}
}
}
The problem is that flush doesn't seem to work right. If I send two Strings close to each other, the server receives only one message with both strings. The whole thing works if I do a Thread.sleep(1000) between calls, this is obviously not a solution.
What am I missing?
flush() doesn't guarantee that a data packet gets shipped off. Your TCP/IP stack is free to bundle your data for maximum efficiency. Worse, there are probably a bunch of other TCP/IP stacks between you and your destination, and they are free to do the same.
I think you shouldn't rely on packet bundling. Insert a logical terminator/divider in your data and you will be on the safe side.
You shouldn't worry about how the data is broken up into packets.
You should include the length of the string in your messages, and then on the receiving end you would read the length first. So for example to send you would do
byte[] arbytes = ar.getBytes(Charset.forName("UTF-8"));
output.writeInt(arbytes.length)
output.write(arbytes)
and then in your reader you do
byte[] arbytes = new byte[input.readInt()];
for(int i = 0; i < len; i++){
arbytes[i] = input.read();
}
//convert bytes back to string.
You can't just call input.read(arbytes) because the read function doesn't necessarily read the entire length of the array. You can do a loop where you read a chunk at a time but the code for that is a bit more complex.
Anyway, you get the idea.
Also, if you really want to control what goes in what packets, you can use Datagram Sockets, but if you do that then delivery of the packet is not guaranteed.
Socket send a stream of data, not messages.
You shouldn't rely on the packets you receive being the same size as they are sent.
Packets can be grouped together as you have seen but they can also be broken up.
Use #Chad Okere's suggestion on how to ensure you get blocks the same was they are sent.
However in your case, you can just use
dataOutputStream.writeUTF(ar); // sends a string as UTF-8
and
String text = dataInputStream.readUTF(); // reads a string as UTF-8