Develop Reference

Java BufferedInputStream read() does not receive data after not reading from Stream for some time

I wrote an application that listens for status messages from a robot. I have an open Socket connection to the robot. The robots sends constantly status messages which I read ( and parse ) to find out when the robot is done with his current task.
/**
* Listens on the Socket for RobotStateMessage (RSM). A RSM has messagetype 16 at byte[4]. The constructor of the RobotStateMessage class
* throws an exception when the message is found to be corrupt. If that happens the method waits for the next uncorrupted message.
*
* #param command the robot state message is added to the command
* #throws IOException
* #throws GeneralURMessage_Parse_Exception
* #throws TimeExpired_Exception
*/
public void executeRobotStateCommand(RobotStateCommand command) throws IOException, RobotStateMessage.GeneralURMessage_Parse_Exception, TimeExpired_Exception {
byte[] data = new byte[3000];
boolean run = true;
RobotStateMessage message = null;
long momentToQuit = System.currentTimeMillis() + 5000;
while (run) {
if (System.currentTimeMillis() > momentToQuit) {
throw new TimeExpired_Exception(5000);
}
ur_in.read(data);
int type = data[4];
if (type == 16) {
run = false;
try {
// contructor of RobotStateMessage parses the data
message = new RobotStateMessage(data);
} catch (RobotStateMessage.CorruptRobotStateMessage_Exception e) {
// the message received form ur was corrupted , ignore and wait for next message
run = true;
}
}
}
command.robotStateMessage = message;
}
The problem occurs when I stop reading from the BufferedInputStream ur_in for a while. Any pause longer than aproximatly a minute and the ur_in.read(data) blocks after consuming all the buffered old data.
After consuming buffered data there seems to be no new data on the BufferedInputStream. Using a second tool to listen to the data from the robot, I can clearly see that the status messages are still broadcasted.
The Socket is still alive but the Inputstream seems to have "died". I put a timeout on the socket so that the read() does not block forever, but that does not solve my problem, that I receive no more data over the BufferdInputStream.
The only thing that helped was reconnecting the socket, which is not a satisfying solution for my problem.
Any help or suggestions how to solve this are appreciated. If you need more information please ask.

I assume that you are using TCP. If you don't read from the buffer the the robot will be notified by your socket to stop transmission until the buffer has capacity to receive more data. The robot software will evetually have to handle that no data can be written and probably disconnects your client. You should not use the socket buffer to store data and instead as quickly as possible move the data to an other buffer. Setting a larger receive buffer will probably help a little, but is not a good idea.

Too much data on Node.js socket?

I'm currently developing a system that gets data from a battery pack of an electric vehicle, stores it in a database and display it on a screen.
So I have a Java - Application that reads the data from a hardware interface, interprets the values and sends it via Socket to a Node.js-Server. (Java App and Webserver are running on the same computer, so Url = localhost)
JAVA APP:
s = new Socket();
s.connect(new InetSocketAddress(URL, PORT));
out = new PrintWriter( s.getOutputStream(), true);
for (DataEntry e : entries){
out.printf(e.toJson());
}
NODE:
sock.on('data', function(data) {
try{
var data = JSON.parse(data);
db.serialize(function(){
db.run("INSERT INTO DataEntry(value, MessageField, time) values(" + data.value + "," + data.messageFieldID + ", STRFTIME('%Y-%m-%d %H:%M:%f'))");
});
} catch(e){}
});
I get about 20 Messages per second from the hardware interface which are converted into 100 Json - Strings. So the webserver has to process one message in 10 ms, which I thought, is manageable.
But here is the problem: If my entries - List (foreach loop) has more than 2 elements, the webserver gets 2 or more of the Json's in one message.
So the first message was divided into 2 parts (ID 41,42) and was processed correctly. But the second message was divided into 5 parts (ID 43-47), and the first 4 of them weren't sent alone, so only the last one was saved correctly.
How can I ensure, that every Json is sent one another?
Isn't there something like a buffer so that the socket.on method is called correctly for every message I send?
I hope somebody of you can help me
Thank you!
Benedikt :)

TCP sockets are just streams and you shouldn't make any assumptions about how much of a "message" is contained in a single packet.
A simple solution to this is to terminate each message with a newline character since JSON cannot contain such a character. From there it's a simple matter of buffering data until you see a newline character. Then call JSON.parse() on your buffer. For example:
var buf = '';
sock.on('data', function(data) {
buf += data;
var p;
// Use a while loop as it may be possible to have multiple
// messages buffered depending on chunk contents
while (~(p = buf.indexOf('\n'))) {
try {
var msg = JSON.parse(buf.slice(0, p));
} catch (ex) {
console.log('Bad JSON message: ' + ex);
}
buf = buf.slice(p + 1);
}
});
You will also need to change printf() to println() on the Java-side so that a newline character will be appended to each message.

Blocking read function in c in a client/server application

I'm currently developing a client/server architecture between a tablet (client) and a MAC/PC (server). I am doing on both side some real-time rendering and I need communication between the two.
The problem is that I need to do some operation on the string I get from my client (which is basically a rotation matrix). This string is therefore to be at most 16 float numbers that I previously transform into a coma-separated-value string.
Therefore what I should get from my client is something like:
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
Server-side, I do some processing of that string to get back my rotation matrix as a float array of 16 elements. The problem is that sometimes I get more than just 16 elements from the client on the server side at once. I for instance get
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
So that when I try to split it, I go above the 16 element limits which is not good at all for me.
My question is: is there a way to prevent the server and/or the client to read/send more than one complete matrix at a time? Since I'm using a tablet and some real-time rendering I would like to be able to save as much processing power as possible.
Here is the code that I'm using (just snippets as files are quite big)
Client:
if (connected == true && matrixupdated == true && this.hasMatrixChanged()){
try {
this.inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
this.outToServer= new DataOutputStream(clientSocket.getOutputStream());
this.sentence = this.getStringFromMatrix();
outToServer.writeBytes(sentence + '\n');
this.hasServerProcessed = false ;
System.arraycopy(matrix, 0, previousMatrix, 0, 16); //I check whether the matrix changes enough for me to send it to the server
}catch (Exception e) {
Log.e("ClientActivity", "S: Error", e);
}
this.matrixupdated = false ;
Server :
while( (read_size = recv(sock , client_message , 2000 , 0)) > 0 )
{
smatrix = client_message ; //smatrix is a true c++ string
pthread_mutex_lock(&mymutex);
pthread_cond_wait(&mycondition, &mymutex); // prevent real-time rendering to try and use the matrix at the same time as this function
std::stringstream ss(smatrix);
while(std::getline(ss, tok, ',')) {
matrix[i] = ::atof(tok.c_str());
i++ ;
}
i = 0 ;
pthread_mutex_unlock(&mymutex);
}

Working as designed. TCP is a byte stream protocol. There are no message boundaries. If you want messages you have to implement them yourself.

Android to computer FTP resuming upload strange phenomenon

I have a strange phenomenon when resuming a file transfer.
Look at the picture below you see the bad section.
This happens apparently random, maybe every 10:th time.
Im sending the picture from my Android phone to java server over ftp.
What is it that i forgot here.
I see the connection is killed due to java.net.SocketTimeoutException:
The transfer is resuming like this
Resume at : 287609 Sending 976 bytes more
The bytes are always correct when file is completely received.
Even for the picture below.
Dunno where to start debug this since its working most of the times.
Any suggestions or ideas would be grate i think i totally missed something here.
The device Sender code (only send loop):
int count = 1;
//Sending N files, looping N times
while(count <= max) {
String sPath = batchFiles.get(count-1);
fis = new FileInputStream(new File(sPath));
int fileSize = bis.available();
out.writeInt(fileSize); // size
String nextReply = in.readUTF();
// if the file exist,
if(nextReply.equals(Consts.SERVER_give_me_next)){
count++;
continue;
}
long resumeLong = 0; // skip this many bytes
int val = 0;
buffer = new byte[1024];
if(nextReply.equals(Consts.SERVER_file_exist)){
resumeLong = in.readLong();
}
//UPDATE FOR #Justin Breitfeller, Thanks
long skiip = bis.skip(resumeLong);
if(resumeLong != -1){
if(!(resumeLong == skiip)){
Log.d(TAG, "ERROR skip is not the same as resumeLong ");
skiip = bis.skip(resumeLong);
if(!(resumeLong == skiip)){
Log.d(TAG, "ERROR ABORTING skip is not the same as resumeLong);
return;
}
}
}
while ((val = bis.read(buffer, 0, 1024)) > 0) {
out.write(buffer, 0, val);
fileSize -= val;
if (fileSize < 1024) {
val = (int) fileSize;
}
}
reply = in.readUTF();
if (reply.equals(Consts.SERVER_file_receieved_ok)) {
// check if all files are sent
if(count == max){
break;
}
}
count++;
}
The receiver code (very truncated):
//receiving N files, looping N times
while(count < totalNrOfFiles){
int ii = in.readInt(); // File size
fileSize = (long)ii;
String filePath = Consts.SERVER_DRIVE + Consts.PTPP_FILETRANSFER;
filePath = filePath.concat(theBatch.getFileName(count));
File path = new File(filePath);
boolean resume = false;
//if the file exist. Skip if done or resume if not
if(path.exists()){
if(path.length() == fileSize){ // Does the file has same size
logger.info("File size same skipping file:" + theBatch.getFileName(count) );
count++;
out.writeUTF(Consts.SERVER_give_me_next);
continue; // file is OK don't upload it again
}else {
// Resume the upload
out.writeUTF(Consts.SERVER_file_exist);
out.writeLong(path.length());
resume = true;
fileSize = fileSize-path.length();
logger.info("Resume at : " + path.length() +
" Sending "+ fileSize +" bytes more");
}
}else
out.writeUTF("lets go");
byte[] buffer = new byte[1024];
// ***********************************
// RECEIVE FROM PHONE
// ***********************************
int size = 1024;
int val = 0;
bos = new BufferedOutputStream(new FileOutputStream(path,resume));
if(fileSize < size){
size = (int) fileSize;
}
while (fileSize >0) {
val = in.read(buffer, 0, size);
bos.write(buffer, 0, val);
fileSize -= val;
if (fileSize < size)
size = (int) fileSize;
}
bos.flush();
bos.close();
out.writeUTF("file received ok");
count++;
}

Found the error and the problem was bad logic from my part.
say no more.
I was sending pictures that was being resized just before they where sent.
The problem was when the resume kicked in after a failed transfer
the resized picture was not used, instead the code used the original
pictured that had a larger scale size.
I have now setup a short lived cache that holds the resized temporary pictures.
In the light of the complexity of the app im making I simply forgot that the files during resume was not the same as original.

With a BufferedOutputStream, BufferedInputStream, you need to watch out for the following
Create BufferedOutputStream before BuffererdInputStream (on both client and server)
And flush just after create.
Flush after every write (not just before close)
That worked for me.
Edited
Add sentRequestTime, receivedRequestTime, sentResponseTime, receivedResponseTime to your packet payload. Use System.nanoTime() on these, run your server and client on the same host, use ExecutorService to run multiple clients for that server, and plot your (received-sent) for both request and response packets, time delay on a excel chart (some csv format). Do this before bufferedIOStream and afterIOStream. You will be pleased to know that your performance has boosted by 100%. Made me very happy to plot that graph, took about 45 mins.
I have also heard that using custom buffer's further improves performance.
Edited again
In my case I am using Object IOStreams, I have added a payload of 4 long variables to the object, and initialize sentRequestTime when I send the packet from the client, initialize receivedRequestTime when the server receives the response, so and so forth for the response from server to client too. I then find the difference between received and sent time to find out the delay in response and request. Be careful to run this test on localhost. If you run it between different hardware/devices, their actual time difference may interfere with your test results. Since requestReceivedTime is time stamped at the server end and the requestSentTime is time stamped at the client end. In other words, their own local time is stamped (obviously). And both of these devices running the exact same time to the nano second is not possible. If you must run it between different devices atleast make sure that you have ntp running (to keep them time synchronized). That said, you hare comparing the performance before and after bufferedio (you dont really care about the actual time delays right ?), so time drift should not really matter. Comparing a set of results before buffered and after buffered is your actual interest.
Enjoy!!!

Serial Communication between Java RXTX and Arduino

I'm trying to communicate between my PC (Windows 7 using Netbeans and RXTX) with an Arduino Pro, using the serial port. The Arduino is actually connected to the PC using an FTDI cable.
The code is based on the Java SimpleRead.Java found here.
Currently the Arduino simply prints out a string when it starts up. My Java program should print the number of bytes that have been read and then print out the contents. The Java program works, sort of...
If the string is long (>10 bytes or so) the output will get broken up.
So if on the Arduino I print
Serial.println("123456789123456789"); //20 bytes including '\r' and '\n'
The output of my Java program may look something like:
Number of Bytes: 15
1234567891234
Number of Bytes: 5
56789
or
Number of Bytes: 12
1234567891
Number of Bytes: 8
23456789
I'm thinking it's a timing problem, because when I manually go through the code using the debugger, the result string is always what it should be: one 20 byte string.
I've been messing with various things but I haven't been able to fix the problem.
Here is the part of the code that is giving me problems:
static int baudrate = 9600,
dataBits = SerialPort.DATABITS_8,
stopBits = SerialPort.STOPBITS_1,
parity = SerialPort.PARITY_NONE;
byte[] readBuffer = new byte[128];
...
...
public void serialEvent(SerialPortEvent event)
{
if (event.getEventType() == SerialPortEvent.DATA_AVAILABLE) {
try {
if (input.available() > 0) {
//Read the InputStream and return the number of bytes read
numBytes = input.read(readBuffer);
String result = new String(readBuffer,0,numBytes);
System.out.println("Number of Bytes: " + numBytes);
System.out.println(result);
}
} catch (IOException e) {
System.out.println("Data Available Exception");
}
}

Serial data is just a stream of data. Depending on when you read it and the buffering that is happening, only part of the data may be available when you read it.
Since you are using line oriented data, what you will want to do is buffer the data until you see the line terminator and only then process the data.

I haven't used Java RXTX, but I've played with Arduino and Processing and it's pretty easy to read/write values from Arduino.
Here is a read sample that comes with Processing(File > Examples > Libraries > Serial > SimpleRead)
/**
* Simple Read
*
* Read data from the serial port and change the color of a rectangle
* when a switch connected to a Wiring or Arduino board is pressed and released.
* This example works with the Wiring / Arduino program that follows below.
*/
import processing.serial.*;
Serial myPort; // Create object from Serial class
int val; // Data received from the serial port
void setup()
{
size(200, 200);
// I know that the first port in the serial list on my mac
// is always my FTDI adaptor, so I open Serial.list()[0].
// On Windows machines, this generally opens COM1.
// Open whatever port is the one you're using.
String portName = Serial.list()[0];
myPort = new Serial(this, portName, 9600);
}
void draw()
{
if ( myPort.available() > 0) { // If data is available,
val = myPort.read(); // read it and store it in val
}
background(255); // Set background to white
if (val == 0) { // If the serial value is 0,
fill(0); // set fill to black
}
else { // If the serial value is not 0,
fill(204); // set fill to light gray
}
rect(50, 50, 100, 100);
}
/*
// Wiring / Arduino Code
// Code for sensing a switch status and writing the value to the serial port.
int switchPin = 4; // Switch connected to pin 4
void setup() {
pinMode(switchPin, INPUT); // Set pin 0 as an input
Serial.begin(9600); // Start serial communication at 9600 bps
}
void loop() {
if (digitalRead(switchPin) == HIGH) { // If switch is ON,
Serial.print(1, BYTE); // send 1 to Processing
} else { // If the switch is not ON,
Serial.print(0, BYTE); // send 0 to Processing
}
delay(100); // Wait 100 milliseconds
}
*/
As far as I remember, the baud thingy you setup in Arduino when you instantiate Serial is pretty important. If you use 9600 to send for example, you should use the same number to listen.
Also it's pretty important to send your information as BYTE, otherwise you'll have stuff like \r or \n in the way.
Shorter version, try:
Serial.println(123456789123456789,BYTE);
The simpler the better.

I think you need to use event driven design patterns to solve this problem. I highly recommend you to visit: http://www.whatisarduino.org/bin/Tutorials/Java+Serial+API+and+Arduino