Develop Reference

Too much data on Node.js socket?

I'm currently developing a system that gets data from a battery pack of an electric vehicle, stores it in a database and display it on a screen.
So I have a Java - Application that reads the data from a hardware interface, interprets the values and sends it via Socket to a Node.js-Server. (Java App and Webserver are running on the same computer, so Url = localhost)
JAVA APP:
s = new Socket();
s.connect(new InetSocketAddress(URL, PORT));
out = new PrintWriter( s.getOutputStream(), true);
for (DataEntry e : entries){
out.printf(e.toJson());
}
NODE:
sock.on('data', function(data) {
try{
var data = JSON.parse(data);
db.serialize(function(){
db.run("INSERT INTO DataEntry(value, MessageField, time) values(" + data.value + "," + data.messageFieldID + ", STRFTIME('%Y-%m-%d %H:%M:%f'))");
});
} catch(e){}
});
I get about 20 Messages per second from the hardware interface which are converted into 100 Json - Strings. So the webserver has to process one message in 10 ms, which I thought, is manageable.
But here is the problem: If my entries - List (foreach loop) has more than 2 elements, the webserver gets 2 or more of the Json's in one message.
So the first message was divided into 2 parts (ID 41,42) and was processed correctly. But the second message was divided into 5 parts (ID 43-47), and the first 4 of them weren't sent alone, so only the last one was saved correctly.
How can I ensure, that every Json is sent one another?
Isn't there something like a buffer so that the socket.on method is called correctly for every message I send?
I hope somebody of you can help me
Thank you!
Benedikt :)

TCP sockets are just streams and you shouldn't make any assumptions about how much of a "message" is contained in a single packet.
A simple solution to this is to terminate each message with a newline character since JSON cannot contain such a character. From there it's a simple matter of buffering data until you see a newline character. Then call JSON.parse() on your buffer. For example:
var buf = '';
sock.on('data', function(data) {
buf += data;
var p;
// Use a while loop as it may be possible to have multiple
// messages buffered depending on chunk contents
while (~(p = buf.indexOf('\n'))) {
try {
var msg = JSON.parse(buf.slice(0, p));
} catch (ex) {
console.log('Bad JSON message: ' + ex);
}
buf = buf.slice(p + 1);
}
});
You will also need to change printf() to println() on the Java-side so that a newline character will be appended to each message.

Bluetooth Serial Data Transfer Hangs at same points

I am to transfer two files from a Java app to an Arduino of roughly 200k each to be saved to uSD. With a USB connection this is going fine using the RXTX library and simple data parsing on the Arduino. I now want to replace the USB connect with a bluetooth using a HC-06 module.
Without changing anything the transfer works but it takes 12 minutes rather than 2 minutes. The transfer hangs at the same 4 (maybe 50) points of the transfer, seen by percentage progress.
The bottleneck must be in the RXTX library or bluteooth TX but can't find what's happening. Is there anything I can tweak or is this a limitation of a basic bluetooth module?
Here are snippets from code FWIW, nothing out of the ordinary (I think!) :
Java :
int perc = data.length / 100;
for (int i = 0 ; i < data.length ; i++) {
try{
output.write(data[i]);
}
catch(Exception e){
System.err.println(e.toString());
}
if(i % perc == 0) System.out.println(i / perc);
}
Arduino :
boolean waiting = true;
while( lastData + 2000 > millis() || waiting){
while(Serial.available()) {
waiting = false;
if(Serial.available() >= 255) nodes[nodeCount++] = byteCount;
buff[bytInc++] = Serial.read();
if(bytInc == 512) {
bytInc = 0;
in.write(buff,512);
}
lastData = millis();
byteCount++;
}
}
in.write(buff,bytInc);
in.flush();
Any optimisation in the snippets above would be much appreciated. The baud rate is 38400. Any faster and the buffer overloads when writing to uSD.
Arduino board is a Mega with 256 byte serial buffer.
Much appreciated. Have hunted about for relevant posts but can't find anything on bluetooth file transfers like this.

Java sockets and dataoutputstream delay

I am a beginner in Java and I am working at a project involving sockets and external devices.
I have done a small listener for the units and I am having some problems sending messages to the devices.
The unit has 2 messages that is sending to the server:
1. the Heart Beat message witch is in hex
2. the actual data which is a string and looks like: field1,field2,field3,field4
The listener is opening a new thread for each device and is wait for messages.
I use an DataInputStream for receiving the messages and DataOutputStream for sending the messages. Each time a unit is connecting to the server, it opens a new thread and gets the heart beat, from that heart beat I know what type of device is and who is it. If the device is new I will configure it based on 2 unique serials which the unit knows to send me and I have to compare them to the database.
My problem is in the part in which I ask the device for the 2 serials. It takes a long time 30 to 40 seconds. It has to be under 10 seconds for send the command (tested on another tcp server).
Here is that part of code:
String [] commands = deviceDetails.getConfigCommands().split(";"); // I take the commands from a file, and looks like: command1;command2
for (int i=0; i<commands.length; i++) {
sendCommand = commands[i].getBytes(); // sendCommand is a byte[]
out.write(sendCommand);
out.flush();
loop: while (true) {
in.read(buffer); // buffer is a byte[]
String[] result = new String(buffer, "ASCII").split(":|="); // the response from the unit looks like: ok:commandname=code, i need the code
if (configCounter ==2 && imei.equals("")) {
i = 0;
break loop;
} else if(configCounter == 2 && simid.equals("")) {
i = 1;
break loop;
}
configCounter++;
switch (result[1]) {
case "IMEI":
imei = result[2];
break loop;
case "SIMID":
simid = result[2];
break loop;
}
}
}
in the while I am waiting for the response from the unit. but before that I am trying to send the command to the unit and it takes about 30-40 second for each command.
Can anyone tell me what's the problem in the code above? How can I make the DataOutputStream send the command faster than 30-40 seconds??
Thank you!

Processing / Reading Serial Port Data Streams Crashing Program

I'm working on a simple program to read a continuous stream of data from a plain old serial port. The program is written in Processing. Performing a simple read of data and dumping into to the console works perfectly fine, but whenever I add any other functionality (graphing,db entry) to the program, the port starts to become de-synchronized and all data from the serial port starts to become corrupt.
The incoming data from the serial port is in the following format :
A [TAB] //start flag
Data 1 [TAB]
Data 2 [TAB]
Data 3 [TAB]
Data 4 [TAB]
Data 5 [TAB]
Data 6 [TAB]
Data 7 [TAB]
Data 8 [TAB]
COUNT [TAB] //count of number of messages sent
Z [CR] //end flag followed by carriage return
So as stated, if I run the program below and simply have it output to the console, it runs fine without issue for several hours. If I add the graphing functionality or database connectivity, the serial data starts to come in garbled and serial port handler is never able to decode the message correctly again. I've tried all sorts of workarounds to this issue, thinking it is a timing problem but reducing the speed of the serial port doesn't seem to make a change.
If you see the serial port handler, I provide a large buffer just in case the terminating Z character is chopped off. I check to make sure the A and Z characters are in the correct place and in turn that the created "substring" is the correct length. When the program starts to fail, the substring will continuously fail this check until the program just crashes. Any ideas? I've tried several different ways of reading the serial port and am just beginning to wonder if I am missing something stupid here.
//Serial Port Tester
import processing.serial.*;
import processing.net.*;
import org.gwoptics.graphics.graph2D.Graph2D;
import org.gwoptics.graphics.graph2D.traces.ILine2DEquation;
import org.gwoptics.graphics.graph2D.traces.RollingLine2DTrace;
import de.bezier.data.sql.*;
SQLite db;
RollingLine2DTrace r1,r2,r3,r4;
Graph2D g;
Serial mSerialport; //the serial port
String[] svalues = new String[8]; //string values
int[] values = new int[8]; //int values
int endflag = 90; //Z
byte seperator = 13; //carriage return
class eq1 implements ILine2DEquation {
public double computePoint(double x,int pos) {
//data function for graph/plot
return (values[0] - 32768);
}
}
void connectDB()
{
db = new SQLite( this, "data.sqlite" );
if ( db.connect() )
{
db.query( "SELECT name as \"Name\" FROM SQLITE_MASTER where type=\"table\"" );
while (db.next())
{
println( db.getString("Name") );
}
}
}
void setup () {
size(1200, 1000);
connectDB();
println(Serial.list());
String portName = Serial.list()[3];
mSerialport = new Serial(this, portName, 115200);
mSerialport.clear();
mSerialport.bufferUntil(endflag); //generate serial event when endflag is received
background(0);
smooth();
//graph setup
r1 = new RollingLine2DTrace(new eq1(),250,0.1f);
r1.setTraceColour(255, 0, 0);
g = new Graph2D(this, 1080, 500, false);
g.setYAxisMax(10000);
g.addTrace(r1);
g.position.y = 50;
g.position.x = 100;
g.setYAxisTickSpacing(500);
g.setXAxisMax(10f);
}
void draw () {
background(200);
//g.draw(); enable this and program crashes quickly
}
void serialEvent (Serial mSerialport)
{
byte[] inBuffer = new byte[200];
mSerialport.readBytesUntil(seperator, inBuffer);
String inString = new String(inBuffer);
String subString = "";
int startFlag = inString.indexOf("A");
int endFlag = inString.indexOf("Z");
if (startFlag == 0 && endFlag == 48)
{
subString = inString.substring(startFlag+1,endFlag);
}
else
{
println("ERROR: BAD MESSAGE DISCARDED!");
subString = "";
}
if ( subString.length() == 47)
{
svalues = (splitTokens(subString));
values = int(splitTokens(subString));
println(svalues);
// if (db.connect()) //enable this and program crashes quickly
// {
// if ( svalues[0] != null && svalues[7] != null)
// {
// statement = svalues[7] + ", " + svalues[0] + ", " + svalues[1] + ", " + svalues[2] + ", " + svalues[3] + ", " + svalues[4] + ", " + svalues[5] + ", " + svalues[6];
// db.execute( "INSERT INTO rawdata (messageid,press1,press2,press3,press4,light1,light2,io1) VALUES (" + statement + ");" );
// }
// }
}
}

While I'm not familiar with your specific platform, my first thought from reading your problem description is that you still have a timing problem. At 115,200bps, data is coming in rather quickly-- more than 10 characters every millisecond. As such, if you spend precious time opening a database (slow file IO) or drawing graphics (also potentially slow), you might well not be able to keep up with the data.
As such, it might be a good idea to put the serial port processing on its own thread, interrupt, etc. That might make the multitasking much easier. Again, this is just an educated guess.
Also, you say that your program "crashes" when you enable the other operations. Do you mean that the entire process actually crashes, or that you get corrupted data, or both? Is it possible that you are overrunning your 200 byte inBuffer[]? At 115kbps, it wouldn't take but 20ms to do so.

Serial Communication between Java RXTX and Arduino

I'm trying to communicate between my PC (Windows 7 using Netbeans and RXTX) with an Arduino Pro, using the serial port. The Arduino is actually connected to the PC using an FTDI cable.
The code is based on the Java SimpleRead.Java found here.
Currently the Arduino simply prints out a string when it starts up. My Java program should print the number of bytes that have been read and then print out the contents. The Java program works, sort of...
If the string is long (>10 bytes or so) the output will get broken up.
So if on the Arduino I print
Serial.println("123456789123456789"); //20 bytes including '\r' and '\n'
The output of my Java program may look something like:
Number of Bytes: 15
1234567891234
Number of Bytes: 5
56789
or
Number of Bytes: 12
1234567891
Number of Bytes: 8
23456789
I'm thinking it's a timing problem, because when I manually go through the code using the debugger, the result string is always what it should be: one 20 byte string.
I've been messing with various things but I haven't been able to fix the problem.
Here is the part of the code that is giving me problems:
static int baudrate = 9600,
dataBits = SerialPort.DATABITS_8,
stopBits = SerialPort.STOPBITS_1,
parity = SerialPort.PARITY_NONE;
byte[] readBuffer = new byte[128];
...
...
public void serialEvent(SerialPortEvent event)
{
if (event.getEventType() == SerialPortEvent.DATA_AVAILABLE) {
try {
if (input.available() > 0) {
//Read the InputStream and return the number of bytes read
numBytes = input.read(readBuffer);
String result = new String(readBuffer,0,numBytes);
System.out.println("Number of Bytes: " + numBytes);
System.out.println(result);
}
} catch (IOException e) {
System.out.println("Data Available Exception");
}
}

Serial data is just a stream of data. Depending on when you read it and the buffering that is happening, only part of the data may be available when you read it.
Since you are using line oriented data, what you will want to do is buffer the data until you see the line terminator and only then process the data.

I haven't used Java RXTX, but I've played with Arduino and Processing and it's pretty easy to read/write values from Arduino.
Here is a read sample that comes with Processing(File > Examples > Libraries > Serial > SimpleRead)
/**
* Simple Read
*
* Read data from the serial port and change the color of a rectangle
* when a switch connected to a Wiring or Arduino board is pressed and released.
* This example works with the Wiring / Arduino program that follows below.
*/
import processing.serial.*;
Serial myPort; // Create object from Serial class
int val; // Data received from the serial port
void setup()
{
size(200, 200);
// I know that the first port in the serial list on my mac
// is always my FTDI adaptor, so I open Serial.list()[0].
// On Windows machines, this generally opens COM1.
// Open whatever port is the one you're using.
String portName = Serial.list()[0];
myPort = new Serial(this, portName, 9600);
}
void draw()
{
if ( myPort.available() > 0) { // If data is available,
val = myPort.read(); // read it and store it in val
}
background(255); // Set background to white
if (val == 0) { // If the serial value is 0,
fill(0); // set fill to black
}
else { // If the serial value is not 0,
fill(204); // set fill to light gray
}
rect(50, 50, 100, 100);
}
/*
// Wiring / Arduino Code
// Code for sensing a switch status and writing the value to the serial port.
int switchPin = 4; // Switch connected to pin 4
void setup() {
pinMode(switchPin, INPUT); // Set pin 0 as an input
Serial.begin(9600); // Start serial communication at 9600 bps
}
void loop() {
if (digitalRead(switchPin) == HIGH) { // If switch is ON,
Serial.print(1, BYTE); // send 1 to Processing
} else { // If the switch is not ON,
Serial.print(0, BYTE); // send 0 to Processing
}
delay(100); // Wait 100 milliseconds
}
*/
As far as I remember, the baud thingy you setup in Arduino when you instantiate Serial is pretty important. If you use 9600 to send for example, you should use the same number to listen.
Also it's pretty important to send your information as BYTE, otherwise you'll have stuff like \r or \n in the way.
Shorter version, try:
Serial.println(123456789123456789,BYTE);
The simpler the better.

I think you need to use event driven design patterns to solve this problem. I highly recommend you to visit: http://www.whatisarduino.org/bin/Tutorials/Java+Serial+API+and+Arduino