Develop Reference

Too much data on Node.js socket?

I'm currently developing a system that gets data from a battery pack of an electric vehicle, stores it in a database and display it on a screen.
So I have a Java - Application that reads the data from a hardware interface, interprets the values and sends it via Socket to a Node.js-Server. (Java App and Webserver are running on the same computer, so Url = localhost)
JAVA APP:
s = new Socket();
s.connect(new InetSocketAddress(URL, PORT));
out = new PrintWriter( s.getOutputStream(), true);
for (DataEntry e : entries){
out.printf(e.toJson());
}
NODE:
sock.on('data', function(data) {
try{
var data = JSON.parse(data);
db.serialize(function(){
db.run("INSERT INTO DataEntry(value, MessageField, time) values(" + data.value + "," + data.messageFieldID + ", STRFTIME('%Y-%m-%d %H:%M:%f'))");
});
} catch(e){}
});
I get about 20 Messages per second from the hardware interface which are converted into 100 Json - Strings. So the webserver has to process one message in 10 ms, which I thought, is manageable.
But here is the problem: If my entries - List (foreach loop) has more than 2 elements, the webserver gets 2 or more of the Json's in one message.
So the first message was divided into 2 parts (ID 41,42) and was processed correctly. But the second message was divided into 5 parts (ID 43-47), and the first 4 of them weren't sent alone, so only the last one was saved correctly.
How can I ensure, that every Json is sent one another?
Isn't there something like a buffer so that the socket.on method is called correctly for every message I send?
I hope somebody of you can help me
Thank you!
Benedikt :)

TCP sockets are just streams and you shouldn't make any assumptions about how much of a "message" is contained in a single packet.
A simple solution to this is to terminate each message with a newline character since JSON cannot contain such a character. From there it's a simple matter of buffering data until you see a newline character. Then call JSON.parse() on your buffer. For example:
var buf = '';
sock.on('data', function(data) {
buf += data;
var p;
// Use a while loop as it may be possible to have multiple
// messages buffered depending on chunk contents
while (~(p = buf.indexOf('\n'))) {
try {
var msg = JSON.parse(buf.slice(0, p));
} catch (ex) {
console.log('Bad JSON message: ' + ex);
}
buf = buf.slice(p + 1);
}
});
You will also need to change printf() to println() on the Java-side so that a newline character will be appended to each message.

Canceling filewatcher run by TimerTask after receiving websocket message in Java

I've been coding around in circles trying to work this one out, new to java and have been lurking here to find things out for a while but I really can't get passed this one. I adapted some code by Desmond Shaw (http://www.codepool.biz/how-to-implement-a-java-websocket-server-for-image-transmission-with-jetty.html) to create a websocket to tranfer jpg images from a server to remote clients. I want to send files from the server to the browser windows of connected clients when some specific files on the server change (they are pages of music scores that are created using Max/MSP in real-time), but I don't seem to be able to cancel the timers I'm creating to watch these files in my home directory for changes.
More specifically I'm sending messages from the remote browser clients (through javascript buttons operated by the users) over a websocket connection to specify which of the files they wish to see updated on their screen (i.e part one, which refers to a file on the server called "1.png" and is the violin part, or part 2 which is the server file "2.png" and is the cello part etc). This is then used within the websocket handler running on my server to send the right files to that client when a filewatcher detects they have changed on the server. I can get everything going except stopping the timers running the filewatchers, when a different part is requested by the client (say the violin player wants to look at the cello players part). Below is the method I have edited to respond to the messages from the clients:
#OnWebSocketMessage //part request from websocket client (remote browser)
public void onMessage( String message) {
System.out.println("Message: '" + message + "' received");
sFclient = message;
if (sFclient == "1" || sFclient == "2" || sFclient == "3" || sFclient == "4") {
System.out.println("Part " + sFclient + " joined");
}
else {
sFclientOut = 0;
}
}
public void onChange( File file ) {
System.out.println( "File "+ file.getName() +" has changed!" );
TimerTask task = new FileWatcher(new File("/Users/benedict/" + message + ".png")) {
try {
File f = new File("/Users/benedict/" + sFclient + ".png");
BufferedImage bi = ImageIO.read(f);
ByteArrayOutputStream out = new ByteArrayOutputStream();
ImageIO.write(bi, "png", out);
ByteBuffer byteBuffer = ByteBuffer.wrap(out.toByteArray());
mSession.getRemote().sendBytes(byteBuffer);
out.close();
byteBuffer.clear();
}
catch (IOException e) {
e.printStackTrace();
}
Timer timer1 = new Timer(); {
timer1.schedule(task , new Date(), 20 );
if (sFclientOut == 0){
task.cancel();
timer1.cancel();
}
}
}
I had it working mostly using an if statement which I've now abandoned but have been editing and now it doesn't make as much sense probably. Any help at all would be appreciated, but my main question is should I be trying to cancel the threads handling the timertasks or use a completely different approach altogether like a switch statement for example. I have tried sending a message before every new message from the browsers ("0") to cancel the old threads but the Timertasks just don't start at all, which I think is because that cancels the timertask and doesn't let it run again?
Thanks,
Benedict

Ok here's the final working solution, I just had to cancel the tasks based on a message from the clients (in this case a 0)
else if (message.equals("0")) {
zerocounter = zerocounter + 1;
if (zerocounter >= 2) {
task.cancel();
}

Blocking read function in c in a client/server application

I'm currently developing a client/server architecture between a tablet (client) and a MAC/PC (server). I am doing on both side some real-time rendering and I need communication between the two.
The problem is that I need to do some operation on the string I get from my client (which is basically a rotation matrix). This string is therefore to be at most 16 float numbers that I previously transform into a coma-separated-value string.
Therefore what I should get from my client is something like:
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
Server-side, I do some processing of that string to get back my rotation matrix as a float array of 16 elements. The problem is that sometimes I get more than just 16 elements from the client on the server side at once. I for instance get
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0
So that when I try to split it, I go above the 16 element limits which is not good at all for me.
My question is: is there a way to prevent the server and/or the client to read/send more than one complete matrix at a time? Since I'm using a tablet and some real-time rendering I would like to be able to save as much processing power as possible.
Here is the code that I'm using (just snippets as files are quite big)
Client:
if (connected == true && matrixupdated == true && this.hasMatrixChanged()){
try {
this.inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
this.outToServer= new DataOutputStream(clientSocket.getOutputStream());
this.sentence = this.getStringFromMatrix();
outToServer.writeBytes(sentence + '\n');
this.hasServerProcessed = false ;
System.arraycopy(matrix, 0, previousMatrix, 0, 16); //I check whether the matrix changes enough for me to send it to the server
}catch (Exception e) {
Log.e("ClientActivity", "S: Error", e);
}
this.matrixupdated = false ;
Server :
while( (read_size = recv(sock , client_message , 2000 , 0)) > 0 )
{
smatrix = client_message ; //smatrix is a true c++ string
pthread_mutex_lock(&mymutex);
pthread_cond_wait(&mycondition, &mymutex); // prevent real-time rendering to try and use the matrix at the same time as this function
std::stringstream ss(smatrix);
while(std::getline(ss, tok, ',')) {
matrix[i] = ::atof(tok.c_str());
i++ ;
}
i = 0 ;
pthread_mutex_unlock(&mymutex);
}

Working as designed. TCP is a byte stream protocol. There are no message boundaries. If you want messages you have to implement them yourself.

SeekableByteChannel.read() always returns 0, InputStream is fine

We have a data file for which we need to generate a CRC. (As a placeholder, I'm using CRC32 while the others figure out what CRC polynomial they actually want.) This code seems like it ought to work:
broken:
Path in = ......;
try (SeekableByteChannel reading =
Files.newByteChannel (in, StandardOpenOption.READ))
{
System.err.println("byte channel is a " + reading.getClass().getName() +
" from " + in + " of size " + reading.size() + " and isopen=" + reading.isOpen());
java.util.zip.CRC32 placeholder = new java.util.zip.CRC32();
ByteBuffer buffer = ByteBuffer.allocate (reasonable_buffer_size);
int bytesread = 0;
int loops = 0;
while ((bytesread = reading.read(buffer)) > 0) {
byte[] raw = buffer.array();
System.err.println("Claims to have read " + bytesread + " bytes, have buffer of size " + raw.length + ", updating CRC");
placeholder.update(raw);
loops++;
buffer.clear();
}
// do stuff with placeholder.getValue()
}
catch (all the things that go wrong with opening files) {
and handle them;
}
The System.err and loops stuff is just for debugging; we don't actually care how many times it takes. The output is:
byte channel is a sun.nio.ch.FileChannelImpl from C:\working\tmp\ls2kst83543216xuxxy8136.tmp of size 7196 and isopen=true
finished after 0 time(s) through the loop
There's no way to run the real code inside a debugger to step through it, but from looking at the source to sun.nio.ch.FileChannelImpl.read() it looks like a 0 is returned if the file magically becomes closed while internal data structures are prepared; the code below is copied from the Java 7 reference implementation, comments added by me:
// sun.nio.ch.FileChannelImpl.java
public int read(ByteBuffer dst) throws IOException {
ensureOpen(); // this throws if file is closed...
if (!readable)
throw new NonReadableChannelException();
synchronized (positionLock) {
int n = 0;
int ti = -1;
Object traceContext = IoTrace.fileReadBegin(path);
try {
begin();
ti = threads.add();
if (!isOpen())
return 0; // ...argh
do {
n = IOUtil.read(fd, dst, -1, nd);
} while (......)
.......
But the debugging code tests isOpen() and gets true. So I don't know what's going wrong.
As the current test data files are tiny, I dropped this in place just to have something working:
works for now:
try {
byte[] scratch = Files.readAllBytes(in);
java.util.zip.CRC32 placeholder = new java.util.zip.CRC32();
placeholder.update(scratch);
// do stuff with placeholder.getValue()
}
I don't want to slurp the entire file into memory for the Real Code, because some of those files can be large. I do note that readAllBytes uses an InputStream in its reference implementation, which has no trouble reading the same file that SeekableByteChannel failed to. So I'll probably rewrite the code to just use input streams instead of byte channels. I'd still like to figure out what's gone wrong in case a future scenario comes up where we need to use byte channels. What am I missing with SeekableByteChannel?

Check that 'reasonable_buffer_size' isn't zero.

Serial Communication between Java RXTX and Arduino

I'm trying to communicate between my PC (Windows 7 using Netbeans and RXTX) with an Arduino Pro, using the serial port. The Arduino is actually connected to the PC using an FTDI cable.
The code is based on the Java SimpleRead.Java found here.
Currently the Arduino simply prints out a string when it starts up. My Java program should print the number of bytes that have been read and then print out the contents. The Java program works, sort of...
If the string is long (>10 bytes or so) the output will get broken up.
So if on the Arduino I print
Serial.println("123456789123456789"); //20 bytes including '\r' and '\n'
The output of my Java program may look something like:
Number of Bytes: 15
1234567891234
Number of Bytes: 5
56789
or
Number of Bytes: 12
1234567891
Number of Bytes: 8
23456789
I'm thinking it's a timing problem, because when I manually go through the code using the debugger, the result string is always what it should be: one 20 byte string.
I've been messing with various things but I haven't been able to fix the problem.
Here is the part of the code that is giving me problems:
static int baudrate = 9600,
dataBits = SerialPort.DATABITS_8,
stopBits = SerialPort.STOPBITS_1,
parity = SerialPort.PARITY_NONE;
byte[] readBuffer = new byte[128];
...
...
public void serialEvent(SerialPortEvent event)
{
if (event.getEventType() == SerialPortEvent.DATA_AVAILABLE) {
try {
if (input.available() > 0) {
//Read the InputStream and return the number of bytes read
numBytes = input.read(readBuffer);
String result = new String(readBuffer,0,numBytes);
System.out.println("Number of Bytes: " + numBytes);
System.out.println(result);
}
} catch (IOException e) {
System.out.println("Data Available Exception");
}
}

Serial data is just a stream of data. Depending on when you read it and the buffering that is happening, only part of the data may be available when you read it.
Since you are using line oriented data, what you will want to do is buffer the data until you see the line terminator and only then process the data.

I haven't used Java RXTX, but I've played with Arduino and Processing and it's pretty easy to read/write values from Arduino.
Here is a read sample that comes with Processing(File > Examples > Libraries > Serial > SimpleRead)
/**
* Simple Read
*
* Read data from the serial port and change the color of a rectangle
* when a switch connected to a Wiring or Arduino board is pressed and released.
* This example works with the Wiring / Arduino program that follows below.
*/
import processing.serial.*;
Serial myPort; // Create object from Serial class
int val; // Data received from the serial port
void setup()
{
size(200, 200);
// I know that the first port in the serial list on my mac
// is always my FTDI adaptor, so I open Serial.list()[0].
// On Windows machines, this generally opens COM1.
// Open whatever port is the one you're using.
String portName = Serial.list()[0];
myPort = new Serial(this, portName, 9600);
}
void draw()
{
if ( myPort.available() > 0) { // If data is available,
val = myPort.read(); // read it and store it in val
}
background(255); // Set background to white
if (val == 0) { // If the serial value is 0,
fill(0); // set fill to black
}
else { // If the serial value is not 0,
fill(204); // set fill to light gray
}
rect(50, 50, 100, 100);
}
/*
// Wiring / Arduino Code
// Code for sensing a switch status and writing the value to the serial port.
int switchPin = 4; // Switch connected to pin 4
void setup() {
pinMode(switchPin, INPUT); // Set pin 0 as an input
Serial.begin(9600); // Start serial communication at 9600 bps
}
void loop() {
if (digitalRead(switchPin) == HIGH) { // If switch is ON,
Serial.print(1, BYTE); // send 1 to Processing
} else { // If the switch is not ON,
Serial.print(0, BYTE); // send 0 to Processing
}
delay(100); // Wait 100 milliseconds
}
*/
As far as I remember, the baud thingy you setup in Arduino when you instantiate Serial is pretty important. If you use 9600 to send for example, you should use the same number to listen.
Also it's pretty important to send your information as BYTE, otherwise you'll have stuff like \r or \n in the way.
Shorter version, try:
Serial.println(123456789123456789,BYTE);
The simpler the better.

I think you need to use event driven design patterns to solve this problem. I highly recommend you to visit: http://www.whatisarduino.org/bin/Tutorials/Java+Serial+API+and+Arduino