Develop Reference

Missing character off incoming string from Bluetooth

I am currently trying to create a water level readout as a progress bar in a simple Android app. Currently, I am using an Arduino Mega 2560 with a HC-05 to transmit the readout of the water level sensor. To simplify things, the arduino code is just counting up and down from 0 to 1000 and back, as follows.
void setup() {
// put your setup code here, to run once:
Serial.begin(9600);
Serial.println("Test for Water Sensor");
Serial1.begin(9600);
}
void loop() {
// put your main code here, to run repeatedly:
for (int i = 0; i <= 1000; i++)
{
Serial1.println(i);
Serial.println(i);
delay(100);
}
for (int i = 1000; i >= 0; i--)
{
Serial1.println(i);
Serial.println(i);
delay(100);
}
}
On the android end, I am using this to convert to int, then change the progress bar. It also currently displays the unconverted message in a TextView.
mHandler = new Handler(Looper.getMainLooper()){
#Override
public void handleMessage(Message msg){
if(msg.what == MESSAGE_READ){
String readMessage = null;
try {
readMessage = new String((byte[]) msg.obj, "UTF-8");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
mReadBuffer.setText(readMessage);
try {
waterLevelValue = NumberFormat.getInstance().parse(readMessage).intValue();
waterLevel.setProgress(waterLevelValue);
} catch (ParseException e) {
e.printStackTrace();
}
}
if(msg.what == CONNECTING_STATUS){
if(msg.arg1 == 1)
mBluetoothStatus.setText("Connected to Device: " + msg.obj);
else
mBluetoothStatus.setText("Connection Failed");
}
}
};
The issue I am getting is that quite often (maybe 1-2 times a second) it is not reading the first digit. I can see on the Serial Monitor that all digits are going there, but on the android app, it will sometimes miss the first (eg: 443, 444, 45, 446, 447, etc)
What could be causing the issue here, I am very new to Bluetooth, so please help! More than happy to send more portions of code if needed.
EDIT: Adding code for reading input stream. Probably was important in the first place.
public void run() {
byte[] buffer = new byte[1024]; // buffer store for the stream
int bytes; // bytes returned from read()
// Keep listening to the InputStream until an exception occurs
while (true) {
try {
// Read from the InputStream
bytes = mmInStream.available();
if(bytes != 0) {
SystemClock.sleep(100); //pause and wait for rest of data. Adjust this depending on your sending speed.
bytes = mmInStream.available(); // how many bytes are ready to be read?
bytes = mmInStream.read(buffer, 0, bytes); // record how many bytes we actually read
mHandler.obtainMessage(MESSAGE_READ, bytes, -1, buffer)
.sendToTarget(); // Send the obtained bytes to the UI activity
}
} catch (IOException e) {
e.printStackTrace();
break;
}
}
}

Java socket data works only when sending time is delayed

I am sending a protobuf from C++ to Java via a raw socket, the C++ program being the client and the java program being the server. The C++ program generates packets almost every 1ms which is sent to the java program.
If I run the program normally, I see that there are only the half the packets being received.
If I set a breakpoint in the C++ program and then run the client and the server, all the packets are received.
How do I ensure that all packets are received without setting a breakpoint? Can I introduce a delay?
All the packets have bytes sizes upto a maximum of 15 bytes.

By default TCP sockets use the "Nagle Algorithm" which will delay transmission of the next "unfilled" fragment in order to reduce congestion. Your packet size is small enough and the time delay between packets is small enough that the nagle algorithm will have an effect on your transmissions.

As already discussed in the comments, what you are trying to do won't work in a reliable way. This is also described in the Protobuf documentation:
If you want to write multiple messages to a single file or stream, it
is up to you to keep track of where one message ends and the next
begins. The Protocol Buffer wire format is not self-delimiting, so
protocol buffer parsers cannot determine where a message ends on their
own. The easiest way to solve this problem is to write the size of
each message before you write the message itself. When you read the
messages back in, you read the size, then read the bytes into a
separate buffer, then parse from that buffer. (If you want to avoid
copying bytes to a separate buffer, check out the CodedInputStream
class (in both C++ and Java) which can be told to limit reads to a
certain number of bytes.)
The bold italic part is where you code isn't correct.
On the write side you should write
the Protobuf's length in some format that is understandable for both sender and receiver (selecting the proper format is especially important when transporting between systems whose endianness is different).
the protobuf
On the receiving end you need to
perform a read with the fixed, known size of the length field
a read for the length learned in step 1. This read will retriev the protobuf.
There's example code here on SO in this question: Sending struct via Socket using JAVA and C++

#fvu: This is my code which I am trying:
import Visualization.DataSetProtos.PacketData; // protos import
import java.io.InputStream;
import java.util.Arrays;
import javax.swing.JFrame;
import javax.swing.JScrollBar;
import javax.swing.JScrollPane;
class WorkerThread extends Thread {
Socket service;
static DynamicData demo;
static int size;
static int times;
static byte[] buffer;
WorkerThread(Socket service)
{
this.service = service;
buffer = new byte[500];
size = 1;
times = 0;
}
static void Print(PacketData packetData)
{
System.out.print("Packet Number: " + (++times));
System.out.print(" DataSet Size: " + packetData.getLength() + "\n");
}
static void Print(PacketHeader packetHeader)
{
System.out.print("Packet Number: " + (++times));
System.out.print(" DataSet Size: " + packetHeader.getLength() + "\n");
}
public void run() {
boolean flag=true; //you can change this flag's condition, to test if the client disconects
if(demo == null)
{
demo = new DynamicData("GridMate Data Visualization");
demo.pack();
RefineryUtilities.centerFrameOnScreen(demo);
//demo.setVisible(true);
}
try
{
while (flag)
{
InputStream inputStream = service.getInputStream();
int read;
read = inputStream.read(buffer);
byte[] readBuffer = new byte[read];
readBuffer = Arrays.copyOfRange(buffer, 0, read);
PacketData packetData = PacketData.parseFrom(readBuffer);
Print(packetData);
}
service.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
}
public class Test
{
Test()
{
server = null;
client= null;
}
public static void main(final String[] args) {
int i =0;
try
{
server = new ServerSocket(25715);
System.out.println("Server setup and waiting for client connection ...");
while(true)
{
client = server.accept();
WorkerThread wt = new WorkerThread(client);
wt.start();
i++;
}
}
catch(IOException e)
{ System.out.println("IO Error in streams " + e);
e.printStackTrace();
}
}
public void finalize()
{
try
{
server.close();
client.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
static ServerSocket server;
static Socket client;
}

Why isn't my Socket closing on server side?

I'm trying to make a simple text editor which can be shared accross multiple terminals at the same time. I have a Server waiting for new users, when a user enters the shared editor it just starts waiting for input characters.
public class Server {
public static final int PORT = 8080;
public static void main(String[] args) throws IOException {
ServerSocket ss = new ServerSocket(PORT);
while (true) {
Socket socket = ss.accept();
System.out.println("A new user entered the sever");
new Thread(() -> serve(socket)).start();
}
}
private static void serve(Socket socket) {
try {
while (!socket.isClosed() && !socket.isInputShutdown()) {
System.out.println("hey " + socket.isClosed() + " " + socket.isInputShutdown());
System.out.print(new String(SocketUtil.receiveBytes(socket,1)));
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
When a user closes the editor, the socket is closed on client side. However, the socket on the server side does not get closed and the server starts looping infinitly in the "wait for input" loop.
The Client is a singleton containing the following methods, called at the openning and closing of the editor.
public static void init() {
try {
if (socket == null) socket = new Socket(HOST,Server.PORT);
} catch (IOException e) {
e.printStackTrace();
kill();
throw new Error(e.getMessage());
}
}
public static void kill() {
Check.notNull(socket);
try {
SocketUtil.terminateCommunication(socket);
System.out.println(socket.isClosed());
} catch (IOException e) {
e.printStackTrace();
}
}
Finally, here are utilitary methods (in SocketUtil) used in both classes :
public static void terminateCommunication(Socket socket) throws IOException {
socket.shutdownInput();
socket.shutdownOutput();
socket.close();
}
public static char[] receiveBytes(Socket socket, int nBytes) throws IOException {
char[] bytes = new char[nBytes];
InputStreamReader isr = new InputStreamReader(socket.getInputStream());
isr.read(bytes);
return bytes;
}
Any idea of why the socket on server side is not closed after the Client gets killed ?

It is not quite clear from the Javadoc, but isClosed() only returns true when you have explicitly called close() on the socket (see the sources to confirm that). You should check for exceptions and the return value of read() instead. If you read -1 or catch an IOException while trying to read (or write, for that matter), it essentially means that the other side has closed the connection, so you should close your socket as well (better to it in a finally block) and you're done with that particular connection. You don't check for -1 in receiveBytes(), but you really should. Perhaps throw a EOFException() if you want to merge these two possibility into one, so that the code up the stack (in serve()) doesn't have to figure out what exactly happened:
public static char[] receiveBytes(Socket socket, int nBytes) throws IOException {
char[] bytes = new char[nBytes];
InputStreamReader isr = new InputStreamReader(socket.getInputStream());
if (isr.read(bytes) == -1)
throw new EOFException();
return bytes;
}
One exception from the IOException rule (sorry for the pun) is the SocketTimeoutException. If you get this, the connection is still alive, and you may just as well retry your read(). But I believe that in order to get these, you must call Socket.setSoTimeout() somewhere, and if you haven't, then you probably shouldn't worry about SocketTimeoutException.
You should also note that read() may sometimes return partial reads (that is, less than bytes.length). If it's important that receiveBytes() reads exactly nBytes (which probably is, since you never return the number of actual characters read), then you should call it in a loop, like this:
int pos = 0;
while (pos < bytes.length) {
int l;
if ((l = isr.read(bytes, pos, bytes.length - pos)) == -1) {
throw new EOFException();
}
pos += l;
}
I know this is cumbersome, which is exactly why many developers create utility methods like your receiveBytes().

The proper way to detect that the client has closed its connection is by checking the reception of 0 bytes.
System.out.print(new String(SocketUtil.receiveBytes(socket,1)));
just check if the string is empty should do the trick.
Note that I am not that familiar with java, but I do know socket programming.
Receiving 0 bytes, checking for that, and closing the socket if you do is a good solution.
You can use exception handling too, but you'll detect that the peer closed it socket an iteration later. Receiving 0 bytes is not really an error condition it is just a signal from the peer that he has closed its end of the socket and won't send anymore data. If you ignore this, and keep using the socket, you'll receive an exception in the next iteration because there is nothing to receive anymore.

why client can not receive message from server (java)

I have just started learning java. I modified the client side code for a server/client communication program, by creating two threads for the client side, main thread for receiving user's input, and inputThread for receiving server's response. I am sure that server has sent the response to client, however, no response message is obtain at client.
Here is my code. Can anyone help me to figure it out? Thanks
package clientnio;
import java.net.*;
import java.nio.*;
import java.io.*;
import java.nio.channels.*;
import java.util.Scanner;
public class ClientNIO {
public static int bufferLen = 50;
public static SocketChannel client;
public static ByteBuffer writeBuffer;
public static ByteBuffer readBuffer;
public static void main(String[] args) {
writeBuffer = ByteBuffer.allocate(bufferLen);
readBuffer = ByteBuffer.allocate(bufferLen);
try {
SocketAddress address = new InetSocketAddress("localhost",5505);
System.out.println("Local address: "+ address);
client=SocketChannel.open(address);
client.configureBlocking(false);
//readBuffer.flip();
new inputThread(readBuffer);
/*
String a="asdasdasdasddffasfas";
writeBuffer.put(a.getBytes());
writeBuffer.clear();
int d=client.write(writeBuffer);
writeBuffer.flip();
*/
while (true) {
InputStream inStream = System.in;
Scanner scan = new Scanner(inStream);
if (scan.hasNext()==true) {
String inputLine = scan.nextLine();
writeBuffer.put(inputLine.getBytes());
//writeBuffer.clear();
System.out.println(writeBuffer.remaining());
client.write(writeBuffer);
System.out.println("Sending data: "+new String(writeBuffer.array()));
writeBuffer.flip();
Thread.sleep(300);
}
}
}
catch(Exception e) {
System.out.println(e);
}
}
}
class inputThread extends Thread {
private ByteBuffer readBuffer;
public inputThread(ByteBuffer readBuffer1) {
System.out.println("Receiving thread starts.");
this.readBuffer = readBuffer1;
start();
}
#Override
public void run() {
try {
while (true) {
readBuffer.flip();
int i=ClientNIO.client.read(readBuffer);
if(i>0) {
byte[] b=readBuffer.array();
System.out.println("Receiving data: "+new String(b));
//client.close();
//System.out.println("Connection closed.");
//break;
}
Thread.sleep(100);
}
}
catch (Exception e) {
System.out.println(e);
}
}
}

Disclaimer: I'm not an active user of Java. (I only used it in school.)
Advice: I think it will greatly simplify the debugging process if you use blocking mode, at least until your code example is working correctly. (Currently your code does not seem to benefit from the non-blocking mode.)
I have identified two issues, culminating into four possible lines of code that may require changing:
When a ByteBuffer allocates its backing array, it sets itself ready to write by setting position to zero and limit to the capacity of that array. Your two uses of ByteBuffer.flip() (in the writing loop and the reading loop respectively) seem to be contrary to the convention.
Calling the ByteBuffer.array() method always returns the whole backing array, thus it always has size bufferLen. Because of this, a String constructed from the full-size array may contain junk from a previous transmission.
Typically, the array needs to be trimmed to the transmission size, and the conversion between a String and a byte array must use the same encoding as the server.
My suggested changes for first issue: (Note: I don't know how to fix the array trimming and encoding issue.)
writeBuffer.put(inputLine.getBytes());
writeBuffer.flip(); // <--here
client.write(writeBuffer);
...
writeBuffer.clear(); // <-- should be clear() instead of flip()
Thread.sleep(300);
// readBuffer.flip(); // <-- remove this line
int i=ClientNIO.client.read(readBuffer);
if(i>0) {
readBuffer.flip(); // <-- move it here
byte[] b=readBuffer.array();
System.out.println("Receiving data: "+new String(b));
...
}
References
http://docs.oracle.com/javase/1.4.2/docs/api/java/nio/ByteBuffer.html
http://docs.oracle.com/javase/1.4.2/docs/api/java/nio/channels/SocketChannel.html
Socketchannel always null
http://www.exampledepot.com/egs/java.nio.charset/ConvertChar.html

Calling flip() on a buffer prior to reading it is wrong. Don't do that. You need to flip it prior to writing from it, or getting from it, and compact() afterwards.

Help needed with asynchronous Java NIO using Executors.newFixedThreadPool

Hey guys, I'm working on a server program that is meant to scale well and serve potentially thousands of clients. The thing is, I feel that Apache MINA is too heavyweight so I decided to not use it and wrote my own client listener instead. I never really performed asynchronous socket operations in Java (C# made that so much easier, but I really preferred to write this project in Java since I'm more familiar with it in everything besides socket reads), so trying to understand how to use the thread pool correctly is hard for me. I used Apache MINA documentation to get an idea of how things should be done. I got two questions:
Is the thread pool used correctly? Apache MINA's default thread size is the number of CPU cores + 1, but should I really use a 3 thread thread pool for my Core 2 Duo in order to accept thousands of clients?
I know that reallocating the buffer twice for each message received from the client (each message is two packets, one header that is a constant 4 bytes and a content packet that has its length specified in the header). Is there an easy way to use a fixed size buffer that checks for buffer overruns so that behavior is still the same but the buffer doesn't have to be constantly reallocated?
Here's how I start the listener:
ClientListener cl = new ClientListener(1234);
cl.init();
new Thread(cl).start();
Here is the relevant code for ClientListener:
private static final int THREADS = Runtime.getRuntime().availableProcessors() + 1;
private ServerSocket socket;
private ExecutorService threadPool;
private int port;
public ClientListener(int port) {
this.port = port;
threadPool = Executors.newFixedThreadPool(THREADS);
}
public void init() {
try {
socket = new ServerSocket(port);
} catch (IOException ex) {
}
}
public void run() {
while (true) {
try {
ClientSession s = new ClientSession(socket.accept());
threadPool.execute(s);
} catch (IOException ex) {
}
}
}
ClientSession relevant code:
private Socket socket;
private byte[] buffer;
private boolean isHeader;
public ClientSession(Socket socket) {
this.socket = socket;
this.buffer = new byte[4];
this.isHeader = true;
}
public void run() {
InputStream in;
try {
in = socket.getInputStream();
out = socket.getOutputStream();
} catch (IOException ex) {
return;
}
while (!socket.isClosed()) {
try {
int read = in.read(buffer);
if (read == -1)
break;
receive(read);
} catch (IOException ex) {
break;
}
}
}
private void receive(int readBytes) {
if (isHeader) {
if (readBytes >= 4) {
buffer = new byte[getPacketLength(buffer)];
isHeader = false;
} else {
System.out.println("Not enough data received from client " + socket.getInetAddress() + " to decode packet.");
}
} else {
if (readBytes >= buffer.length) {
processMessage(new LittleEndianByteArrayReader(decryptData(buffer)), this);
buffer = new byte[4];
isHeader = true;
} else {
System.out.println("Not enough data received from client " + socket.getInetAddress() + " to decode packet (needed " + buffer.length + ", received " + readBytes + ").");
}
}
}
You don't need to know the code for getPacketLength, processMessage, decryptData, and the class LittleEndianByteArrayReader, but I'm pretty sure the purposes of those methods/classes are obvious.

The number of threads in blocking IO scenario have to be calculated by the number of clients and the time each client connection will be open.
Each connection of each user requires on thread.
With only three threads a user could simply block your server until connection timeout by just opening three TCP connections and not sending any data to your server.

Nevermind guys. I realized that Apache MINA actually uses NIO which is why I got confused. It really needs only one thread to process requests with the use of selectors. Thanks for all your answers and sorry about the confusion!