Develop Reference

Am I reading the Arduino incoming data the wrong way using jSerialComm?

Im writing a JavaFX application that has to comunicate with my Arduino UNO. For this im using the jSerialComm library.
Just for testing puroposes I've upload a very simple sketch to my Arduino that prints to the Serial a "Hello" word every 2 seconds:
void setup() {
//put your setup code here, to run once:
Serial.begin(9600);
}
void loop() {
// put your main code here, to run repeatedly:
delay(2000);
Serial.print("Hello");
}
And in my JavaFX scene im reading the incoming data with this:
public void setDevice(SerialPort device) {
this.device = device;
device.openPort();
device.addDataListener(new SerialPortDataListener() {
#Override
public int getListeningEvents() {
return SerialPort.LISTENING_EVENT_DATA_RECEIVED;
}
#Override
public void serialEvent(SerialPortEvent serialPortEvent) {
if (serialPortEvent.getEventType() == SerialPort.LISTENING_EVENT_DATA_RECEIVED){
byte [] data = serialPortEvent.getReceivedData();
String msg = new String(data);
System.out.println(msg);
}
}
});
}
I can read the data from Arduino, but it comes in a strange way. Like the string is sended in 2 diffent strings. Here is an image of the console output:
Am I doing something wrong? Thank you very much!

You are using jSerialComm in default = NonBlocking mode. So what happens is (as pseudo code steps)
LISTENING_EVENT_DATA_RECEIVED triggered
get the char H
get the char e
Because we are nonBlocking we have to move on in the program
Print what we have so far -> HE
.... do other stuff or check if other stuff has to be processed
LISTENING_EVENT_DATA_RECEIVED triggered
get the char L
get the char L
get the char O
Because we are nonBlocking we have to move on in the program
Print what we have so far -> LLO
.... do other stuff or check if other stuff has to be processed
So you could do 2 things -> change the mode to e.g. blocking while receiving (ensure a proper timeout to prevent deadlocks) or rewrite the function (my preferred way) to check for end terminators in the communication flow and then process the content of the buffer (which should be implemented in a non-blocking way)

Runnable no longer runs in Service to periodically retrieve from URL

I'm working on an Android app that runs on remotely-sited, unattended devices. It has a Service to periodically retrieve settings from a fixed URL. It does this by daisy-chaining posts of a Runnable to a Handler obtained from a HandlerThread.getLooper(): it does an initial post() after creating the Handler in the Service's startup, and thereafter it does a postDelayed() at the end of each run through the Runnable.
This works "forever" on some devices in the project (fetching the settings successfully every 30 seconds for thousands of hours), but on other devices it stops working after a variable amount of time (hours to tens of hours) and thereafter never works again. I've even added code to monitor the Service's activity and restart the process by issuing a new post() if it's been more than 5 minutes since the last run, and that code kicks in when as intended when the process stalls, but still the Runnable no longer runs.
Some pared-down code snippets, and then further discussion of the problem/symptoms:
public class SettingsMonitor extends Service {
private HandlerThread mHandlerThread = new HandlerThread("SettingsMonitorHandler");
private Handler mSettingsMonitorHandler;
private long mLastTryMillis = 0;
#Override
public int onStartCommand(Intent intent, int flags, int startId) {
mHandlerThread.start();
mSettingsMonitorHandler = new Handler(mHandlerThread.getLooper());
mSettingsMonitorHandler.post(checkSettings);
return START_STICKY;
}
final Runnable checkSettings = new Runnable()
{
#Override
public void run() {
Log.d(TAG, "checking latest settings");
mLastTryMillis = System.currentTimeMillis();
getSettingsFromUrl(SETTINGS_URL);
// queue up next check
mSettingsMonitorHandler.postDelayed(checkSettings, appSettings.getSettingsMonitorIntervalMillis());
}
};
public void restartCheckingIfStalled() {
long settingsMonitorIntervalMillis = appSettings.getSettingsMonitorIntervalMillis();
long maxStalledMillis = 10 * settingsMonitorIntervalMillis;
long millisSinceLastCheck = System.currentTimeMillis() - mLastTryMillis;
if (millisSinceLastCheck > maxStalledMillis) {
mSettingsMonitorHandler.removeCallbacksAndMessages(null);
Log.d(TAG, "checkSettings stalled -- restarting...");
mSettingsMonitorHandler.post(checkSettings);
}
}
}
The app logs extensive debug info, including every exception caught in local catch blocks (e.g., inside getSettingsFromUrl()) as well as any exception caught in a global app-level UncaughtExceptionHandler -- there are no exceptions being thrown by the app. There are log entries made at many points in getSettingsFromUrl(), as well as the first line of the Runnable, but at some point all of these log entries cease to appear, so I infer that the Runnable itself is no longer actually running.
I monitor the Service from elsewhere in the app and so I know that it is running throughout the lifetime of the app (even after the Runnable no longer runs); at the same time, I call restartCheckingIfStalled() and I do see the log entries indicating that the re-start is being requested, but apparently the post() it issues does not cause the Runnable to run again.
It's a bit difficult to quantify, but it appears that this problem is occurring on devices that are showing other signs of having network connection problems (some are on wi-fi, some are on sim cards). I don't really know what to make of that possible correlation, though.
I am mystified that this process goes off the rails without any sign of trouble -- no exception is thrown, and the last successful run through the Runnable and getSettingsFromUrl() look completely normal. I am looking for suggestions about ways to isolate, identify, and ultimately fix the problem, because I've exhausted a bunch of ideas/tests and I'm still no closer to understanding it.
#x-code: Ok, here's getSettingsFromUrl() -- I don't (explicitly) set a timeout (I'll go search now for whether there is any kind of default). How might lack of timeout relate to or explain the symptoms I'm describing?
private boolean getSettingsFromUrl(String settingsUrl) {
Log.d(TAG, "trying settings file: " + settingsUrl);
try {
URL url = new URL(settingsUrl);
Scanner scanner = new Scanner(url.openStream());
while (scanner.hasNextLine()) {
String s = scanner.nextLine().split(SETTINGS_FILE_COMMENT_CHAR)[0].trim(); // remove any comments and trim leading/trailing whitespace
if (s.length() > 0) { // skip blank lines
AdminRequest req = new AdminRequest(s);
if (req.isSetReq()) { // only apply SET requests, not GET (or other nonsense)
String reqType = req.getReqType();
if (settingsToApply.containsKey(reqType)) { // override earlier settings with later
settingsToApply.remove(reqType);
}
settingsToApply.put(reqType, req);
}
}
}
scanner.close();
return true;
}
catch (FileNotFoundException e) {
Log.d(TAG, "settings file not found at " + settingsUrl);
}
catch (Exception e) {
Log.d(TAG, "error checking settings: " + Log.getStackTraceString(e));
}
return false;
}
This may beg further questions (about whether/how I use this function's return value -- which I do; about what these AdminRequests are and what I'm doing with them in the settingsToApply HashMap, etc). I'm willing to continue providing more info if it seems likely to lead to insight, but it's not clear to me why, at a minimum, I don't see the first "checking latest settings" message at the top of checkSettings()... how would a problem (e.g., lack of a timeout in accessing the URL) affect subsequent calls to checkSettings()? Do all invocations of checkSettings() run on the same thread, and it's tied up waiting "forever" to access the URL? If that's the case, why is restartCheckingIfStalled() not also blocked? I do get the Log.d() messages from that.
Edited to show the changes I made to add timeouts, which solved the problem per the responses I received: in getSettingsFromUrl() change these two lines
URL url = new URL(settingsUrl);
Scanner scanner = new Scanner(url.openStream());
to these:
URL url = new URL(settingsUrl);
URLConnection cxn = url.openConnection();
cxn.setConnectTimeout(URL_CONNECT_TIMEOUT_MILLIS);
cxn.setReadTimeout(URL_READ_TIMEOUT_MILLIS);
Scanner scanner = new Scanner(cxn.getInputStream());

Do all invocations of checkSettings() run on the same thread
Yes.
If that's the case, why is restartCheckingIfStalled() not also blocked?
Because it runs on your app's main (ui) thread.
The advantage of running a HandlerThread in a Service is that each message or Runnable that you post to it is placed in a queue, so you don't have to write thread-safe code.
One disadvantage is that this kind of blocking can occur. As GreyBeardedGeek points out, you should set a timeout other than 0 on your URL connection.

I suspect that the default connectionTimeout is 0 (wait forever).
And if the connection blocks 'forever', your handler thread will block as well - your subsequent posts will have no effect.
You should probably set both the connection timeout and the read timeout.

ZeroMQ: Disappearing messages

We have a Java application which is acting as a server. Client applications (written in C#) are communicating with it using ZeroMQ. We are (mostly) following the Lazy Pirate pattern.
The server has a Router socket, implemented as follows (using JeroMQ):
ZContext context = new ZContext();
Socket socket = context.createSocket(ZMQ.ROUTER);
socket.bind("tcp://*:5555");
The clients connect and send messages like this:
ZContext context = ZContext.Create();
ZSocket socket = ZSocket.Create(context, ZSocketType.REQ);
socket.Identity = Encoding.UTF8.GetBytes("Some identity");
socket.Connect("tcp://my_host:5555");
socket.Send(new ZFrame("request data"));
We have experienced lost messages when multiple clients are sending messages at the same time. With a single client, there doesn't appear to be any problem.
Are we implementing this the right way for a multiple-client-single-server setup?
Update: Example client and server exhibiting this behaviour:
Server:
import org.zeromq.ZContext;
import org.zeromq.ZMQ;
import org.zeromq.ZMQ.PollItem;
import org.zeromq.ZMQ.Poller;
import org.zeromq.ZMQ.Socket;
import org.zeromq.ZMsg;
public class SimpleServer
{
public static void main(String[] args) throws InterruptedException
{
ZContext context = new ZContext();
Socket socket = context.createSocket(ZMQ.ROUTER);
socket.setRouterMandatory(true);
socket.bind("tcp://*:5559");
PollItem pollItem = new PollItem(socket, Poller.POLLIN);
int messagesReceived = 0;
int pollCount = 0;
while ((pollCount = ZMQ.poll(new PollItem[]{pollItem}, 3000)) > -1)
{
messagesReceived += pollCount;
for (int i = 0 ; i < pollCount ; i++)
{
ZMsg msg = ZMsg.recvMsg(socket);
System.out.println(String.format("Received message: %s. Total messages received: %d", msg, messagesReceived));
}
if (pollCount == 0)
{
System.out.println(String.format("No messages on socket. Total messages received: %d", messagesReceived));
}
}
}
}
Client:
using NetMQ;
using System;
using System.Text;
namespace SimpleClient
{
class Program
{
static byte[] identity = Encoding.UTF8.GetBytes("id" + DateTime.UtcNow.Ticks);
static void Main(string[] args)
{
for (int i = 0; i < 100; i++)
{
SendMessage();
}
}
private static void SendMessage()
{
using (NetMQContext context = NetMQContext.Create())
{
using (NetMQSocket socket = context.CreateRequestSocket())
{
socket.Options.Identity = identity;
socket.Connect("tcp://localhost:5559");
socket.Send(Encoding.UTF8.GetBytes("hello!"));
}
}
}
}
}
If I run the server and a single client, I can see all my 100 messages arrive. If I run, say, 5 clients simultaneously, I only get around 200 -> 300 messages arrive, instead of the full 500. As an aside, it appears that closing the socket in the client is somehow stopping the router socket on the server from receiving messages briefly, although this is just a theory.

Part 1 - poll may return more than one event
ZMQ.poll() returns the number of events that were found:
int rc = ZMQ.poll(new PollItem[]{pollItem}, 3000);
You currently assume that one return from poll is one event. Instead, you should loop over ZMsg msg = ZMsg.recvMsg(socket); for the number of events that are indicated by the return of ZMQ.Poll().
From the source of JeroMQ:
/**
* Polling on items. This has very poor performance.
* Try to use zmq_poll with selector
* CAUTION: This could be affected by jdk epoll bug
*
* #param items
* #param timeout
* #return number of events
*/
public static int zmq_poll(PollItem[] items, long timeout)
{
return zmq_poll(items, items.length, timeout);
}
Part 2 - ZMsg.receive() may return multiple frames
When you receive a ZMsg from ZMsg msg = ZMsg.recvMsg(socket);, the ZMsg may contain multiple ZFrames, each containing client data.
From the comments of the ZMsg class in JeroMQ's source:
* // Receive message from ZMQSocket "input" socket object and iterate over frames
* ZMsg receivedMessage = ZMsg.recvMsg(input);
* for (ZFrame f : receivedMessage) {
* // Do something with frame f (of type ZFrame)
* }
Part 3 - messages can be split across multiple ZFrames
From ZFrame's source in JeroMQ:
* The ZFrame class provides methods to send and receive single message
* frames across 0MQ sockets. A 'frame' corresponds to one underlying zmq_msg_t in the libzmq code.
* When you read a frame from a socket, the more() method indicates if the frame is part of an
* unfinished multipart message.
If I'm understanding this correctly, then for each event you may get multiple frames, and one client message may map to 1..N frames (if the message is big?).
So to summarize:
One return from poll may indicate multiple events.
One event and thus one ZMsg.receive() may contain multiple frames
One frame could contain one complete client message or only part of a client message; one client message maps to 1..N frames.

Unfortunately we couldn't solve this particular issue, and have moved away from using ZeroMQ for this interface. In case it helps anyone else, the only things we worked out for definite is that rapidly opening/closing the request sockets caused undesirable behaviour (lost messages) on the router socket end. The problem was exacerbated by a poorly performing server CPU, and didn't appear at all when the server was on a fast multi-core machine.

Unfortunatley I was not even close working with ZMQ at the time this question was active. But I had the same problem today and found this page. And your answer (not using ZMQ) was not satisfying for me. So I searched a bit more and finally found out what to do.
Just as a reminder: this works with the "POLLER" in ZMQ [1]
If you use "PAIR" connection you will for sure do NOT lose nay files, BUT send/recive takes approx. the same time. So you can not speed up and was not a solution for me.
Solution:
in zmq_setsockopt (python: zmq.setsockopt) you can set ZMQ_HWM (zmq.SNDHWM, zmq.RCVHWM) to '0' [2]
in python: sock.setsockopt(zmq.SNDHWM , 0) resp. sock.setsockopt(zmq.RCVHWM, 0) for the Sender resp. Reciver
note: i think notation changed from HWM to SNDWHM/RCVHWM
HWM = 0 means that there is "NO limit" for the number of messages (so be careful, maybe set a (hvery high) limit)
there is also ZMQ_SNDBUF/ ZMQ_RCVBUF (python: zmq.SNDBUF/zmq.RCVBUF) which you can give as well, ie. sock.setsockopt(zmq.RCVBUF, 0) resp. ..... [2]
so this will set the operating system "SO_RCVBUF" to default (here my knowledge ends)
setting this parameter or not did NOT influence my case but I think it might
Performance:
So with this I could "send" 100'000 files with 98kB in ~8s (~10GB): this will fill your RAM (if this is full I think your program will slow down), see also picture
in the mean time I "recived" and saved the files in about ~enter image description here118s and freeing the RAM again
Also, with that I NERVER lost a file up to now. (you might if you hit the limits of your PC)
data loss is "GOOD":
if you realy NEED all the data you should use this method
if you can regard some losses are fine (e.g. live plotting: as long as your FPS > ~50 you will smoothly see the plots and you do not care if you lose something)
--> you can save RAM and avoid blocking your whole PC!
Hope this post helps for the next person coming by...
[1]: https://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/multisocket/zmqpoller.htm
[2]: http://api.zeromq.org/2-1:zmq-setsockopt
You find a Picture of the RAM her:
RAM is loading in about 8s. Afterwords the disk is saving the files from the buffer

Android - How to listen for Bluetooth output stream to be emptied

i've got two threads which continuously manage I/O operations on the streams of a BluetoothSocket.
For the input, no problem.. the InputStream.read() method is blocking and everything's fine.
For the output.. here it comes the problem.
public void write(final byte[] buffer) {
synchronized (this) {
try {
this.mOutStream.write(buffer);
this.mHandler.sendEmptyMessage(Constants.MESSAGE_SENT);
} catch (IOException e) {
Log.e(LOGTAG, "Exception during write", e);
}
}
}
I use the code above for writing messages on the output stream. It's synchronized on the instance of the thread so that nobody else will be able to write on the buffer competitively.
The problem is the following: if I try to write two messages, the one immediately after the other, the first one seems to be skipped by the receiving device.
Reading what OutputStream.write(byte[]) does, I understand what's the main reason: it's like calling OutputStream(byte[], 0, length), so the '0' offset will overwrite the previous message.
But I cannot set timers or timeouts every time I need to write two messages sequentially, just to be sure that in the meantime the first message has already been read by the recipient.
I want something more, something like this:
int offset = 0;
this.mOutputStream.setOnStreamEmptiedCallback(new OnStreamEmptiedCallback() {
public void onStreamEmptied() {
this.offset = 0;
}
});
public void write(byte[] buffer) {
this.mOutputStream.write(buffer, 0, buffer.length);
this.offset += buffer.length;
}
I know that the setOnStreamEmptiedCallback doesn't exist, but.. Is there a way to do something similar? Something to be sure that I will never overwrite the stream content?
Thanks everybody..

flush() should correct your problem.
this.mOutStream.write(buffer);
this.mOutStream.flush();

Check servers for active Webserver fast (multithreaded)

I want to check an huge amount (thousands) of Websites, if they are still running. Because I want to get rid of unececarry entries in my HostFile Wikipage about Hostfiles.
I want to do it in a 2 Stage process.
Check if something is running on Port 80
Check the HTTP response code (if it's not 200 I have to check the site)
I want to multithread, because if I want to check thousands of addresses, I cant wait for timeouts.
This question is just about Step one.
I have the problem, that ~1/4 of my connect attempts don't work. If I retry the not working ones about ~3/4 work? Do I not close the Sockets correctly? Do I run into a limit of open Sockets?
Default I run 16 threads, but I have the same problems with 8 or 4.
Is there something I'm missing
I have simplified the code a little.
Here is the code of the Thread
public class SocketThread extends Thread{
int tn;
int n;
String[] s;
private ArrayList<String> good;
private ArrayList<String> bad;
public SocketThread(int tn, int n, String[] s) {
this.tn = tn;
this.n = n;
this.s = s;
good = new ArrayList<String>();
bad = new ArrayList<String>();
}
#Override
public void run() {
int answer;
for (int i = tn * (s.length / n); i < ((tn + 1) * (s.length / n)) - 1; i++) {
answer = checkPort80(s[i]);
if (answer == 1) {
good.add(s[i]);
} else {
bad.add(s[i]);
}
System.out.println(s[i] + " | " + answer);
}
}
}
And here is the checkPort80 Method
public static int checkPort80(String host)
Socket socket = null;
int reachable = -1;
try {
//One way of doing it
//socket = new Socket(host, 80);
//socket.close();
//Another way I've tried
socket = new Socket();
InetSocketAddress ina = new InetSocketAddress(host, 80);
socket.connect(ina, 30000);
socket.close();
return reachable = 1;
} catch (Exception e) {
} finally {
if (socket != null) {
if (socket.isBound()) {
try {
socket.close();
return reachable;
} catch (Exception e) {
e.getMessage();
return reachable;
}
}
}
}
}
About Threads, I make a ArrayList of Threads, create them and .start() them and right afterwards I .join() them, get the "God" and the "Bad" save them to files.
Help is appreciated.
PS: I rename the Hosts-file first so that it doesn't affect the process, so this is not an issue.
Edit:
Thanks to Marcelo Hernández Rishr I discovered, that HttpURLConnection seems to be the better solution. It works faster and I can also get the HttpResponseCode, which I was also interested anyways (just thought it would be much slower, then just checking Port 80). I still after a while suddenly get Errors, I guess this has to do with the DNS server thinking this is a DOS-Attack ^^ (but I should examine futher if the error lies somewhere else) also fyi I use OpenDNS, so maybe they just don't like me ^^.
x4u suggested adding a sleep() to the Threads, which seems to make things a little better, but will it help me raise entries/second i don't know.
Still, I can't (by far) get to the speed I wanted (10+ entries/second), even 6 entries per second doesn't seem to work.
Here are a few scenarios I tested (until now all without any sleep()).
number of time i get first round how many entries where entries/second
threads of errors processed until then
10 1 minute 17 seconds ~770 entries 10
8 3 minute 55 seconds ~2000 entries 8,51
6 6 minute 30 seconds ~2270 entries 5,82
I will try to find a sweet spot with Threads and sleep (or maybe simply pause all for one minute if I get many errors).
Problem is, there are Hostfiles with one million entries, which at one entry per second would take 11 Days, which I guess all understand, is not expectable.
Are there ways to switch DNS-Servers on the fly?
Any other suggestions?
Should I post the new questions as separate questions?
Thanks for the help until now.
I'll post new results in about a week.

I have 3 suggestions that may help you in your task.
Maybe you can use the class HttpURLConnection
Use a maximum of 10 threads because you are still limited by cpu, bandwidth, etc.
The lists good and bad shouldn't be part of your thread class, maybe they can be static members of the class were you have your main method and do static synchronized methods to add members to both lists from any thread.

Sockets usually try to shut down gracefully and wait for a response from the destination port. While they are waiting they are still blocking resources which can make successive connection attempts fail if they were executed while there have still been too many open sockets.
To avoid this you can turn off the lingering before you connect the socket:
socket.setSoLinger(false, 0);