I have producers that takes data A ,produce data B and send it
public interface Producer<T>{
void produce(T data);
void flush();
}
public class DataBaseProducer implements Producer<String>{
List<String> producedData = new ArrayList<>();
// create data
public void produce(String data){
producedData.add(transformData(data));
}
// send created data
public void flush(){
sendDataToDatabase(producedData);
}
}
public class MessageProducer implements Producer<String>{
public void produce(String data){
String line =transformData(data)
sendDataToMessageQueue(line);
}
public void flush(){
}
}
public static void main(String[] args) {
// get producer
Producer producer = getProducer(producerName)
BufferedReader reader;
try {
reader = new BufferedReader(new FileReader(file..));
String line = reader.readLine();
while (line != null) {
producer.produce(line)
line = reader.readLine();
}
reader.close();
producer.flush()
} catch (IOException e) {
e.printStackTrace();
}
}
To demonstrate my question, imagine i have producers like above. One loads all data, and then bulk send it at once after it done, and second sends data right away ( so it does not bulk send data, but whenever new data is createdit sends it right away)
Most of the producers will send the data after every data is loaded, but few of them will send them right away. If the producer sends data right away, than the flush() method remains empty. This however seems like bad pratcise and may violates some OOP principles. What is the correct way to implement this?
Thanks for answers!
In my opinion, and after checking the main() code, I think that MessageProducer is not really a Producer because the method produce actually produces and flushes at the same time.
But since it seems you need to treat what you get from getProducer(producerName) the same way independently, then what you may do is something like this:
public class MessageProducer implements Producer<String> {
private boolean flushNeeded;
private String line;
public void produce(String data){
line =transformData(data)
flushNeeded = true;
flush();
}
public void flush(){
if (flushNeeded) {
sendDataToMessageQueue(line);
flushNeeded = false;
}
}
}
But in my humble opinion, it doesn't really make any significant different with leaving the flush() implementation empty, if not simply to make the code a bit clearer to readers (reading an empty flush() may in fact induce in error thinking your implementation never flushes, while it does it but just into the produce method).
Related
I have an Android application that receives ASCII strings (so every character in the string corresponds to exactly one byte) from a BLE device in thread A.
These strings come in chunks with a maximum length. For example, lets say that the max length is 4, and we receive the following strings:
"ABCD" (4), "EFGH" (4), "I\r\n" (3)
At the other hand, I have another thread B that needs to read these strings but as a complete line. In the example, after receiving all three packets, this thread should read a line:
"ABCDEFGHI"
My first bet was to implement a custom InputStream and OutputStream using a common underlying BlockingQueue. Then using an OutputStreamWriter to write incoming strings in thread A and an InputStreamReader wrapped inside a BufferedStream to use the readLine() function from thread B, but it is not working.
I can see that bytes (chunks) are added to the queue when using the custom OutputStream on thread A but when I call readLine() from thread B, it blocks and never returns a string even when I know a full line has been added to the underlying queue.
I'm pretty sure I'm reinventing the wheel here and I've been unable to find a definitive answer searching the Web. There must be a better way to do this in Java/Android. It sounds like a very common pattern.
I mostly do things in C# so there might be some class(es) I'm missing. I took a look at ByteBuffer also but it seems that going this way forces me to implement my own readLine() function because there is no InputStream to be used by BufferedReader, etc.
You can easily send data between threads with Greenrobot's EventBus.
Greenrobot's EventBus is a library that allows communication between components (Activity, Fragment, Services and backgrounds threads).
build.gradle
dependencies {
compile 'org.greenrobot:eventbus:3.0.0'
}
1. LISTENER (Thread A)
public class BleListener{
private static Context _context;
private static BleListener _instance;
private static ListenerThread _listenerThread;
private static boolean _isListenerThreadEnable = false;
private BleListener(Context context){
_context = context;
// set ble config and open ble port in here
// ....
// enable listener thread
if (!_isListenerThreadEnable) {
_listenerThread = new ListenerThread();
_listenerThread.start();
_isListenerThreadEnable = true;
}
}
// call this function from outer class
public static BleListener getInstance(Context context) {
if (_instance == null) {
_instance = new BleListener(Context context);
}
return _instance;
}
private class ListenerThread extends Thread {
ListenerThread() {
// setting your receive buffer, thread priority in here
}
#Override
public void run() {
while (_isListenerThreadEnable) {
synchronized (_bleDevice) {
int _receivedCount = _bleDevice.getQueueStatus();
while (_receivedCount > 0) {
// append your received data in here with ByteBuffer or StringBuffer
// ..
// parsing data for get valid data
// ..
// send valid data out when receive special character (end of message flag) or when timeout received with EventBus
EventBus.getDefault().post( ValidModal);
}
}
Thread.Yield();
}
}
}
}
2. MAIN (Thread B - Read data from Thread A)
Subscribers also need to register themselves to and unregister from the bus. Only while subscribers are registered, they will receive events. In Android, in activities and fragments you should usually register according to their life cycle. For most cases onStart/onStop works fine:
#Override
public void onStart() {
super.onStart();
EventBus.getDefault().register(this);
}
#Override
public void onStop() {
EventBus.getDefault().unregister(this);
super.onStop();
}
Subscribers implement event handling methods (also called “subscriber methods”) that will be called when an event is posted. These are defined with the #Subscribe annotation.
#Subscribe(threadMode = ThreadMode.MAIN)
public void onMessage(ValidModal) {
// You will get valid data from thread A here.
//..
}
As recommended by Ted Hopp I finally used a PipedInputStream and PipedOutputStream (wrapped inside OutputStreamWriter and BufferedReader).
It works like a charm and does exactly what I needed. Thank you!
When I run the below locally (on my own computer) it works fine - I can send messages to it and it reads them in properly. As soon as I put this on a remote server and send a message, only half the message gets read.
try {
this.asynchronousServerSocketChannel = AsynchronousServerSocketChannel.open().bind(new InetSocketAddress(80));
this.asynchronousServerSocketChannel.accept(null, new CompletionHandler<AsynchronousSocketChannel, Void>() {
#Override
public void completed(AsynchronousSocketChannel asynchronousSocketChannel, Void att) {
try {
asynchronousServerSocketChannel.accept(null, this);
ByteBuffer byteBuffer = ByteBuffer.allocate(10485760);
asynchronousSocketChannel.read(byteBuffer).get(120000, TimeUnit.SECONDS);
byteBuffer.flip();
System.out.println("request: " + Charset.defaultCharset().decode(byteBuffer).toString());
} catch (CorruptHeadersException | CorruptProtocolException | MalformedURLException ex) {
} catch (InterruptedException | ExecutionException | TimeoutException ex) {
}
}
#Override
public void failed(Throwable exc, Void att) {
}
});
} catch (IOException ex) {
}
I've looked around at other questions and tried some of the answers but nothing worked so far. I thought the cause might be that it's timing out due to it being slower over the network when it's placed remotely but increasing the timeout didn't resolve the issue. I also considered that the message might be too large but allocating more capacity to the ByteBuffer didn't resolve the issue either.
I believe your issue is with the Asynchronous nature of the code you're using. What you have is an open connection and you've called the asynchronous read method on your socket.
This reads n bytes from the channel where n is anything from 0 to the size of your available buffer.
I firmly believe that you have to read in a loop. That is, with Java's A-NIO; you'd need to call read again from your completed method on your CompletionHandler by, possibly, passing in the AsynchronousSocketChannel as an attachment to a new completed method on a CompletionHandler you create for read , not the one you already have for accept methods.
I think this is the same sort of pattern you'd use where you'd call accept again with this as the completion handler from your completed method in the CompletionHandler you're using for the accept method call.
It then becomes important to put an "Escape" clause into your CompletionHandler for instance, if the result is -1 or if the ByteBuffer had read X number of bytes based on what you're expecting, or based on if the final byte in the ByteBuffer is a specific message termination byte that you've agreed with the sending application.
The Java Documentation on the matter goes so far as to say the read method will only read the amount of bytes on the dst at the time of invocation.
In Summary; the completed method call for the handler for the read seems to execute once something was written to the channel; but if something is being streamed you could get half of the bytes, so you'd need to continue reading until you're satisfied you've got the end of what they were sending.
Below is some code I knocked together on reading until the end, responding whilst reading, asynchronously. It, unlike myself, can talk and listen at the same time.
public class ReadForeverCompletionHandler implements CompletionHandler<Integer, Pair<AsynchronousSocketChannel, ByteBuffer>> {
#Override
public void completed(Integer bytesRead, Pair<AsynchronousSocketChannel, ByteBuffer> statefulStuff) {
if(bytesRead != -1) {
final ByteBuffer receivedByteBuffer = statefulStuff.getRight();
final AsynchronousSocketChannel theSocketChannel = statefulStuff.getLeft();
if (receivedByteBuffer.position()>8) {
//New buffer as existing buffer is in use
ByteBuffer response = ByteBuffer.wrap(receivedByteBuffer.array());
receivedByteBuffer.clear(); //safe as we've not got any outstanding or in progress reads, yet.
theSocketChannel.read(receivedByteBuffer,statefulStuff,this); //Basically "WAIT" on more data
Future<Integer> ignoredBytesWrittenResult = theSocketChannel.write(response);
}
}
else {
//connection was closed code
try {
statefulStuff.getLeft().shutdownOutput(); //maybe
}
catch (IOException somethingBad){
//fire
}
}
}
#Override
public void failed(Throwable exc, Pair<AsynchronousSocketChannel, ByteBuffer> attachment) {
//shout fire
}
The read is originally kicked off by a call from the completed method in the handler from the very original asynchronous accept on the server socket like
public class AcceptForeverCompletionHandler implements CompletionHandler<AsynchronousSocketChannel, Pair<AsynchronousServerSocketChannel, Collection<AsynchronousSocketChannel>>> {
private final ReadForeverCompletionHandler readForeverAndEverAndSoOn = new ReadForeverCompletionHandler();
#Override
public void completed(AsynchronousSocketChannel result, Pair<AsynchronousServerSocketChannel, Collection<AsynchronousSocketChannel>> statefulStuff) {
statefulStuff.getLeft().accept(statefulStuff, this); //Accept more new connections please as we go
statefulStuff.getRight().add(result); //Collect these in case we want to for some reason, I don't know
ByteBuffer buffer = ByteBuffer.allocate(4098); //4k seems a nice number
result.read(buffer, Pair.of(result, buffer ),readForeverAndEverAndSoOn); //Kick off the read "forever"
}
#Override
public void failed(Throwable exc, Pair<AsynchronousServerSocketChannel, Collection<AsynchronousSocketChannel>> attachment) {
//Shout fire
}
}
I have this really simple JCSP(Java Communicating Sequential Processes) code sample in which I'm trying to write an integer to a One2OneInt channel and then read it.
package jcsp;
import org.jcsp.lang.*;
public class JCSP {
public static void main(String[] args) {
One2OneChannelInt chan = Channel.one2oneInt();
chan.out().write(5);
System.out.println("Written...");
System.out.println(chan.in().read());
}
}
It seems that value never gets written on the channel and program just keeps running. "Written..." is never printed out.
So I learned about BlockingQueue and its implementation SynchronousQueue. As stated here, SynchronousQueue works in similar way in which CSP Channels work. This helped me realize what was wrong with my code. Simply put, you can't write and read from channel in same process. Channel is way for processes to communicate.
Similarly to SynchronousQueue's put() which will wait for other process to call take(), CSP Channel's write() which will wait for corresponding read() to be called. The difference is that CSP Channels have objects ChannelOutput and ChannelInput through which objects are written and red. Conversely, you can call put and take directly on instance of SynchronousQueue. Personally, I find SynchronousQueue much easier to understand, which probably relates to JCSP not being very popular.
Still, if you're interested how I made the above code work in JCSP, here it is:
public static class Process1 implements CSProcess {
private ChannelOutputInt output;
public Process1(ChannelOutputInt out) {
output = out;
}
#Override
public void run() {
for (int i = 0; i < 1; i++) {
System.out.println("Written...");
output.write(5);
}
output.write(-1);
}
}
public static class Process2 implements CSProcess {
private ChannelInputInt input;
public Process2(ChannelInputInt in) {
input = in;
}
#Override
public void run() {
int x = 0;
while ((x = input.read()) > 0) {
System.out.println(x);
}
}
}
public static void main(String[] args) {
One2OneChannelInt chan = Channel.one2oneInt();
Process1 process1 = new Process1(chan.out());
Process2 process2 = new Process2(chan.in());
Parallel parallel = new Parallel();
parallel.addProcess(process1);
parallel.addProcess(process2);
parallel.run();
}
The problem is the channel is unbuffered, so your write() call will block until another process reads from the channel. As soon as another process calls read(), 'Written...' will be printed out.
A BlockingQueue with a capacity of 0 behaves similarly to a JCSP channel
I'm trying to make a chat application for the network in my college. It's actually two programs: One for the server and the other for the clients. All client messages will be sent to server with their sender's name and intended target prepended to them. The server, using this information, sends the message to the target.
I wrote a program which simulates the server side of things with 4 classes: Model, MessageCentre, Receiver and Sender.
Receiver, on an independent thread, generates strings and adds them to the queue in MessageCentre with random time-outs. Sender checks if queue is empty, and if not, it 'sends' the message (just prints it).The Model class simply contains the main method which starts the Receiver and Sender threads.
This is the code of the simulation:
Model class->
package model;
public class Model {
public static void main(String[] args) {
Receiver receiver = new Receiver();
Sender sender = new Sender();
receiver.start();
sender.start();
}
}
MessageCentre class->
package model;
import java.util.LinkedList;
import java.util.Queue;
public class MessageCentre {
private static Queue<String> pendingMessages = new LinkedList<>();
public static synchronized boolean centreIsEmpty() {
return pendingMessages.isEmpty();
}
public static synchronized String readNextAndRemove() {
return pendingMessages.remove();
}
public static synchronized boolean addToQueue(String message) {
return pendingMessages.add(message);
}
}
Receiver class->
package model;
import java.util.Random;
public class Receiver extends Thread {
private int instance;
public Receiver() {
instance = 0; //gets incremented after each message
}
#Override
public void run() {
while (true) {
boolean added = MessageCentre.addToQueue(getMessage());
if (!added) {
System.out.println("Message " + instance + " failed to send");
}
try {
//don't send for another 0 to 10 seconds
Thread.sleep(new Random().nextInt(10_000));
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
}
private String getMessage() {
int copyInstance = instance;
instance++;
return "Message " + copyInstance;
}
}
Sender class->
package model;
public class Sender extends Thread {
#Override
public void run() {
while(true) {
if(!MessageCentre.centreIsEmpty()) {
System.out.println(MessageCentre.readNextAndRemove());
}
}
}
}
Question: If the getMessage() method of the Receiver class were to be replaced by a method which accepts messages from a socket input stream, is there a chance that some messages would be lost?
It is crucial that all received messages be written to the queue so that no messages are lost. This simulation seems to run fine, but I have no way to test a scenario where a large influx of messages are being received through a socket.
The case which I fear might occur is the following:
Receiver gets a message and attempts to write it to the queue.
Sender has a hold of the queue to read and remove items from it, thereby preventing Receiver from writing the newest message. The Receiver finally gets the opportunity to write the current message to the queue, but a new message simultaneously enters the socket input stream to be lost forever.
Is this scenario possible? If so, can it be prevented by setting the priority level of Receiver to be higher than Sender?
a new message simultaneously enters the socket input stream to be lost forever
is always possible, but large numbers of dropped requests are unlikely unless you're under heavy load or have unnecessarily large critical sections. Dropped requests also happen due to factors outside your control, so your system needs to be robust in the face of them anyway.
Use a queue implementation from java.util.concurrent instead of manually synchronizing on a LinkedList and the queue portion of your code should be fine.
I've been using this tutorial for a simple file transfer client/server using socket IO. I changed the response handler to accept multiple reads as a part of one file, as I will be dealing with large files, potentially up to 500 MB. The tutorial didn't account for large server responses, so I'm struggling a bit, and I've created a race condition.
Here's the response handler code:
public class RspHandler {
private byte[] rsp = null;
public synchronized boolean handleResponse(byte[] rsp) {
this.rsp = rsp;
this.notify();
return true;
}
public synchronized void waitForResponse() {
while(this.rsp == null) {
try {
this.wait();
} catch (InterruptedException e) {
}
}
System.out.println("Received Response : " + new String(this.rsp));
}
public synchronized void waitForFile(String filename) throws IOException {
String filepath = "C:\\a\\received\\" + filename;
FileOutputStream fos = new FileOutputStream(filepath);
while(waitForFileChunk(fos) != -1){}
fos.close();
}
private synchronized int waitForFileChunk(FileOutputStream fos) throws IOException
{
while(this.rsp == null) {
try {
this.wait();
} catch (InterruptedException e) {
}
}
fos.write(this.rsp);
int length = this.rsp.length;
this.rsp = null;
if(length < NioClient.READ_SIZE)//Probably a bad way to find the end of the file
{
return -1;
}
else
{
return length;
}
}
}
The main thread of the program creates a RspHandler on the main thread, and passes it to a client, created on a separate thread. The main thread tells the client to request a file, then tells the RspHandler to listen for a response. When the client reads from the server(it reads in chunks of about 1KB right now), it calls the handleResponse(byte[] rsp) method, populating the rsp byte array.
Essentially, I'm not writing the received data to a file as fast as it comes. I'm a bit new to threads, so I'm not sure what to do to get rid of this race condition. Any hints?
this is classic consumer/producer. the most straightforward/easiest way to handle this is to use a BlockingQueue. producer calls put(), consumer calls take().
note, using a BlockingQueue usually leads to the "how do i finish" problem. the best way to do that is to use the "poison pill" method, where the producer sticks a "special" value on the queue which signals to the consumer that there is no more data.