Is there a way to make a Stream block and wait for data to be ready in the data source and then I close it when I know there is nothing else to wait for? I already tried to make the data source a blocking one like BlockingQueue but it didn't work obviously as I am looping on the stream using forEeach not using take or peek functions that block.
Streams are designed around Spliterator as the ultimate source of their elements. You could implement the tryAdvance() method to test whether another element exists, blocking until the result is known.
You mentioned a BlockingQueue, which is useful in concurrent processing. If you are "producing" elements in some threads, and trying to "consume" them in others, you might find that a CompletionService fits your application better than a custom Stream.
Spliterator is a fairly simple interface in terms of its operations, but implementing it correctly requires a good understanding of spliterator "characteristics". I would consider it an advanced topic, and while there are cases where a custom implementation is useful, it might also be a warning sign that you are looking at the wrong approach—you don't have to use Stream for everything.
(Create a Stream from a Spliterator with StreamSupport.stream().)
I guess this should work:
BlockingQueue<T> queue = ...; // Signal with a null sentinel.
Iterable<T> collection = () -> new Iterator<T>() {
private boolean hasCurrent;
private T current;
public boolean hasNext() {
if (!hasCurrent) {
current = queue.take();
hasCurrent = true;
}
return current != null;
}
public T next() {
if (hasCurrent) {
hasCurrent = false;
return current;
} else {
return queue.take();
}
}
};
StreamSupport.stream(collection.spliterator(), false)...
There may well be better ways.
Related
I am using a proprietary, 3rd party framework in my Android app -- EMDK from Zebra, to be specific -- and two of their exposed methods:
.read() and .cancelRead() are asynchronous, each taking anywhere from a split second to a 5 whole seconds to complete. I need to be able to spam them without crashing my application and ensure that each one isn't called twice in a row. How can I go about doing this? I don't have any access to the methods themselves and a decompiler will only give me runtime stubs.
Edit: I also have no idea when each of these two calls ever completes.
Changing asynchronous programs into blocking ones is a more general requirement to this problem.
In Java, we can do this with CountDownLatch (as well as Phaser), or LockSupport + Atomic.
For example, if it is required to change an asynchronous call asyncDoSomethingAwesome(param, callback) into a blocking one, we could write a "wrapper" method like this:
ResultType doSomethingAwesome(ParamType param) {
AtomicReference<ResultType> resultContainer = new AtomicReference<>();
Thread callingThread = Thread.currentThread();
asyncDoSomethingAwesome(param, result -> {
resultContainer.set(result);
LockSupport.unpark(callingThread);
});
ResultType result;
while ((result = resultContainer.get()) == null) {
LockSupport.park();
}
return result;
}
I think this will be enough to solve your problem. However, when we are writing blocking programs, we usually want a "timeout" to keep the system stable even when an underlying interface is not working properly, for example:
ResultType doSomethingAwesome(ParamType param, Duration timeout) throws TimeoutException {
AtomicReference<ResultType> resultContainer = new AtomicReference<>();
Thread callingThread = Thread.currentThread();
asyncDoSomethingAwesome(param, result -> {
resultContainer.set(result);
LockSupport.unpark(callingThread);
});
ResultType result;
long deadline = Instant.now().plus(timeout).toEpochMilli();
while ((result = resultContainer.get()) == null) {
if (System.currentTimeMillis() >= deadline) {
throw new TimeoutException();
}
LockSupport.parkUntil(deadline);
}
return result;
}
Sometimes we need more refined management to the signal among threads, especially when writing concurrency libries. For example, when we need to know whether the blocking thread received the signal from another thread calling LockSupport.unpark, or whether that thread successfully notified the blocking thread, it is usually not easy to implement with Java standard library. Thus I designed another library with more complete mechanism to solve this issue:
https://github.com/wmx16835/experimental_java_common/blob/master/alpha/src/main/java/mingxin/wang/common/concurrent/DisposableBlocker.java
With the support of DisposableBlocker, life will become much easier :)
ResultType doSomethingAwesome(ParamType param, Duration timeout) throws TimeoutException {
// We can use org.apache.commons.lang3.mutable.MutableObject instead of AtomicReference,
// because this object will never be accessed concurrently
MutableObject<ResultType> resultContainer = new MutableObject<>();
DisposableBlocker blocker = new DisposableBlocker();
asyncDoSomethingAwesome(param, result -> {
resultContainer.setValue(result);
blocker.unblock();
});
if (!blocker.blockFor(timeout)) {
throw new TimeoutException();
}
return resultContainer.getValue();
}
Might be off on this as I'm not 100% sure what you're trying to achieve/nor the structure, but could you wrap each in an AsyncTask? Then in a parent AsyncTask or background thread:
AsyncTask1.execute().get(); //get will block until complete
AsyncTask2.execute().get(); //get will block until complete
This is assuming there is some way of knowing the calls you're making completed.
I am using an ArrayBlockingQueue but sometimes it gets to full and prevents other objects to be added to it.
What I would like to do is to remove the oldest object in the queue before adding another one when the ArrayBlockingQueue gets full. I need the ArrayBlockingQueue to be like the Guava EvictingQueue but thread safe. I intend to extend the ArrayBlockingQueue and override the offer(E e) method like below:
public class MyArrayBlockingQueue<E> extends ArrayBlockingQueue<E> {
// Size of the queue
private int size;
// Constructor
public MyArrayBlockingQueue(int queueSize) {
super(queueSize);
this.size = queueSize;
}
#Override
synchronized public boolean offer(E e) {
// Is queue full?
if (super.size() == this.size) {
// if queue is full remove element
this.remove();
}
return super.offer(e);
} }
Is the above approach OK? Or is there a better way of doing it?
Thanks
Your MyArrayBlockingQueue doesn't override BlockingQueue.offer(E, long, TimeUnit) or BlockingQueue.poll(long, TImeUnit). Do you actually need a queue with "blocking" features? If you do not then you can create a thread-safe queue backed by an EvictingQueue using Queues.synchronizedQueue(Queue):
Queues.synchronizedQueue(EvictingQueue.create(maxSize));
For an evicting blocking queue, I see a few issues with your proposed implementation:
remove() may throw an exception if the queue is empty. Your offer method is marked with synchronized but poll, remove, etc. are not so another thread could drain your queue in between calls to size() and remove(). I suggest using poll() instead which won't throw an exception.
Your call to offer may still return false (i.e. not "add" the element) because of another race condition where between checking the size and/or removing an element to reduce the size a different thread adds an element filling the queue. I recommend using a loop off of the result of offer until true is returned (see below).
Calling size(), remove() and offer(E) each require a lock so in the worse case scenario your code locks and unlocks 3 times (and even then it might fail to behave as desired due to the previous issues described).
I believe the following implementation will get you what you are after:
public class EvictingBlockingQueue<E> extends ArrayBlockingQueue<E> {
public EvictingBlockingQueue(int capacity) {
super(capacity);
}
#Override
public boolean offer(E e) {
while (!super.offer(e)) poll();
return true;
}
#Override
public boolean offer(E e, long timeout, TimeUnit unit) throws InterruptedException {
while (!super.offer(e, timeout, unit)) poll();
return true;
}
}
Note that this implementation can unnecessarily remove an element if between two calls to super.offer(E) another thread removes an element. This seems acceptable to me and I don't really see a practical way around it (ArrayBlockingQueue.lock is package-private and java.util.concurrent is a prohibited package so we can't place an implementation there to access and use the lock, etc.).
When you say "it gets to full and prevents other objects to be added", does that mean it would be sufficient to ensure that objects can be added anytime? If that's true, you could simply switch to an unbounded queue such as LinkedBlockingQueue. But be aware of the differences compared with ArrayBlockingQueue:
Linked queues typically have higher throughput than array-based queues but less predictable performance in most concurrent applications.
You can find an overview of JDK queue implementations here.
Suppose I have the following RxJava code (which accesses a DB, but the exact use case is irrelevant):
public Observable<List<DbPlaceDto>> getPlaceByStringId(final List<String> stringIds) {
return Observable.create(new Observable.OnSubscribe<List<DbPlaceDto>>() {
#Override
public void call(Subscriber<? super List<DbPlaceDto>> subscriber) {
try {
Cursor c = getPlacseDb(stringIds);
List<DbPlaceDto> dbPlaceDtoList = new ArrayList<>();
while (c.moveToNext()) {
dbPlaceDtoList.add(getDbPlaceDto(c));
}
c.close();
if (!subscriber.isUnsubscribed()) {
subscriber.onNext(dbPlaceDtoList);
subscriber.onCompleted();
}
} catch (Exception e) {
if (!subscriber.isUnsubscribed()) {
subscriber.onError(e);
}
}
}
});
}
Given this code, I have the following questions:
If someone unsubscribes from the observable returned from this method (after a previous subscription), is that operation thread-safe? So are my 'isUnsubscribed()' checks correct in this sense, regardless of scheduling?
Is there a cleaner way with less boilerplate code to check for unsubscribed states than what I'm using here? I couldn't find anything in the framework. I thought SafeSubscriber solves the issue of not forwarding events when the subscriber is unsubscribed, but apparently it does not.
is that operation thread-safe?
Yes. You are receiving an rx.Subscriber which (eventually) checks against a volatile boolean that is set to true when the subscriber's subscription is unsubscribed.
cleaner way with less boilerplate code to check for unsubscribed states
The SyncOnSubscribe and the AsyncOnSubscribe (available as an #Experimental api as of release 1.0.15) was created for this use case. They function as a safe alternative to calling Observable.create. Here is a (contrived) example of the synchronous case.
public static class FooState {
public Integer next() {
return 1;
}
public void shutdown() {
}
public FooState nextState() {
return new FooState();
}
}
public static void main(String[] args) {
OnSubscribe<Integer> sos = SyncOnSubscribe.createStateful(FooState::new,
(state, o) -> {
o.onNext(state.next());
return state.nextState();
},
state -> state.shutdown() );
Observable<Integer> obs = Observable.create(sos);
}
Note that the SyncOnSubscribe next function is not allowed to call observer.onNext more than once per iteration nor can it call into that observer concurrently. Here are a couple of links to the SyncOnSubscribe implementation and tests on the head of the 1.x branch. It's primary usage is to simplify writing observables that iterate or parsing over data synchronously and onNext downstream but doing so in a framework that supports back-pressure and checks if unsubscribed. Essentially you would create a next function which would get invoked every time the downstream operators need a new data element onNexted. Your next function can call onNext either 0 or 1 time.
The AsyncOnSubscribe is designed to play nicely with back pressure for observable sources that operate asynchronously (such as off-box calls). The arguments to your next function include the request count and your provided observable should provide an observable that fulfills data up to that requested amount. An example of this behavior would be paginated queries from an external datasource.
Previously it was a safe practice to transform your OnSubscribe to an Iterable and use Observable.from(Iterable). This implementation gets an iterator and checks subscriber.isUnsubscribed() for you.
I have an object that is being initialized in a separate thread. Initialization can take several seconds while a local DB is being populated.
SpecialAnalysis currentAnalysis = new SpecialAnalysis(params_here);
I'm trying to implement a "cancel" button, that sets the object's isCancelled boolean to true. What is the proper Java way to implement this?
while (currentAnalysis == null) {
}
currentAnalysis.cancel();
This method freezes the program as it appears to have entered a computationally inefficient loop. Is this a case where I could use Object.wait()?
My current bad/semi-successful solution is:
while (currentAnalysis == null) {
Thread.sleep(500);
}
currentAnalysis.cancel();
Thanks!
Firstly, yes Object.wait() and Object.notify() / Object.notifyAll() are what you need. Whether or not you use them directly is a different matter. Due to the ease of making mistakes programming directly with wait/notify it is generally recommended to use the concurrency tools added in Java 1.5 (see second approach below).
The traditional wait/notify approach:
Initialisation:
synchronized (lockObject) {
SpecialAnalysis currentAnalysis = new SpecialAnalysis(params_here);
lockObject.notifyAll();
}
In the 'cancel' thread:
synchronized (lockObject) {
while (currentAnalysis == null) {
try { lockObject.wait(); }
catch Exception(e) { } // FIXME: ignores exception
}
}
currentAnalysis.cancel();
Of course these could be synchronized methods instead of blocks. Your choice of lockObject will depend on how many 'cancel' threads you need etc. In theory it could be anything, i.e. Object lockObject = new Object(); as long as you are careful the correct threads have access to it.
Note that it is important to put the call to wait() in a while loop here due to the possibility of spurious wakeups coming from the underlying OS.
A simpler approach would be to use a CountDownLatch, sparing you from the nuts and bolts of wait()¬ify():
(I'm making a couple of assumptions here in order to suggest a possibly cleaner approach).
class AnalysisInitialiser extends Thread {
private CountDownLatch cancelLatch = new CountDownLatch(1);
private SpecialAnalysis analysis = null;
#Override
public void run() {
analysis = new SpecialAnalysis(params);
cancelLatch.countDown();
}
public SpecialAnalysis getAnalysis() {
cancelLatch.await();
return analysis;
}
}
Then in the thread that needs to send the cancel signal: (obviously you need to get hold of the AnalysisInitialiser object in some way)
analysisInit.getAnalysis.cancel();
No concurrency primitive boilerplate, yay!
i like this question so voted up..
you can do like below
do {
if(currentAnalysis != null){
currentAnalysis.cancel();
}
}
while (currentAnalysis == null)
here your do keeps checking the value of currentAnalysis and once its not null then it performs cancel else keeps looping and checking currentAnalysis value.
this is one better approach i am finding right now
The producer is finite, as should be the consumer.
The problem is when to stop, not how to run.
Communication can happen over any type of BlockingQueue.
Can't rely on poisoning the queue(PriorityBlockingQueue)
Can't rely on locking the queue(SynchronousQueue)
Can't rely on offer/poll exclusively(SynchronousQueue)
Probably even more exotic queues in existence.
Creates a queued seq on another (presumably lazy) seq s. The queued
seq will produce a concrete seq in the background, and can get up to
n items ahead of the consumer. n-or-q can be an integer n buffer
size, or an instance of java.util.concurrent BlockingQueue. Note
that reading from a seque can block if the reader gets ahead of the
producer.
http://clojure.github.com/clojure/clojure.core-api.html#clojure.core/seque
My attempts so far + some tests: https://gist.github.com/934781
Solutions in Java or Clojure appreciated.
class Reader {
private final ExecutorService ex = Executors.newSingleThreadExecutor();
private final List<Object> completed = new ArrayList<Object>();
private final BlockingQueue<Object> doneQueue = new LinkedBlockingQueue<Object>();
private int pending = 0;
public synchronized Object take() {
removeDone();
queue();
Object rVal;
if(completed.isEmpty()) {
try {
rVal = doneQueue.take();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
pending--;
} else {
rVal = completed.remove(0);
}
queue();
return rVal;
}
private void removeDone() {
Object current = doneQueue.poll();
while(current != null) {
completed.add(current);
pending--;
current = doneQueue.poll();
}
}
private void queue() {
while(pending < 10) {
pending++;
ex.submit(new Runnable() {
#Override
public void run() {
doneQueue.add(compute());
}
private Object compute() {
//do actual computation here
return new Object();
}
});
}
}
}
Not exactly an answer I'm afraid, but a few remarks and more questions. My first answer would be: use clojure.core/seque. The producer needs to communicate end-of-seq somehow for the consumer to know when to stop, and I assume the number of produced elements is not known in advance. Why can't you use an EOS marker (if that's what you mean by queue poisoning)?
If I understand your alternative seque implementation correctly, it will break when elements are taken off the queue outside your function, since channel and q will be out of step in that case: channel will hold more #(.take q) elements than there are elements in q, causing it to block. There might be ways to ensure channel and q are always in step, but that would probably require implementing your own Queue class, and it adds so much complexity that I doubt it's worth it.
Also, your implementation doesn't distinguish between normal EOS and abnormal queue termination due to thread interruption - depending on what you're using it for you might want to know which is which. Personally I don't like using exceptions in this way — use exceptions for exceptional situations, not for normal flow control.