I am interested in a data structure identical to the Java BlockingQueue, with the exception that it must be able to batch objects in the queue. In other words, I would like the producer to be able to put objects into the queue, but have the consumer block on take() untill the queue reaches a certain size (the batch size).
Then, once the queue has reached the batch size, the producer must block on put() untill the consumer has consumed all of the elements in the queue (in which case the producer will start producing again and the consumer block untill the batch is reached again).
Does a similar data structure exist? Or should I write it (which I don't mind), I just don't want to waste my time if there is something out there.
UPDATE
Maybe to clarify things a bit:
The situation will always be as follows. There can be multiple producers adding items to the queue, but there will never be more than one consumer taking items from the queue.
Now, the problem is that there are multiple of these setups in parallel and serial. In other words, producers produce items for multiple queues, while consumers in their own right can also be producers. This can be more easily thought of as a directed graph of producers, consumer-producers, and finally consumers.
The reason that producers should block until the queues are empty (#Peter Lawrey) is because each of these will be running in a thread. If you leave them to simply produce as space becomes available, you will end up with a situation where you have too many threads trying to process too many things at once.
Maybe coupling this with an execution service could solve the problem?
I would suggest you use BlockingQueue.drainTo(Collection, int). You can use it with take() to ensure you get a minimum number of elements.
The advantage of using this approach is that your batch size grows dynamically with the workload and the producer doesn't have to block when the consumer is busy. i.e. it self optimises for latency and throughput.
To implement exactly as asked (which I think is a bad idea) you can use a SynchronousQueue with a busy consuming thread.
i.e. the consuming thread does a
list.clear();
while(list.size() < required) list.add(queue.take());
// process list.
The producer will block when ever the consumer is busy.
Here is a quick ( = simple but not fully tested) implementation that i think may be suitable for your requests - you should be able to extend it to support the full queue interface if you need to.
to increase performance you can switch to ReentrantLock instead of using "synchronized" keyword..
public class BatchBlockingQueue<T> {
private ArrayList<T> queue;
private Semaphore readerLock;
private Semaphore writerLock;
private int batchSize;
public BatchBlockingQueue(int batchSize) {
this.queue = new ArrayList<>(batchSize);
this.readerLock = new Semaphore(0);
this.writerLock = new Semaphore(batchSize);
this.batchSize = batchSize;
}
public synchronized void put(T e) throws InterruptedException {
writerLock.acquire();
queue.add(e);
if (queue.size() == batchSize) {
readerLock.release(batchSize);
}
}
public synchronized T poll() throws InterruptedException {
readerLock.acquire();
T ret = queue.remove(0);
if (queue.isEmpty()) {
writerLock.release(batchSize);
}
return ret;
}
}
Hope you find it useful.
I recently developed this utility that batch BlockingQueue elements using a flushing timeout if queue elements doesn't reach the batch size. It also supports fanOut pattern using multiple instances to elaborate the same set of data:
// Instantiate the registry
FQueueRegistry registry = new FQueueRegistry();
// Build FQueue consumer
registry.buildFQueue(String.class)
.batch()
.withChunkSize(5)
.withFlushTimeout(1)
.withFlushTimeUnit(TimeUnit.SECONDS)
.done()
.consume(() -> (broadcaster, elms) -> System.out.println("elms batched are: "+elms.size()));
// Push data into queue
for(int i = 0; i < 10; i++){
registry.sendBroadcast("Sample"+i);
}
More info here!
https://github.com/fulmicotone/io.fulmicotone.fqueue
Not that I am aware of. If I understand correctly you want either the producer to work (while the consumer is blocked) until it fills the queue or the consumer to work (while the producer blocks) until it clears up the queue. If that's the case may I suggest that you don't need a data structure but a mechanism to block the one party while the other one is working in a mutex fasion. You can lock on an object for that and internally have the logic of whether full or empty to release the lock and pass it to the other party. So in short, you should write it yourself :)
This sounds like how the RingBuffer works in the LMAX Disruptor pattern. See http://code.google.com/p/disruptor/ for more.
A very rough explanation is your main data structure is the RingBuffer. Producers put data in to the ring buffer in sequence and consumers can pull off as much data as the producer has put in to the buffer (so essentially batching). If the buffer is full, the producer blocks until the consumer has finished and freed up slots in the buffer.
Related
I'm writing a Java command line application that scrapes a website and downloads video files. The video files range in size from a few megs to 20 GB or more. This means downloading a file can take as little as a few seconds to as much as a few hours. I've decided to implement a produce/consumer pattern to handle the scraping and downloading of files. A producer thread scrapes the site and retrieves the links to the video files and puts those links into an object and puts that object into an unbounded blocking queue. There are N consumer threads that handle the download. They retrieve the objects containing the URLs from the blocking queue and each thread downloads the file. The object that the producer puts on the queue contains the URL along with some other information that the consumer will need to save the file to the correct location in local storage. Before a file is downloaded, the consumer thread first checks if the file already exists in local storage. If the file exists, the download is skipped and the next object is pulled from the queue. If a consumer experiences a problem while downloading a file (connection reset, etc.), the consumer puts the object containing the URL into a separate queue for failed requests and sleeps for 15 minutes. This allows the application to deal with temporary network interruptions. While the producer is active, it checks the failed URLs queue and removes those URLs from that queue and puts them back into the main queue.
After implementing this initial design, I quickly realized that I had a problem. Because I'm using a blocking queue and the worker threads are polling without a timeout, once the producer was finished, it couldn't just complete its execution because it needed to hang around to put failed URLs back into the queue. My first attempt at a solution was to remove the second "failure" queue and have workers put failed URLs back into the main queue. This meant that the application now had N consumers and N + 1 producers. This approach would allow the main producer thread to just exit when it was finished because it didn't have to worry about putting failed requests back into the queue. Once that problem was solved, there was still another problem. The problem of notifying the worker threads that they could exit once the queue was empty. A blocking queue has no mechanism for the producer to signal that it won't be putting more data to the queue. I thought about having the consumers poll the queue with a timeout and have the primary producer set some sort of flag when it exits. When a consumer times out, it checks the flag. If the flag is set, the consumer exits, if not set, it polls the queue again. While this approach will work, I don't like the design. I don't like the idea of having threads sitting around unnecessarily and I hate even more the use of a magic flag. The only interaction between producer and consumers should be via the queue. The consumers have no knowledge of the producer and checking a magic flag breaks that principle.
I ditched the blocking queue and decided to use a regular non-blocking queue. To prevent the worker threads from exiting as soon as they started, I used a CyclicBarrier. When a worker thread starts, it waits at the barrier before polling the queue. Meanwhile, the producer thread was coded to lower the barrier once the queue contained 10 x N URLs. Once the barrier was lowered, the worker threads would begin processing the Queue. This approach quickly failed because in some cases the consumers would consume the queue faster than the producer could replenish it. This happens in cases where a large number of files are already stored on disk so the consumers don't need to download anything. Once the queue was empty, the consumers exited, even though the producer was still scraping the site looking for URLs.
This tells me that I need to use a blocking queue. I'm continuing to try to find a clean, elegant solution that doesn't depend on timeouts and magic flags. I would love to hear your approach to solving this problem given the requirements.
UPDATE:
I finally settled on a solution based on comments made by user Martin James. Since these were comments and not an answer, there isn't an answer for me to accept. If Martin summarizes his comments into an answer, I'll accept it. Now here's the solution.
When the producer thread completes, it places N objects into the queue that contain null as the value for the URL. I updated the consumer thread to check for a null URL when they pull an object from the queue. If the URL is null, the consumer exits. This approach solves the notification to consumers that the producer is complete. However, it doesn't solve the problem of consumers putting URLs into the queue after the producer has exited. To solve that problem, I switched to a priority blocking queue. I made the object that gets put into the queue a Comparable and the compareTo logic was coded such that objects with null values for the URL will always be last in the queue. So when the producer exits and it places the terminating objects in the queue, if/when a consumer places an object back into the queue, those objects will always be ahead of the terminating objects.
Thanks all for the comments and feedback. Very much appreciated.
My approach would be to use framework with back-pressure mechanism support, for example vert.x reactive streams.
Good examples of systems handling back-pressure built on vert.x can be found in the book vert.x in action
ExecutorService in Java, for example, is a producer-consumer model with a series of worker threads trying to fetch tasks from a work queue. I might close the thread pool by ExecutorService#shutdownNow, this method will set the thread pool state to STOP and interrupt each worker. Take a look at shutdownNow method and worker's run method(I removed the irrelevant code):
public List<Runnable> shutdownNow() {
advanceRunState(STOP);
interruptWorkers();
}
final void runWorker(Worker w) {
try {
while (task != null || (task = getTask()) != null) {
// ...
}
}
}
private Runnable getTask() {
boolean timedOut = false; // Did the last poll() time out?
for (;;) {
// ...
// Check if queue empty only if necessary.
if (rs >= SHUTDOWN && (rs >= STOP || workQueue.isEmpty())) {
decrementWorkerCount();
return null;
}
try {
Runnable r = timed ?
workQueue.poll(keepAliveTime, TimeUnit.NANOSECONDS) :
workQueue.take();
if (r != null)
return r;
timedOut = true;
}
}
}
I think this is an example of using Flag & interrupt to stop consumers. I don't think it's inelegant.
I just recently used ArrayBlockingQueue for my multi-thread process. But it seemed like it slowed down rather than speeding up. Can you guys help me out? I'm basically importing a file (about 300k rows) and parsing them and storing them in the DB
public class CellPool {
private static class RejectedHandler implements RejectedExecutionHandler {
#Override
public void rejectedExecution(Runnable arg0, ThreadPoolExecutor arg1) {
System.err.println(Thread.currentThread().getName() + " execution rejected: " + arg0);
}
}
private static class Task implements Runnable {
private JSONObject obj;
public Task(JSONObject obj) {
this.obj = obj;
}
#Override
public void run() {
try {
Thread.sleep(1);
runThis(obj);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
public void runThis(JSONObject obj) {
//where the rows are parsed and stored in the DB, etc
}
}
public static void executeCellPool(String filename) throws InterruptedException {
// fixed pool fixed queue
BlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(300000, true);
ThreadPoolExecutor executor = new ThreadPoolExecutor(90, 100, 1, TimeUnit.MINUTES, queue);
DataSet ds = CommonDelimitedParser.getDataSet(filename);
final String[] colNames = ds.getColumns();
while (ds.next()) {
JSONObject obj = new JSONObject();
//some JSON object
Task t = new Task(obj);
executor.execute(t);
}
}
}
tl;dr Big queue sizes can have a negative impact, as can large thread counts. Ideally, you want your consumers and producers to be working at a similar rate.
The reason the addition of the queue is causing issues is because you're using a very large queue (which is not necessary) that is taking up resources. Typically, a blocking queue blocks producers when there is no space left in the queue and consumers when there are no objects left in the queue. By creating a such a large one of a static size, Java is assigning that space in memory when you almost certainly aren't using all of it. It would be more effective to force your producer to wait for space in the queue to clear up if your consumers are consumers too slowly. You don't need to store all of the lines from your file in the queue at the same time.
Thread Pool Executor Queues are discussed in the javadoc here.
Bounded queues. A bounded queue (for example, an ArrayBlockingQueue) helps prevent resource exhaustion when used with finite maximumPoolSizes, but can be more difficult to tune and control. Queue sizes and maximum pool sizes may be traded off for each other: Using large queues and small pools minimizes CPU usage, OS resources, and context-switching overhead, but can lead to artificially low throughput. If tasks frequently block (for example if they are I/O bound), a system may be able to schedule time for more threads than you otherwise allow. Use of small queues generally requires larger pool sizes, which keeps CPUs busier but may encounter unacceptable scheduling overhead, which also decreases throughput.
Your large thread size of 90, combined with your very large pool size of 300000, is most likely using a lot of memory, and resulting in additional thread scheduling overhead. I would drop both of them considerably. I don't know what hardware you are running on, but since it sounds like you're writing an IO intensive program, I would try double the number of threads your CPU can handle, and play around with sizes for your blocking queue to see what works (note: I haven't researched this, this is based on my experience running queues and executors. Happy for others to suggest a different count!).
Of note, though, is that the execute() method will throw a RejectedExecutionException on failure to add to the queue if your queue is too small. One way of monitoring the queue would be to check it's capacity before scheduling a task. You can do this by calling:
executor.getQueue().remainingCapacity()
Don't use the executor.getQueue() method to alter the queue in any way, but it can be used for monitoring.
An alternative is to use an unbounded queue, such as a LinkedBlockingQueue without a defined capacity. This way, you won't need to deal with queue sizes. However, if your producers are running much faster than your consumers, you will once again have the issue of consuming too much memory.
Also, kostya is right, a JDBC batch insert would be faster.
If you want to persist records from a file into a relational database as fast as possible you should use JDBC batch insert rather than inserting records one by one.
I want to know I am correct about the below code sample.
I have two Threads in java. Thread_W and Thread_R
Both can access the Queue<String> queue.
in Thread_W has a method called put.
private void put(String email){
queue.offer(email);
}
And in the Thread_R there is a method called get AND it is once called when the Thread_R starts.
public void get(){
while(true)
{
if(!queue.isEmpty())
{
String to = queue.poll();
//thread will consume some time here ...may be 5-10 seconds.
}
}
}
so the method put in the Thread_W will called more efficiently by A other method in the Thread_W.may be in a while loop.
If I use this code in my Java project will the Thread_R lose any of the emails put into the queue?
P.S. I really need a Buffer
You should use a implementation of the Blocking Queue interface, as those are thread safe.
The interface offers the methods put() and take(), which block until they've executed. This way the reading thread doesn't consume a lot of CPU cycles and the writing thread doesn't write, if the queue is full.
Your current busy wait
while(true)
{
if(!queue.isEmpty())
{
//...
}
}
isn't very efficient. It is better to use a blocking method call, so you won't need to check, if the queue is empty (or full).
Also you can't overflow the queue's buffer, if your writing thread is way faster than the reading one as put() wait's for space to become available.
Remember that you could always manually reserve a bigger buffer for the queue by setting the capicitiy in it's construtor beforehand, e.g. ArrayBlockingQueue(int capacity).
If you want to use an unbounded concurrent queue I would recommend taking a look at thread-safe implementations of Deque, for instance LinkedBlockingDeque. LinkedBlockingDeque can be unbounded and take() will block the calling thread if the queue is empty. You do not need to worry about synchronization if you use classes from the java.util.concurentpackage.
I am new to java and I want to write a threaded library as an exercise. It will work this way,
In the main thread, some jobs (as string) will be added to job queue, and when worker thread finishes the jobs, it will add it to finished queue. Main thread will get results from the finished queue. When all jobs are done, main thread will signal workers to stop. Here are some code I wrote so far:
public List<int> get() {
WorkerThread[] threads = new WorkerThread[numThreads];
LinkedList<int> results = new LinkedList<>();
int workCount = 0;
for (int i = 0; i < numThreads; i++) {
threads[i] = new WorkerThread();
threads[i].start();
}
// reader is a BufferedReader
while ((String line = reader.readLine()) != null) {
// put string to job queue
workCount++
}
while(workCount) {
//result = get result from finished queue, and add it to results LinkedList
workCount--;
}
for (int i = 0; i < numThreads; i++) {
threads[i].canStop(); // this sets a private variable that makes infinite while loop to stop
threads[i].join();
}
return results;
}
But I am confused with what kind of Queue implementation to use for this. As documentation shows 11 different kinds of Queue implementations.
A ConcurrentLinkedQueue is a unbounded thread-safe queue based on linked nodes.
It is an appropriate choice when many threads will share access to a
common collection but this class does not permit the use of null
elements.
This link contains a comparison between different queue implementation which can be used for multi-threading.
All the Queue's implementation in java.util.concurrent are advised if more than one thread access the same queue (actually they are all BlockingQueue implementations too). Even on read only functions (otherwise you might have the risk of two thread polling the same job from the queue and executing it). That is if you do not want to spend a great amount of time at synchronizing the accesses yourself (and testing the synchronization).
Then the choice depends on what extra functionality you want in your queue. The difference between LinkedBlockingQueue and ArrayBlockingQueue is in the performances of the various basic operations (insertion, deletion, selection, ...). PriorityBlockingQueue allows you to "push" (aka prioritize) some jobs compared to other depending on your own conditions (I believe it is implemented as a heap but do not quote me on that). DelayQueue does what the name says, it allows you to delay by a certain amount of time some jobs.
I like ConcurrentLinkedQueue since it uses a "wait-free" algorithm which in theories is better than synchronized (waiting for resource) functions.
On another note: I would advise you taking a look at Executor, Executors and ExecutorService rather than manage your own collection of threads.
Since the queue will be accessed from many different threads it'll probably be more efficient to use one of the queue implementations from java.util.concurrent. Other queues may require explicit synchronization.
Try ArrayBlockingQueue for example, it implements a bounded ring buffer. The fact that it's bounded means that it applies automatically back pressure when jobs are submitted faster than they can be processed; the queue doesn't just keep growing until you run out of memory.
The implementations differ mostly in performance characteristics, and different job loads may benefit from different queues. You should test which option is best for your situation.
new SynchronousQueue()
new LinkedBlockingQueue(1)
What is the difference? When I should use SynchronousQueue against LinkedBlockingQueue with capacity 1?
the SynchronousQueue is more of a handoff, whereas the LinkedBlockingQueue just allows a single element. The difference being that the put() call to a SynchronousQueue will not return until there is a corresponding take() call, but with a LinkedBlockingQueue of size 1, the put() call (to an empty queue) will return immediately.
I can't say that i have ever used the SynchronousQueue directly myself, but it is the default BlockingQueue used for the Executors.newCachedThreadPool() methods. It's essentially the BlockingQueue implementation for when you don't really want a queue (you don't want to maintain any pending data).
As far as I understand code above do the same things.
No, the code is not the same at all.
Sync.Q. requires to have waiter(s) for offer to succeed. LBQ will keep the item and offer will finish immediately even if there is no waiter.
SyncQ is useful for tasks handoff. Imagine you have a list w/ pending task and 3 threads available waiting on the queue, try offer() with 1/4 of the list if not accepted the thread can run the task on its own. [the last 1/4 should be handled by the current thread, if you wonder why 1/4 and not 1/3]
Think of trying to hand the task to a worker, if none is available you have an option to execute the task on your own (or throw an exception). On the contrary w/ LBQ, leaving the task in the queue doesn't guarantee any execution.
Note: the case w/ consumers and publishers is the same, i.e. the publisher may block and wait for consumers but after offer or poll returns, it ensures the task/element is to be handled.
One reason to use SynchronousQueue is to improve application performance. If you must have a hand-off between threads, you will need some synchronization object. If you can satisfy the conditions required for its use, SynchronousQueue is the fastest synchronization object I have found. Others agree. See: Implementation of BlockingQueue: What are the differences between SynchronousQueue and LinkedBlockingQueue
[Just trying to put it in (possibly) more clearer words.]
I believe the SynchronousQueue API docs states things very clearly:
A blocking queue in which each insert operation must wait for a corresponding remove operation by another thread, and vice versa.
A synchronous queue does not have any internal capacity, not even a capacity of one. You cannot peek at a synchronous queue because an element is only present when you try to remove it; you cannot insert an element (using any method) unless another thread is trying to remove it; you cannot iterate as there is nothing to iterate.
The head of the queue is the element that the first queued inserting thread is trying to add to the queue; if there is no such queued thread then no element is available for removal and poll() will return null.
And BlockingQueue API docs:
A Queue that additionally supports operations that wait for the queue to become non-empty when retrieving an element, and wait for space to become available in the queue when storing an element.
So the difference is obvious and somewhat critically subtle, especially the 3rd point below:
If the queue is empty when you are retrieving from BlockingQueue, the operation block till the new element is inserted. Also, if the queue is full when you are inserting in the BlockingQueue, the operation will block till the element is removed from the queue and a space is made for the new queue. However note that in SynchronousQueue, as operation is blocked for opposite operation (insert and remove are opposite of each other) to occur on another thread. So, unlike BlockingQueue, the blocking depends on the existence of the operation, instead of existence or non existence of an element.
As the blocking is dependent on existence of opposite operation, the element never really gets inserted in the queue. Thats why the second point: "A synchronous queue does not have any internal capacity, not even a capacity of one."
As a consequence, peek() always returns null (again, check the API doc) and iterator() returns an empty iterator in which hasNext() always returns false. (API doc). However, note that the poll() method neatly retrieves and removes the head of this queue, if another thread is currently making an element available and if no such thread exists, it returns null. (API doc)
Finally, a small note, both SynchronousQueue and LinkedBlockingQueue classes implement BlockingQueue interface.
SynchronousQueue works in a similar fashion with following major differences:
1) The size of SynchronousQueue is 0
2) put() method will only insert an element if take() method will be able to fetch that element from the queue at the same moment i.e. an element cannot be inserted if the consumer take() call is going to take some time to consume it.
SynchronousQueue - Insert only when someone will receive it at that moment itself.
Synchronous queues are basically used for handoff purposes. They do not have any capacity and a put operation is blocked until some other thread performs get operation.
If we want to safely share a variable between two threads, we can put that variable in synchrounous queue and let other thread take it from the queue.
Code Sample from https://www.baeldung.com/java-synchronous-queue
ExecutorService executor = Executors.newFixedThreadPool(2);
SynchronousQueue<Integer> queue = new SynchronousQueue<>();
Runnable producer = () -> {
Integer producedElement = ThreadLocalRandom
.current()
.nextInt();
try {
queue.put(producedElement);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
};
Runnable consumer = () -> {
try {
Integer consumedElement = queue.take();
} catch (InterruptedException ex) {
ex.printStackTrace();
}
};
executor.execute(producer);
executor.execute(consumer);
executor.awaitTermination(500, TimeUnit.MILLISECONDS);
executor.shutdown();
assertEquals(queue.size(), 0);
They are also used in CachedThreadPool to achieve an effect of unlimited(Integer.MAX) thread creation as tasks arrive.
CachedPool has coreSize as 0 and maxPoolSize as Integer.MAX with synchronous queue
As tasks arrive onto queue, other tasks are blocked until the first one is fetched out. Since it does not have any queue capacity, thread pool will create one thread and this thread will take out task allowing more tasks to be put onto the queue. This will continue until thread creation reaches maxPoolSize. Based on timeOut, idle threads maybe terminated and new ones are created without crossing the maxPoolSize.