Develop Reference

How to limit number of threads within a time period

A service I am using starts blocking requests after 5 are made within 1 second.
Using Java in Spring I am looking for a way to queue threads in such a way that up to 5 threads can access the critical section within a second and any other threads are queued up and released once there is bandwidth for them to continue.
Currently I've attempted this with a lock but it causes the thread to wait 1/5th of a second always, even if we wouldn't be at the max calls per second without sleeping.
Lock l = new ReentrantLock();
try {
l.lock();
//critical section
} finally {
try {
Thread.sleep(200);
} catch (InterruptedException e) {
e.printStackTrace();
}
l.unlock();
}
With this implementation I never exceed the 5 per second but I also cause the response to be delayed by 200 milli after everything is ready to be returned to the user.
I need a solution that only delays threads when a delay is needed. In this case the 6th+ call in a second should be delayed but the first 5 do not need to be delayed. Likewise calls 6-11 could all go through at the same time.

This sort of rate-limiting is quite a common problem in microservice architectures, as it is part of the broader issue of addressing cascading failures. There are many libraries around to deal with this issue, and one of the most widely-used modern ones is called Resilience4j, which provides a RateLimiter implementation. You probably want something pretty close to this:
Create the limiter:
RateLimiterConfig config = RateLimiterConfig.custom()
.limitRefreshPeriod(Duration.ofSeconds(1))
.limitForPeriod(5)
.timeoutDuration(Duration.ofSeconds(4)) //or however long you want to wait before failing
.build();
// Create registry
RateLimiterRegistry rateLimiterRegistry = RateLimiterRegistry.of(config);
// Use registry
RateLimiter rateLimiter = rateLimiterRegistry
.rateLimiter("someServiceLimiter", config);
Use it:
// Decorate your call to BackendService.doSomething()
CheckedRunnable restrictedCall = RateLimiter
.decorateCheckedRunnable(rateLimiter, backendService::doSomething);
//Or, you can use an annotation:
#RateLimiter(name = "someServiceLimiter")
public void doSomething() {
//backend call
}

I think solving it using semaphore API would be the best approach.
import java.util.LinkedList;
import java.util.Queue;
import java.util.concurrent.*;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
public class BulkheadSemaphore {
private Queue<Long> enterQueue = new LinkedList<>();
private ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
private Semaphore semaphore;
public BulkheadSemaphore(final Long timeLimit, final int concurrentThreadsLimit) {
this.semaphore = new Semaphore(concurrentThreadsLimit);
executor.scheduleAtFixedRate(() -> {
final Long now = now();
while (!enterQueue.isEmpty() && now - enterQueue.peek() >= timeLimit) {
enterQueue.poll();
semaphore.release();
}
}, timeLimit, 200, TimeUnit.MILLISECONDS);
}
private Long now() {
return System.currentTimeMillis();
}
public void acquire() {
try {
semaphore.acquire();
} catch (InterruptedException e) {
// todo: handle exception
}
}
public void release() {
semaphore.release();
}
}
The api is quite simple:
Each thread entering the critical section, call bulkheadSemaphore.acqure()
After an external call execution finishes, call bulkheadSemaphore.release()
Why does it solve the problem?
This semaphore releases permits for threads which entered the
critical section long time ago.
It releases it's permits at a certain rate (I set it to 200ms, it can be smaller though). It also guarantees that if a work unit has been done quickly, the next thread will be able to start a new work unit.
Some threads would still face redundant waiting, however it doesn't happen every time and they'd spend 200ms at most.
As requests take time, I'd set timeLimit to 1.5 seconds to match your 1 second limitation.
P.S. Don't forget to shutdown executor service

Using Object.wait instead of Thread.sleep for time-synchronization

I have found this example in reliable source (https://developer.android.com/guide/components/services.html#CreatingStartedService) :
public class HelloIntentService extends IntentService {
public HelloIntentService() {
super("HelloIntentService");
}
#Override
protected void onHandleIntent(Intent intent) {
long endTime = System.currentTimeMillis() + 5*1000;
while (System.currentTimeMillis() < endTime) {
synchronized (this) {
try {
wait(endTime - System.currentTimeMillis());
} catch (Exception e) {
}
}
}
}
}
The question is: Why "sleeping" was implemented this way, and not like this:
endTime = System.currentTimeMillis() + 5*1000;
while (System.currentTimeMillis() < endTime) {
try {
Thread.sleep(endTime - System.currentTimeMillis());
} catch (Exception e) {
}
}
The suggested approach is more compact, and there is no any thread that can call 'notify' on this object. Anyway, even if there it was, the program will "go to sleep" again, if time limit was not expired. So what is hidden reason to write more complex and long code to achieve the same result?
I've seen this question Difference between wait() and sleep() , but it has no answer for my question.
I also have seen this Using Object.wait(millisec) to simulate sleep , but my question is more specific.
UPDATE
Can't believe that developer.android.com provides different code samples for different languages (in my case, for English and Russian). Maybe it just have not been updated yet..
See this screen-shot:

There is a significant difference between Object.wait and Thread.sleep. Object.wait releases any monitors and locks owned by the Thread, while Thread.sleep does not. That in itself makes each of those methods have different purposes. Normally, Thread.sleep is used to pause execution so that something out of your control (like a network response) finishes doing something, while Object.wait is used to synchronize your application by waiting and notifying on concrete events (like a consumer/producer or publisher/subscriber pattern).
Thread.sleep docs
Object.wait docs

Because they did it wrong.
There is no reason to implement that 5 second sleeping using Object.wait. Object.wait is used for synchronization and Thread.sleep for sleeping.
Synchronizing on this is another hint that it all is bad programming. You should avoid it in general unless you have a really good reason for it and in such case it should be documented, which is not the case.
Furthermore when I follow your link I see it this way in their page :
/**
* The IntentService calls this method from the default worker thread with
* the intent that started the service. When this method returns, IntentService
* stops the service, as appropriate.
*/
#Override
protected void onHandleIntent(Intent intent) {
// Normally we would do some work here, like download a file.
// For our sample, we just sleep for 5 seconds.
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// Restore interrupt status.
Thread.currentThread().interrupt();
}
}
}
So maybe they have realized of their mistake and corrected it.
Take notice of how in that code interrupts are properly handled rather than discarded as in the two pieces of code in the original question.

How can a Thread or Runnable be killed instantly

I have a little problem. I've a Service which get a SingleTon Thread when onStartCommand() is triggered.
public int onStartCommand(Intent intent, int flags, int startId)
{
Thread t = myThreadFactory.getConnectionThreadWhatever();
if (t.isAlive() && !t.isinterrupted())
{
// do actions when thread is already alive
}
else
{
// do actions to start and run the thread. e.g. t = new ConnectionThread().start();
}
}
Now the Thread have a Runnable in a loop which is like (pseudocode!)
public static boolean isRunning = false;
public void run()
{
isRunning = true;
while (isRunning)
{
// open the httpconnection with a (read)timeout of 300 (long polling, whatever)
}
}
Now i=I would like to kill the Thread as soon as the connection drops in a Network Broadcast Receiver or whatever case.
What is the common way killing it instantly without waiting before the timeout (e.g. 300 seconds) occurred ?
Currently I am doing this in another class with
public void stopThreadconnectionInstantlyWhatever()
{
ConnectionThread.isRunning = false;
Thread t = myFactory.getConnectionThread();
t.interrupt();
}
Now the problem seems to be that the Thread may wait until the timout happen but every second is more battery usage which should be avoided. So.. any idea? :-)
Well, I could get the httpurlconnection with a singleton pattern aswell and kill it before the timeout appear, but this is just a case

Try to read this article
Implementing cancelable tasks Nothing in the language specification gives interruption any specific semantics, but in larger
programs, it is difficult to maintain any semantics for interruption
other than cancellation. Depending on the activity, a user could
request cancellation through a GUI or through a network mechanism such
as JMX or Web Services. It could also be requested by program logic.
For example, a Web crawler might automatically shut itself down if it
detects that the disk is full, or a parallel algorithm might start
multiple threads to search different regions of the solution space and
cancel them once one of them finds a solution. Just because a task is
cancelable does not mean it needs to respond to an interrupt request
immediately. For tasks that execute code in a loop, it is common to
check for interruption only once per loop iteration. Depending on how
long the loop takes to execute, it could take some time before the
task code notices the thread has been interrupted (either by polling
the interrupted status with Thread.isInterrupted() or by calling a
blocking method). If the task needs to be more responsive, it can poll
the interrupted status more frequently. Blocking methods usually poll
the interrupted status immediately on entry, throwing
InterruptedException if it is set to improve responsiveness. The one
time it is acceptable to swallow an interrupt is when you know the
thread is about to exit. This scenario only occurs when the class
calling the interruptible method is part of a Thread, not a Runnable
or general-purpose library code, as illustrated in Listing 5. It
creates a thread that enumerates prime numbers until it is interrupted
and allows the thread to exit upon interruption. The prime-seeking
loop checks for interruption in two places: once by polling the
isInterrupted() method in the header of the while loop and once when
it calls the blocking BlockingQueue.put() method.
public class PrimeProducer extends Thread {
private final BlockingQueue<BigInteger> queue;
PrimeProducer(BlockingQueue<BigInteger> queue) {
this.queue = queue;
}
public void run() {
try {
BigInteger p = BigInteger.ONE;
while (!Thread.currentThread().isInterrupted())
queue.put(p = p.nextProbablePrime());
} catch (InterruptedException consumed) {
/* Allow thread to exit */
}
}
public void cancel() { interrupt(); }}

How to cancel all the thread/ threads in ExcecutorService?

I've written following multi thread program. I want to cancel the all the thread if one of the thread sends back false as return. However though I'm canceling the thread by canceling individual task. Its not working. What changes I need to make inorder to cancel the thread?
I've written following multi thread program. I want to cancel the all the thread if one of the thread sends back false as return. However though I'm canceling the thread by canceling individual task. Its not working. What changes I need to make inorder to cancel the thread?
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.Callable;
public class BeamWorkerThread implements Callable<Boolean> {
private List<BeamData> beamData;
private String threadId;
public BeamScallopingWorkerThread(
List<BeamData> beamData, String threadId) {
super();
this.beamData = beamData;
this.threadId = threadId;
}
#Override
public Boolean call() throws Exception {
Boolean result = true;
DataValidator validator = new DataValidator();
Iterator<BeamScallopingData> it = beamData.iterator();
BeamData data = null;
while(it.hasNext()){
data = it.next();
if(!validator.validateDensity(data.getBin_ll_lat(), data.getBin_ll_lon(), data.getBin_ur_lat(), data.getBin_ur_lon())){
result = false;
break;
}
}
return result;
}
}
ExecutorService threadPool = Executors.newFixedThreadPool(100);
List<Future<Boolean>> results = new ArrayList<Future<Boolean>>();
long count = 0;
final long RowLimt = 10000;
long threadCount = 1;
while ((beamData = csvReader.read(
BeamData.class, headers1, processors)) != null) {
if (count == 0) {
beamDataList = new ArrayList<BeamData>();
}
beamDataList.add(beamData);
count++;
if (count == RowLimt) {
results.add(threadPool
.submit(new BeamWorkerThread(
beamDataList, "thread:"
+ (threadCount++))));
count = 0;
}
}
results.add(threadPool.submit(new BeamWorkerThread(
beamDataList, "thread:" + (threadCount++))));
System.out.println("Number of threads" + threadCount);
for (Future<Boolean> fs : results)
try {
if(fs.get() == false){
System.out.println("Thread is false");
for(Future<Boolean> fs1 : results){
fs1.cancel(true);
}
}
} catch(CancellationException e){
} catch (InterruptedException e) {
} catch (ExecutionException e) {
} finally {
threadPool.shutdownNow();
}
}
My comments
Thanks all for your input I'm overwhelmed by the response. I do know that, well implemented thread takes an app to highs and mean time it a bad implementation brings the app to knees. I agree I'm having fancy idea but I don't have other option. I've a 10 million plus record hence I will have memory constraint and time constraint. I need to tackle both. Hence rather than swallowing whole data I'm breaking it into chunks and also if one data is invalid i don't want to waste time in processing remaining million data. I find #Mark Peters suggestion is an option. Made the changes accordingly I mean added flag to interrupt the task and I'm pretty confused how the future list works. what I understand is that looping through each field of future list starts once all the thread returns its value. In that case, there is no way to cancel all the task in half way from main list. I need to pass on the reference of object to each thread. and if one thread finds invalid data using the thread refernce call the cancel mathod of each thread to set the interrupt flag.
while(it.hasNext() && !cancelled) {
if(!validate){
// loop through each thread reference and call Cancel method
}
}

Whatever attempt you make to cancel all the remaining tasks, it will fail if your code is not carefully written to be interruptible. What that exactly entails is beyond just one StackOverflow answer. Some guidelines:
do not swallow InterruptedException. Make its occurrence break the task;
if your code does not spend much time within interruptible methods, you must insert explicit Thread.interrupted() checks and react appropriately.
Writing interruptible code is in general not beginner's stuff, so take care.

Cancelling the Future will not interrupt running code. It primarily serves to prevent the task from being run in the first place.
While you can provide a true as a parameter, which will interrupt the thread running the task, that only has an effect if the thread is blocked in code that throws an InterruptedException. Other than that, nothing implicitly checks the interrupted status of the thread.
In your case, there is no blocking; it's busy work that is taking time. One option would be to have a volatile boolean that you check at each stage of your loop:
public class BeamWorkerThread implements Callable<Boolean> {
private volatile boolean cancelled = false;
#Override
public Boolean call() throws Exception {
//...
while(it.hasNext() && !cancelled) {
//...
}
}
public void cancel() {
cancelled = true;
}
}
Then you would keep references to your BeamWorkerThread objects and call cancel() on it to preempt its execution.
Why don't I like interrupts?
Marko mentioned that the cancelled flag above is essentially reinventing Thread.interrupted(). It's a valid criticism. Here's why I prefer not to use interrupts in this scenario.
1. It's dependent on certain threading configurations.
If your task represents a cancellable piece of code that can be submitted to an executor, or called directly, using Thread.interrupt() to cancel execution in the general case assumes that the code receiving the interrupt will be the code that should know how to cleanly cancel the task.
That might be true in this case, but we only know so because we know how both the cancel and the task work internally. But imagine we had something like this:
Task does piece of work
Listeners are notified on-thread for that first piece of work
First listener decides to cancel the task using Thread.interrupt()
Second listener does some interruptible piece of work, and is interrupted. It logs but otherwise ignores the interrupt.
Task does not receive interrupt, and task is not cancelled.
In other words, I feel that interrupt() is too global of a mechanism. Like any shared global state, it makes assumptions about all of the actors. That's what I mean by saying that using interrupt() exposes/couples to details about the run context. By encapsulating it in a cancel() method applicable only for that task instance, you eliminate that global state.
2. It's not always an option.
The classic example here is an InputStream. If you have a task that blocks on reading from an InputStream, interrupt() will do nothing to unblock it. The only way to unblock it is to manually close the stream, and that's something best done in a cancel() method for the task itself. Having one way to cancel a task (e.g. Cancellable), regardless of its implementation, seems ideal to me.

Use the ExecutorService.shutdownNow() method. It will stop the executor from accepting more submissions and returns with the Future objects of the ongoing tasks that you can call cancel(true) on to interrupt the execution. Of course, you will have to discard this executor as it cannot be restarted.
The cancel() method may not terminate the execution immediately if the Thread is not waiting on a monitor (not blocked interruptibly), and also if you swallow the InterruptedException that will be raised in this case.

Timed-out lock in java

I am looking for mechanism which will help me to implement following pattern (pseudocode):
TimeoutLock lock = new TimeoutLock();
while(condition) {
prepare();
lock.enter(); //cannot enter until specified lock timeout elapses
execute();
lock.lockFor(2 minutes);
finish();
}
I need to limit invocations to execute to occur no more often, than some specified interval (for example, two minutes), but I do not want to block prepare or execute if it is not necessary. I am wondering if java supports any locking mechanism, which 'vanishes' after some time. Requirement is that, of course, lock does not pass through even if it's entered by the same thread, which locked it.
I was thinking about solution involving semaphore and TimerTask, or calculating deadline by myself and sleeping for superfluous time, but I wonder if something like this is already available.
Thanks

The below will do
basically you have a semphore which will only let you access if there is a permit available, I this case zero permits. So it will try for 2000 seconds before finally giving up
->
Semaphore s = new Semaphore(0);
Object lock = new Object();
synchronized(lock)
{
execute();
s.tryAcquire(2,TimeUnit.Minutes)
}
Thread.sleep is a lame and low level way of doing it. Not recommended

No need for a special class:
synchronized(lock) {
execute();
Thread.sleep(120 * 1000)
}

As Marko says, you very likely want to do this by handing the work off to a scheduler of some sort, rather than blocking the thread.
But if you do want to do this, i would suggest that you do it by recording a timestamp on exiting the critical section, and having entering threads wait for a period after that to pass. Something like:
public class TimeoutLock {
private boolean held;
private long available;
public void enter() throws InterruptedException {
acquire();
long pause;
while ((pause = available - System.currentTimeMillis()) > 0L) {
Thread.sleep(pause);
}
}
private synchronized void acquire() throws InterruptedException {
while (held) {
wait();
}
held = true;
}
public synchronized void lockFor(long period) {
held = false;
available = System.currentTimeMillis() + period;
notify();
}
}

You could use the sleep
sleep(1000);