Develop Reference

How to create a non-blocking #RestController webservice in spring?

I'm having a #RestController webservice method that might block the response thread with a long running service call. As follows:
#RestController
public class MyRestController {
//could be another webservice api call, a long running database query, whatever
#Autowired
private SomeSlowService service;
#GetMapping()
public Response get() {
return service.slow();
}
#PostMapping()
public Response get() {
return service.slow();
}
}
Problem: what if X users are calling my service here? The executing threads will all block until the response is returned. Thus eating up "max-connections", max threads etc.
I remember some time ago a read an article on how to solve this issue, by parking threads somehow until the slow service response is received. So that those threads won't block eg the tomcat max connection/pool.
But I cannot find it anymore. Maybe somebody knows how to solve this?

there are a few solutions, such as working with asynchronous requests. In those cases, a thread will become free again as soon as the CompletableFuture, DeferredResult, Callable, ... is returned (and not necessarily completed).
For example, let's say we configure Tomcat like this:
server.tomcat.max-threads=5 # Default = 200
And we have the following controller:
#GetMapping("/bar")
public CompletableFuture<String> getSlowBar() {
return CompletableFuture.supplyAsync(() -> {
silentSleep(10000L);
return "Bar";
});
}
#GetMapping("/baz")
public String getSlowBaz() {
logger.info("Baz");
silentSleep(10000L);
return "Baz";
}
If we would fire 100 requests at once, you would have to wait at least 200 seconds before all the getSlowBar() calls are handled, since only 5 can be handled at a given time. With the asynchronous request on the other hand, you would have to wait at least 10 seconds, because all requests will likely be handled at once, and then the thread is available for others to use.
Is there a difference between CompletableFuture, Callable and DeferredResult? There isn't any difference result-wise, they all behave the similarly.
The way you have to handle threading is a bit different though:
With Callable, you rely on Spring executing the Callable using a TaskExecutor
With DeferredResult you have to to he thread-handling by yourself. For example by executing the logic within the ForkJoinPool.commonPool().
With CompletableFuture, you can either rely on the default thread pool (ForkJoinPool.commonPool()) or you can specify your own thread pool.
Other than that, CompletableFuture and Callable are part of the Java specification, while DeferredResult is a part of the Spring framework.
Be aware though, even though threads are released, connections are still kept open to the client. This means that with both approaches, the maximum amount of requests that can be handled at once is limited by 10000, and can be configured with:
server.tomcat.max-connections=100 # Default = 10000

in my opinion.the async may be better for the sever.for this particular api, async not works well.the clients also hold the connections. finally it will eating up "max-connections".you can send the request to messagequeue(kafka)and return success to clients. then you get the request and pass it to the slow sevice.

WLPs MicroProfile (FaultTolerance) Timeout Implementation does not interrupt threads?

I'm testing the websphere liberty's fault tolerance (microprofile) implementation. Therefore I made a simple REST-Service with a ressource which sleeps for 5 seconds:
#Path("client")
public class Client {
#GET
#Path("timeout")
public Response getClientTimeout() throws InterruptedException {
Thread.sleep(5000);
return Response.ok().entity("text").build();
}
}
I call this client within the same application within another REST-service:
#Path("mpfaulttolerance")
#RequestScoped
public class MpFaultToleranceController {
#GET
#Path("timeout")
#Timeout(4)
public Response getFailingRequest() {
System.out.println("start");
// calls the 5 seconds-ressource; should time out
Response response = ClientBuilder.newClient().target("http://localhost:9080").path("/resilience/api/client/timeout").request().get();
System.out.println("hello");
}
}
Now I'd expect that the method getFailingRequest() would time out after 4 ms and throw an exception. The actual behaviour is that the application prints "start", waits 5 seconds until the client returns, prints "hello" and then throws an "org.eclipse.microprofile.faulttolerance.exceptions.TimeoutException".
I turned on further debug information:
<logging traceSpecification="com.ibm.ws.microprofile.*=all" />
in server.xml. I get these information, that the timeout is registered even bevor the client is called! But the thread is not interrupted.
(if someone tells me how to get the stacktrace pretty in here... I can do that.)
Since this a very basic example: Am I doing anything wrong here? What can I do to make this example run properly.
Thanks
Edit: Running this example on WebSphere Application Server 18.0.0.2/wlp-1.0.21.cl180220180619-0403) auf Java HotSpot(TM) 64-Bit Server VM, Version 1.8.0_172-b11 (de_DE) with the features webProfile-8.0, mpFaultTolerance-1.0 and localConnector-1.0.
Edit: Solution, thanks to Andy McCright and Azquelt.
Since the call cannot be interrupted I have to make it asynchronous. So you got 2 threads: The first an who invoke the second thread with the call. The first thread will be interrupted, the second remains until the call finishes. But now you can go on with failure handling, open the circuit and stuff like that to prevent making further calls to the broken service.
#Path("mpfaulttolerance")
#RequestScoped
public class MpFaultToleranceController {
#Inject
private TestBase test;
#GET
#Path("timeout")
#Timeout(4)
public Response getFailingRequest() throws InterruptedException, ExecutionException {
Future<Response> resp = test.createFailingRequestToClientAsynch();
return resp.get();
}
}
And the client call:
#ApplicationScoped
public class TestBase {
#Asynchronous
public Future<Response> createFailingRequestToClientAsynch() {
Response response = ClientBuilder.newClient().target("http://localhost:9080").path("/resilience/api/client/timeout").request().get();
return CompletableFuture.completedFuture(response);
}
}

It does interrupt threads using Thread.interrupt(), but unfortunately not all Java operations respond to thread interrupts.
Lots of things do respond to interrupts by throwing an InterruptedException (like Thread.sleep(), Object.wait(), Future.get() and subclasses of InterruptableChannel) but InputStreams and Sockets don't.
I suspect that you (or the library you're using to make the request) is using a Socket which isn't interruptible so you don't see your method return early.
It's particularly unintuitive because Liberty's JAX-RS client doesn't respond to thread interrupts as Andy McCright mentioned. We're aware it's not a great situation and we're working on making it better.

I had the same problem. For some URLs I consume, the Fault Tolerance timeout doesn't work.
In my case I use RestClient. I solved my problem using the readTimeout() of the RestClientBuilder:
MyRestClientClass myRestClientClass = RestClientBuilder.newBuilder().baseUri(uri).readTimeout(3l, TimeUnit.SECONDS) .build(MyRestClientClient.class);
One advantage of using this Timeout control is that you can pass the timeout as a parameter.

How to prevent Spring app context shutdown until shutdown hook is fired

I have a spring-boot application.
I have implemented SmartLifecycle interface in my bean which starts async snmp server in it's start method and stops it in it's stop method.
All working fine, except the fact that main application context stops right after start, so my server bean also stops right after start.
All I need is to make spring context to stop only when shutdown hook is fired.
This is not a web application, so I don't need spring-boot-starter-web, which is solves this problem by starting webserver which prevents context stop until webserver stops.
I can use something like CountDownLatch and waiting for it to be zero in my main method right after context starts. Somethig like this:
public static void main(String[] args) throws InterruptedException {
ConfigurableApplicationContext ctx = SpringApplication.run(SnmpTrapRetranslatorApplication.class, args);
CountDownLatch snmpServerCloseLatch = ctx.getBean("snmpServerCloseLatch", CountDownLatch.class);
snmpServerCloseLatch.await();
}
And my server bean's start method will create this latch with count 1, while stop method will call snmpServerCloseLatch.countDown().
This technique is described here.
But what wrong with this is that my main method is responsible for waiting my custom server bean to stop. I feel this just not right.
How for example spring-boot-starter-web do this? When it starts tomcat, it keeps running until shutdown hook is received and it don't need to have any managing code in the main method. It stops only when context receiving shoutdown signal.
The same behaviour is for example when I have #Scheduled method in my bean. Spring also doesn't stops context automatically. Only on CTRL-C.
I want to achieve similar effect. My main method should have only one line: start the context. Context should start and stop my async server when it starts or stops (already achieved by SmartLifecycle) and should not stop until shutdown is requested (CTRL-C, SIGINT etc).

My investigation lead me to the core of the problem: daemon threads.
The snmp server implementation which I use (snmp4j) use daemon threads internally. So even when snmp server started, there are no more live user threads in JVM, so it exits.
TL/DR:
Just add this method to any bean (snmp server bean is good candidate for this):
#Scheduled(fixedDelay = 1000 * 60 * 60) // every hour
public void doNothing() {
// Forces Spring Scheduling managing thread to start
}
(Do not forget to add #EnableScheduling to your spring configuration).
Explanation:
To prevent stopping spring context, while SNMP server is still running, we need any non-daemon thread to be alive in JVM. Not necessarily main thread. So we can let main method to finish.
We can run new non-daemon thread from our server bean's start method. This thread will wait on some lock in while loop checking for some running variable, while our stop method will set this running variable to false and notifyAll on this lock.
This way, our non-daemon thread will be alive until shotdown hook is triggered (and prevents JVM to exit).
After shutdown hook, spring context lifecycle close method will call all SmartLifecycle bean's close methods, that will lead to SNMP server bean's stop method call, that will lead to set running to false, that will lead to our non-daemon thread to stop, that allow JVM to stop gracefully.
Or instead we can use Spring's scheduling thread in similar way. It also is non-daemon thread, so it will prevent JVM to exit. And Spring manages this thread itself, so it will automatically stop it when shutdown hook is triggered.
To make Spring's scheduling thread to start we need any #Scheduled method in any bean.
I think that first (manual) approach is still more "correct", while requires more async coding (which is error-prone as we all know). Who knows how Spring will change it's scheduling implementation in the future.

SpringApplication app = new SpringApplication(Main.class);
app.setRegisterShutdownHook(false);
ConfigurableApplicationContext applicationContext= app.run();
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
#Override
public void run() {
//do your things
applicationContext.close();
}
}));

jersey ws 2.0 #suspended AsyncResponse, what does it do?

I am analyzing some jersey 2.0 code and i have a question on how the following method works:
#Stateless
#Path("/mycoolstuff")
public class MyEjbResource {
…
#GET
#Asynchronous //does this mean the method executes on child thread ?
public void longRunningOperation(#Suspended AsyncResponse ar) {
final String result = executeLongRunningOperation();
ar.resume(result);
}
private String executeLongRunningOperation() { … }
}
Lets say im at a web browser and i type in www.mysite/mycoolstuff
this will execute the method but im not understanding what the asyncResponse is used for neither the #Asynchronous annotation. From the browser how would i notice its asychnronous ? what would be the difference in removing the annotation ? Also the suspended annotation after reading the documentation i'm not clear its purpose.
is the #Asynchronous annotation simply telling the program to execute this method on a new thread ? is it a convenience method for doing "new Thread(.....)" ?
Update: this annotation relieves the server of hanging onto the request processing thread. Throughput can be better. Anyway from the official docs:
Request processing on the server works by default in a synchronous processing mode, which means that a client connection of a request is processed in a single I/O container thread. Once the thread processing the request returns to the I/O container, the container can safely assume that the request processing is finished and that the client connection can be safely released including all the resources associated with the connection. This model is typically sufficient for processing of requests for which the processing resource method execution takes a relatively short time. However, in cases where a resource method execution is known to take a long time to compute the result, server-side asynchronous processing model should be used. In this model, the association between a request processing thread and client connection is broken. I/O container that handles incoming request may no longer assume that a client connection can be safely closed when a request processing thread returns. Instead a facility for explicitly suspending, resuming and closing client connections needs to be exposed. Note that the use of server-side asynchronous processing model will not improve the request processing time perceived by the client. It will however increase the throughput of the server, by releasing the initial request processing thread back to the I/O container while the request may still be waiting in a queue for processing or the processing may still be running on another dedicated thread. The released I/O container thread can be used to accept and process new incoming request connections.

#Suspended have more definite if you used it, else it will not make any difference of using it.
Let's talk about benefits of it:
#Suspended will pause/Suspend the current thread until it gets response,by default #NO_TIMEOUT no suspend timeout set. So it doesn't mean your request response (I/O)thread will get free and be available for other request.
Now Assume you want your service to be a response with some specific time, but the method you are calling from resource not guarantee the response time, then how will you manage your service response time? At that time, you can set suspend timeout for your service using #Suspended, and even provide a fall back response when time get exceed.
Below is some sample of code for setting suspend/pause timeout
public void longRunningOperation(#Suspended AsyncResponse ar) {
ar.setTimeoutHandler(customHandler);
ar.setTimeout(10, TimeUnit.SECONDS);
final String result = executeLongRunningOperation();
ar.resume(result);
}
for more details refer this

The #Suspended annotation is added before an AsyncResponse parameter on the resource method to tell the underlying web server not to expect this thread to return a response for the remote caller:
#POST
public void asyncPost(#Suspended final AsyncResponse ar, ... <args>) {
someAsyncMethodInYourServer(<args>, new AsyncMethodCallback() {
#Override
void completed(<results>) {
ar.complete(Response.ok(<results>).build());
}
#Override
void failed(Throwable t) {
ar.failed(t);
}
}
}
Rather, the AsyncResponse object is used by the thread that calls completed or failed on the callback object to return an 'ok' or throw an error to the client.
Consider using such asynchronous resources in conjunction with an async jersey client. If you're trying to implement a ReST service that exposes a fundamentally async api, these patterns allow you to project the async api through the ReST interface.
We don't create async interfaces because we have a process that takes a long time (minutes or hours) to run, but rather because we don't want our threads to ever sleep - we send the request and register a callback handler to be called later when the result is ready - from milliseconds to seconds later - in a synchronous interface, the calling thread would be sleeping during that time, rather than doing something useful. One of the fastest web servers ever written is single threaded and completely asynchronous. That thread never sleeps, and because there is only one thread, there's no context switching going on under the covers (at least within that process).

The #suspend annotation makes the caller actually wait until your done work. Lets say you have a lot of work to do on another thread. when you use jersey #suspend the caller just sits there and waits (so on a web browser they just see a spinner) until your AsyncResponse object returns data to it.
Imagine you had a really long operation you had to do and you want to do it on another thread (or multiple threads). Now we can have the user wait until we are done. Don't forget in jersey you'll need to add the " true" right in the jersey servlet definition in web.xml to get it to work.

ThreadPoolExecutor for running AbortableHttpRequest - how to call abort?

I'm running a networking service in android where I direct all my http requests to run and get callbacks from the service when the requests are complete. I run the requests in a ThreadPoolExecutor to limit the number of concurrent requests. As the requests run within the pool, they eventually create an HttpGet or HttpPost, both of which indirectly implement AbortableHttpRequest, which allows one to cancel the connection (say, if it's blocking for a long time).
If a user cancels a request, I'd like to somehow drill into the thread queue and call the abort routine for that request. If, for example, a web site is not responding and the user chooses to do something else, right now my only option is to wait for the standard 5 minute http timeout to occur for that hung request before that thread is freed up. If I could access the thread that has my request and call abort, that would free things up right away.
From what I can understand, it appears once my request has gone into the thread pool, it's a black box until it comes out the other end. Querying the queue will only hand back futures, which hides the runnable.
Is there a better approach for this? I'm fairly new to java and threading (I mostly do perl, which doesn't do threads very well at all).

Just because you give a task to a thread pool executor doesn't mean you can't hold a reference on it. Keep a reference on the task, and if the user chooses to cancel it, then call abort on your task.
public class MyAbortableRunnable implements Runnable {
private final Object lock = new Object();
private AbortableHttpRequest request;
public void abort() {
synchronized(lock) {
if (request != null) {
request.abort();
}
}
}
#Override
public void run() {
...
// create the request
synchronized(lock) {
this.request = ...;
}
...
}
}