Google App Engine allows you to create threads if you use their ThreadManager.currentRequestThreadFactory() in conjunction with an ExecutorService. So, in order to allow the same frontend instance to handle multiple servlet requests at the same time, I am planning on writing code that looks like the following:
public class MyServlet implements HttpServlet {
private RequestDispatcher dispatcher;
// Getter and setter for 'dispatcher'.
#Override
public void doGet(HttpServletRequest request, HttpServletResponse response) {
MyResponse resp = dispatcher.dispatch(request);
PrintWriter writer = response.getWriter();
// Use the 'resp' and 'writer' objects to produce the resultant HTML
// to send back to the client. Omitted for brevity.
}
}
public class RequestDispatcher {
private ThreadFactory threadFactory =
ThreadManager.currentRequestThreadFactory();
private Executor executor;
// Getters and setters for both properties.
public MyResponse dispatch(HttpServletRequest request) {
if(executor == null)
executor = Executors.newCachedThreadPool(threadFactory);
// MyTask implements Callable<MyResponse>.
MyTask task = TaskFactory.newTask(request);
MyResponse myResponse = executor.submit(task);
}
}
So now I believe we have a setup where each frontend will have a servlet that can accept up to 10 (I believe that's the max for what GAE allows) requests at the same time, and process all of them concurrently without blocking. So first off, if I've mistaken the use of ThreadManager and am not using it correctly, or if my setup for this type of concurrent behavior is incorrect, please begin by correcting me!
Assuming I'm more or less on track, I have some concurrency-related concerns with how Google App Engine threads utilize the object tree underneath the MyTask object.
The MyTask callable is responsible for actually processing the HTTP request. In EJB land, this would be the "business logic" code that does stuff like: (1) placing messages on a queue, (2) hitting the Google Datastore for data, (3) saving stuff to a cache, etc. The point is, it spawns a big "object tree" (lots of subsequent child objects) when its call() method is executed by the Executor.
Do I have to make each and every object that gets created from inside MyTask#call thread-safe? Why or why not? Thanks in advance!
You don't need all this to enable an instance to process multiple requests concurrently. GAE allows you to spawn threads if you need to perform multiple tasks in parallel when handling a single given request.
It could be useful, for example, if you need to contact several external URLs in parallel to get information needed to respond to a given request. This would be more efficient tha contacting all the URLs in sequence.
Related
I have a sprint boot (v1.5.15) based Restful application that provides user based services, particularly login and get user details.
The login activity is slightly heavy where as the get user details api is pretty light weight.
I have a controller akin to this
#RestController
public class UserController{
#PostMapping("/login")
public LoginResponse userLogin(#RequestBody LoginRequest loginRequest){
...
}
#GetMapping("/users/{id}")
public LoginResponse userIdGet(#PathVariable("id") String id){
...
}
}
Is there any way I could limit the number of concurrent calls to the /login api. Basically I want to limit this to say x as the /users/{id} can handle in the same resources around 10x of that calls.
The application uses the embedded tomcat server and I know of server.tomcat.max-connections, server.tomcat.max-threads and server.tomcat.min-spare-threads however these restrict the calls at the application level rather than at the API.
There are solutions which limit the number of active connections, see e.g. https://dzone.com/articles/how-to-limit-number-of-concurrent-user-session-in .
However, afaik, such solutions are just rejecting further request.
If you do not like to reject request, you might limit the concurrent work done by using using an application wide fixed thread pool ExecutorService ( https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html#newFixedThreadPool(int) ) and submit your request body to that thread pool and imediatly call get on the returned Future.
So you can replace
#PostMapping("/api/xyzMethod")
public Response xyzMethod(#RequestBody Request request) {
return handleXyzMethod(request); });
}
by
#PostMapping("/api/xyzMethod")
public Response xyzMethod(#RequestBody Request request) throws InterruptedException, ExecutionException {
return xyzMethodExecutor.submit(() -> { return handleXyzMethod(request); }).get();
}
with some
private static ExecutorService xyzMethodExecutor = Executors.newFixedThreadPool(10);
A drawback is that the user might have to wait for the reply and / or that multiple request will fill the threads pool queue until the service becomes (too) unresponsive. So maybe you have to endow this solution with some kind of timeout on the FutureTasks or combine the two solution (that is also have a larger limit on the number of concurrent sessions).
Is it safe for my Java (Tomcat 8) web server to spawn threads in response to a HTTP request? I'm seeing posts and forums where some people say it's absolutely fine, and others say not to do it.
My use case would be something like this:
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
...
...
final MyResult res = new MyResult();
Thread first = new Thread(new Runnable() {
#Override
public void run() {
// put this into res
}
});
Thread second = new Thread(new Runnable() {
#Override
public void run() {
// put that into res
}
});
first.start();
second.start();
first.join(10000);
second.join(10000);
// return res
}
When I say safe, I mean is there anything inherently dangerous about what I'm proposing with regards to the stability of the web server. As #Burrman points out, a thread pool is good idea here, and I will do that. If I am using a thread pool, is there then any other potential issues with the servlet container that I should be concerned about or need to address?
I suppose what I'm thinking about is, for example, JDBC connections. I believe it's recommended to set that up using JNDI resource etc. and configuring that with Tomcat config. Is anything like that necessary or recommended for spawning arbitrary threads like in my example?
First, it looks you're modifying the result object in both threads. This is not thread safe because what the first and second threads do might not be visible to each other or to the thread the servlet is running on. See this article for more info.
Second, if you are modifying the response in these other threads, no, this will not be safe. Once you exit the doGet method, you should consider the response sent. In your example, there's a chance the response will get sent back to the client before those two threads have run.
Suppose the MyResult result affects the response object (you're either adding the result to the response, it's affecting the response code, etc). There are a few ways to handle this.
Use ExecutorService and Future:
public void doGet(HttpServletRequest request, HttpServletResponse response) {
// Creating a new ExecutorService for illustrative purposes.
// As mentioned in comments, it is better to create a global
// instance of ExecutorService and use it in all servlets.
ExecutorService executor = Executors.newFixedThreadPool(2);
Future<Result1> f1 = executor.submit(new Callable<Result1>() {
#Override
public Result1 call() throws Exception {
// do expensive stuff here.
return result;
}
});
Future<Result2> f2 = executor.submit(new Callable<Result2>() {
#Override
public Result2 call() throws Exception {
// do expensive stuff here.
return result;
}
});
// shutdown allows the executor to clean up its threads.
// Also prevents more Callables/Runnables from being submitted.
executor.shutdown();
// The call to .get() will block until the executor has
// completed executing the Callable.
Result1 r1 = f1.get();
Result2 r2 = f2.get();
MyResult result = new MyResult();
// add r1 and r2 to result.
// modify response based on result
}
A more advanced technique is Asynchronous Processing. Using async processing is a good idea if your requests take a long time to process. It does not improve the latency of any one request, but it does allow Tomcat to handle more requests at any given point in time.
A simple example would be:
#WebServlet(urlPatterns={"/asyncservlet"}, asyncSupported=true)
// Rather than #WebServlet, you can also specify these parameters in web.xml
public class AsyncServlet extends HttpServlet {
#Override
public void doGet(HttpServletRequest request, HttpServletResponse response) {
response.setContentType("text/html;charset=UTF-8");
final AsyncContext acontext = request.startAsync();
acontext.start(new Runnable() {
public void run() {
// perform time consuming steps here.
acontext.complete();
}
}
Also, besides excellent answer of kuporific I really advise you to think if your result computation could be expressed in terms of map / filter / group operations on lists or maps, because in 90% of cases it's possible to do so.
And if that's the case I would really advise you to use Java 8 Stream.parallelStream functionality as outlined in this official tutorial
Please ask separate question if you are interested if/how it's possible to express your computation in that way
Also, answering your initial question - it's perfectly fine to spawn threads to parallelise your computation anywhere (including servlets), however I would really advise to measure performance before and after optimization mainly because of reasons described in this superb article
In principle, yes. But you have to keep an eye on the total number of threads you are spawning to make sure you're not using too many resources.
Using a thread pool can help keep your number of threads under control.
In my opinion
this idea has not sense if thread should give effect in same request (cit. in order to satisfy a request)
may have sense (if done properly) when effect of thread (running / ended) will be visible in next requests (especially via AJAX). Most servlet frameworks have way how to do "in orthodox way" long operations.
To be clear: this is not normal way to do typical everyday AJAX tasks (must divide design to "main" request and next children ajax requests, using your framework), but AJAX can present state of background long thread.
In my conception Thread or equivalent can be started from few/rare request, far below limits etc, but if it is done from every request, it is very bad design.
JEE server stores some information in thread local variables. E.g. security context for JAAS, JTA transaction, ... New plain java thread has no access to such information. AsyncContext and JEE ExecuterService are integrated into server and can transparently spread request state to managed threads.
Alright, I've already asked one question regarding this, but needed a bit more info. I'll try to be coherent with my question as much as I can. (Since I am not sure of the concepts).
Background
I have a java web project(dynamic). I am writing Restful webservices. Below is a snippet from my class
/services
class Services{
static DataSource ds;
static{
ds = createNewDataSource;
}
/serviceFirst
#Consumes(Something)
#produces(Something)
public List<Data> doFirst(){
Connection con = ds.getConnection();
ResultSet res = con.execute(preparedStatement);
//iterate over res, and create list of Data.
return list;
}
}
This is a very basic functionality that I have stated here.
I've got tomcat server where I've deployed this. I've heard that Tomcat has a threadpool, of size 200 (by default). Now my question is that, how exactly does the threadpool work here.
Say I have two requests coming in at the same time. That means that 2 of the threads from the threadpool will get to work. Does this mean that both the threads will have an instance of my class Services? Because below is how I understand the threads and concurrency.
public class myThread extends Thread(){
public void run(){
//do whatever you wan to do here;
}
}
In the above, when I call start on my Thread it will execute the code in run() method and all the objects that it creates in there, will belong to it.
now, coming back to the Tomcat, is there somewhere a run() method written that instantiates the Services class, and that is how the threadpool handles 200 concurrent requests. (Obviously, I understant they will require 200 cores for them to execute concurrently, so ignore that).
Because otherwise, if tomcat does not have 200 different threads having the same path of execution (i.e. my Services class), then how exactly will it handle the 200 concurrent requests.
Thanks
Tomcat's thread pool works, more or less, like what you would get from an ExecutorService (see Executors).
YMMV. Tomcat listens for requests. When it receives a request, it puts this request in a queue. In parallel, it maintains X threads which will continuously attempt to take from this queue. They will prepare the ServletRequest and ServletResponse objects, as well as the FilterChain and appropriate Servlet to invoke.
In pseudocode, this would look like
public void run() {
while (true) {
socket = queue.take();
ServletRequest request = getRequest(socket.getInputStream());
ServletResponse response = generateResponse(socket.getOutputStream());
Servlet servletInstance = determineServletInstance(request);
FilterChain chain = determineFilterChainWithServlet(request, servletInstance);
chain.doFilter(request,response); // down the line invokes the servlet#service method
// do some cleanup, close streams, etc.
}
}
Determining the appropriate Servlet and Filter instances depends on the URL path in the request and the url-mappings you've configured. Tomcat (and every Servlet container) will only ever manage a single instance of a Servlet or Filter for each declared <servlet> or <filter> declaration in your deployment descriptor.
As such, every thread is potentially executing the service(..) method on the same Servlet instance.
That's what the Servlet Specification and the Servlet API, and therefore Tomcat, guarantee.
As for your Restful webservices, read this. It describes how a resource is typically an application scoped singleton, similar to the Servlet instance managed by a Servlet container. That is, every thread is using the same instance to handle requests.
On Google App Engine (GAE) it is possible for frontend instances to create up to 10 threads to maximize throughput. According to this page, such multi-threading can be accomplished as follows:
Runnable myTask = new Runnable({
#Override
public void run() {
// Do whatever
}
});
ThreadFactory threadFactory = ThreadManager.currentRequestThreadFactory();
// GAE caps frontend instances to 10 worker threads per instance.
threadFactory.newRequestThread(myTask);
To hit my GAE server-side, I'll expose many servlets mapped to certain URLs, such as the FizzServlet mapped to http://myapp.com/fizz:
public class FizzServlet extends HttpServlet {
#Override
public void doGet(HttpServletRequest request,
HttpServletResponse response) throws IOException {
// Handle the request here. Somehow send it to an available
// worker thread.
}
}
I guess I'm choking on how to connect these two ideas. As far as I see it, you have 3 different mechanisms/items here:
The App Engine instance itself, whose lifecycle I can "hook" by implementing a ServletContextListener and run custom code when GAE fires up the instance; and
This ThreadFactory/ThreadManager stuff (above)
The servlets/listeners
I guess I'm wondering how to implement code such that every time a new request comes into, say, FizzServlet#doGet, how to make sure that request gets sent to an available thread (if there is one available). That way, if FizzServlet was the only servlet I was exposing, it could get called up to 10 times before it would cause a new (11th) incoming request to hang while a previous request was processing.
I'm looking for the glue code between the servlet and this thread-creating code. Thanks in advance.
I guess I'm wondering how to implement code such that every time a new request comes into, say, FizzServlet#doGet, how to make sure that request gets sent to an available thread (if there is one available). That way, if FizzServlet was the only servlet I was exposing, it could get called up to 10 times before it would cause a new (11th) incoming request to hang while a previous request was processing.
That's what the GAE servlet engine does for you. You deploy an app containing a servlet, and when a request comes in, the servlet engine uses a thread to process the request and calls your servlet. You don't have anything to do.
If your servlet's doGet() or doPost() method, invoked by GAE, needs to perform several tasks in parallel (like contacting several other web sites for example), then you'll start threads by yourself as explained in the page you linked to.
In simple description, I have a servlet and response time is long so I decided to divide it into two parts, one just composes a response to client, and second let's say performs some business logic and stores result in DB. To decrease response time I execute business logic asynchronously using ThreadPoolExecutor in combination with ArrayBlockingQueue. Using ArrayBlockingQueue I can ensure original FIFO ordering if requests were sequential for the same client. This is important prerequisite.
Here is a snippet:
Servlet
public class HelloServlet extends HttpServlet {
AsyncExecutor exe = new AsyncExecutor();
protected void doGet(HttpServletRequest req,
HttpServletResponse resp) throws ServletException, IOException {
PrintWriter w = resp.getWriter();
exe.executeAsync(exe.new Task(this));
w.print("HELLO TO CLIENT");
}
protected void someBusinessMethod(){
// long time execution here
}
}
and Executor
public class AsyncExecutor {
static final BlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(10, true);
static final Executor executor = new ThreadPoolExecutor(3, 5, 20L, TimeUnit.SECONDS, queue);
public void executeAsync(Task t){
boolean isTaskAccepted = false;
while(!isTaskAccepted){
try {
executor.execute(t);
isTaskAccepted = true;
} catch (RejectedExecutionException e){
}
}
}
class Task implements Runnable{
private HelloServlet servlet;
Task(HelloServlet servlet){
this.servlet = servlet;
}
#Override
public void run() {
// just call back to servlet's business method
servlet.someBusinessMethod();
}
}
}
This implementation works fine if I deploy it only to one Tomcat node, since I have only one ArrayBlockingQueue in application. But if I have several nodes and load balancer in front then I can not guarantee FIFO ordering of requests for async execution for the same client since I have already several Queues.
My question is, how it is possible to guarantee the same order of requests to be executed asynchronously for the same client in clustered (multi node) deployment? I think ActiveMQ probably a solution (not preferable for me), or load balancer configuration, or can it be implemented in code?
Hope some of these ideas help.
Thanks Sam for you prompt suggestions.
In the first post I described a problem in very simplified way so to clarify it better let's say I have a legacy web app deployed to Tomcat and it serves some Licensing Model(old one). Then we got a new Licensing Model (this is a GlassFish app) and we need to use it alongside with old one to be in sync. For the end user such integration must be transparent and not intrusive. So user request is served like this.
caller send a request (create subscription for example)
execute business logic of the the new licensing model
execute business logic of the the old licensing model
despite the result of the p.3 return response of p.2 in format of old licensing model back to caller
(optional) handle failure of p.3 if any
This was implemented with Aspect which intercepts requests of p.1 and executes the rest of stuff sequentially. And as I said in previous post p.3 execution time can be long that's why I want to make it asynchronous. Let's have a look at snippet of Aspect (instead of Servlet from the first post).
#Aspect #Component public class MyAspect {
#Autowired
private ApplicationContext ctx;
#Autowired
private AsyncExecutor asyncExecutor;
#Around("#annotation(executeOpi)")
public Object around(ProceedingJoinPoint jp, ExecuteOpi executeOpi) throws Throwable {
LegacyMapper newModelExecutor = ctx.getBean(executeOpi.legacyMapper());
// executes a new model and then return result in the format of old model
Object result = newModelExecutor.executeNewModelLogic(joinPoint.getArgs());
// executes old model logic asynchronously
asyncExecutor.executeAsync(asyncExecutor.new Task(this, jp)
return object
}
public void executeOldModelLogic(ProceedingJoinPoint jp) throws Throwable{
// long time execution here
jp.proceed();
}
}
With this implementation as in the first post, I can guarantee a FIFO order of executeOldModelLogic methods, if requests come to the same Tomcat node. But with multi-node deployment and round robin LB in front I can end-up with such a case when for the same caller "update subscription in old model" can come first to ArrayBlockingQueue than "create subscription in old model", which of course a bad logical bug.
And as for points you suggested:
p1, p2 and p4: I probably can't use it as a solution since I don't have a state of object as such. You see that I pass to Runnable task a references of Aspect and JoinPoint to make a call back of executeOldModelLogic from Runnable to Aspect
p3 Don't know about this might be worthwhile to investigate
p5 This is a direction I want go for further investigation, I have a gut feeling it is only way to solve my problem in the given conditions.
There are some solutions that come to mind off hand.
Use the database: post the jobs to be run in a database table, have a secondary server process run the jobs, and place results in an output table. Then when users call back to the web page, it can pick up any results waiting for them from the output table.
Use a JMS service: this is a pretty lightweight messaging service, which would integrate with your Tomcat application reasonably well. The downside here is that you have to run another server side component, and build the integration layer with your app. But that's not a big disadvantage.
Switch to a full J2EE container (Java App Server): and use an EJB Singleton. I have to admit, I don't have any experience with running a Singleton across separate server instances, but I believe that some of them may be able to handle it.
Use EHCache or some other distributed cache: I built a Queue wrapper around EHCache to enable it to be used like a FIFO queue, and it also has RMI (or JMS) replication, so multiple nodes will see the same data.
Use the Load Balancer: If your load balancer supports session level balancing, then all requests for a single user session can be directed to the same node. In a big web environment where I worked we were unable to share user session state across multiple servers, so we set up load balancing to ensure that the user's session always was directed to same web server.
Hope some of these ideas help.