I am in the process of writing a web-app that uses multiple web APIs.
For a single request of a single user, my app may need to perform up to 30 HTTP requests to other sites. The site housing the web-app can have hundreds of concurrent users.
I've been looking around trying to figure out which library should I use. I'm looking for a mature project that has detailed documentation and tested code, one that will still be around in years to come. Not sure if something like that exists (!)
Couple of questions :
In a case such as described above, should I be using an asynchronous HTTP client (without threading), or a regular (possibly pooled) HTTP client (with threading)? Asynchronicity relieves my app from using threads, but makes the code more scattered - will the above mentioned number of requests burden my server too much? (it says here that asynchronous is more scalable)
Which library is the common one to use? Is it Apache HttpComponenets HttpClient or its asynch couterpart HttpAsynchClient - which is in Alpha...)? How about jfarcand's AsyncHttpClient?
Okay, let's say I will use threads.
After digging around I realize that spawning threads from within a servlet (in my case - a Struts action), may be a big No No :
related questions:
What is recommended way for spawning threads from a servlet in Tomcat
Need help with java web app design to perform background tasks
Can i spawn a thread from a servlet ?
The way I see it, these are my options:
use my own thread pool (container doesn't manage my threads)
use a WorkManager such as CommonJ (seems like an inactive product)
use a 3rd party scheduler such as Quartz (may be an overkill ... ?)
I would appreciate any recommendations for this specific use case - aggregating lotsa data from different web services (this aggregation is invoked by a single user's single request).
Good question. I would try an asynchronous solution first to see how everything works. The asynchronous solution would be the simplest to implement.
If that doesn't work, try a more threaded model.
I would use HttpClient for making your requests. I've worked with it a lot and use it for any http work that I have to do.
A single thread for each remote http connection, and using a synchronous http client will probably be easier. I would try this approach first, and see if it is fast/scalable enough. For the synchronous approach, apache http client is a good choice.
If a synchronous solution is not good enough, something like netty may be a good fit. It uses NIO so you won't get thousands of threads.
I do not know of any existing software to do this for you that will not be overkill. But you might try splitting things up. That is, separate the fetching of the data of the showing of the result. Since you do not provide no further details on the problem at hand I cannot say for you whether that would be feasible or not.
Basically the idea is to create a service that will fetch those 30 subsequent requests for you and if possible process it into a request. The client of this service is the service that is running on the web. It will receive a request from a user and subsequently put it's own request trough to your data service. When the data service is ready it will return it's response. Either synchronously or asynchronously.
You could program your data service in any language you want, even Java, without being bound to servlets ensuring that fetching the subsequent 30 request and combining them in a response isn't being done by the webserver. This could also enhance the responsiveness of your web server itself.
Nutshell: branch of the ``difficult" tasks to a specialised service where you can transparently handle parallelism.
I would use Jetty and in the servlets I'd use the continuation mechanism to free up the thread while waiting for the third party web request to complete. This will allow maximum concurrency on your server, as you can have a lot more suspended requests than threads.
You can use either continuations or the servlet 3.0 asynchronous API, the end result is the same.
Related
Is making REST based web service (POST) asynchronous is the best way to handle thousands of requests at one time (Keeping in mind that I have only single instance of server serving the request)?
Edited:
Jersey is wrongly tagged.
For eg: I have a rest based web service, which is supposed to be consumed by 100 thousand clients within a very short span of time (~60 seconds). I understand that if I am allowed to deploy multiple instance of the server, then I can use a load balancer to handle all my incoming request and delegate them accordingly. But I am restricted to use only single instance. What design could I opt within this restriction?
I could think of making the request asynchronous( which will not respond to client immediately ) in order to be able to let the server be free from this load and handle the requests at it's own pace.
For now we can ignore memory limitations.
Please let me know if this clarifies your doubt?
The term asynchronous could have different meanings in different places. For a web application code, it could refer to a Nonblocking I/O server such as Node or Netty/Akka which is a way for HTTP Requests to time multiplex on the same worker threads. If you're writing callbacks or using async or future constructs, it probably is non-blocking I/O which people sometimes refer to as asynchronous.
However, I could have REST API running on Node which implements non-blocking I/O, but the API or the overall architecture is still fully synchronous. For example, let's say I have an API endpoint POST /photos, which takes in a photo, creates image thumbnails, stores the URLs of the photo in a SQL Db and then stores the images in S3. The REST API could still block from the initial POST until after the image is processed and stored.
A second way is for the server to accept the photo process as a job and return immediately. Then the server could store the photo in a in memory or network based queue to be processed later by some other worker thread. In fact, I could even implement this async architecture even with a blocking server like some good old Java 7 and Jetty.
So, I have this old legacy JAX-WS service, that does a lot of IO per request, so each request takes quite a bit of time, but doesn't consume much CPU/RAM.
With amount of clients increasing lately, there's a huge problem of thread starvation.
At first, I thought about implementing JAX-WS builtin async support, but it requires clients to be aware of it, and thats a no in my situation. I cannot possibly force them all to update their code.
My current idea is to create special async servlet and manually parse SOAP request and encode the response, but I can't seem to find some good way to do that.
Is there any way to utilize Servlet 3.1 async support when handling JAX-WS requests?
I should clarify, that I cannot use some kind of queue, or otherwise "just return early", because request processing can fail, and client must get that failure.
I've found the solution that works perfectly for me, CXF Continuations API.
http://cxf.apache.org/docs/continuations.html
https://github.com/search?l=java&q=ContinuationProvider&type=Code&utf8=%E2%9C%93
I had to enable async for CXF Servlet, and add jboss module dependency to bundled CXF.
While the whole things feels somewhat like a hack, it allowed me to do proper asynchronous processing without changing service external API at all.
As a bonus, I can even decide whether to suspend request, or process it normally, which helps a lot in certain cases.
You could make use of workflow here where say you have webserver's workers accept the request, you handle it and places it in a queue which is shared by thread which will process this event asynchronously.
This would just mean that you have low wait time on client side and you process the client request asynchronously. In that way, you build your system scalable and can increase number of worker threads on either of the side i.e. webserver so you could accept multiple requests from client concurrently and at the same time multiple threads may be concurrently processing the events from client.
I've been reading about asynchronous servlets in an attempt to see if it fits my use case.
The server will receive long-running requests that take time to read from the db, and process each message.
Here is the basic flow:
Client makes a request to validate some data.
Server receives a request for the long-running process (~1-2 minutes)
Request needs to talk to a db (redis, in my case) to grab 100,000 docs in batches.
Processing occurs for each doc, and if it is valid, it is written to the client. Similarly, upto 25 sample docs are written to the client.
When the full data set is processed (but no more than 25 docs are written), a result is calculated and sent.
Connection is successfully closed.
So there is a db wait, which all similar requests will have, and then the processing for each doc, which happens in the same thread.
Is there any benefit to switching to async servlets versus a regular synchronous servlet in this case?
Additionally, if I do use an async servlet, would there be any benefit in also trying to parallelize the processing of the docs with thread pools? It seems like I won't be getting any additional benefit, but want to clarify this as well.
Thanks for your help!
Is there any benefit to switching to async servlets versus a regular synchronous servlet in this case?
Yes. In fact it is the use case that all manuals tells you "async servlets are useful for this"
Additionally, if I do use an async servlet, would there be any benefit in also trying to parallelize the processing of the docs with thread pools? It seems like I won't be getting any additional benefit, but want to clarify this as well.
Unrelated. Async servlets just releases resources so your server keeps answering petitions even if lots of servlets are waiting for results. It is not related to the speed of answering the asynchronized request.
Any benefit that you get from optimizing your logic will be good for the server and the user, but it would be good either if you are using async servlets or sync servlets.
I need to perform an asynchronous task when a RESTful web service endpoint is called. Effectively, the endpoint is asked to perform a body of work with a POST operation. It should immediately return a 200 OK to the caller, spawn a thread and perform it's resource intensive task. On completion, the thread would then POST to a corresponding endpoint on the caller (another REST server) indicating success (passing a token that represents the initial transaction request).
What are the best practice approaches for performing asynchronous actions inside a servlet that I should be aware of?
Servlet 3.0 has support for asynchronous operations. Tomcat 7.0 is already stable, so you can get it and try the new features.
If you don't need to output data continously, but to simply start a background process, then you can use any asynchronous mechanism available:
Executors framework
new Thread(new Runnable())
#Async (spring/ejb)
Aside from the complexities of async coding in Java, another "best practice" in RESTful web services is to use HTTP status codes to describe your server's response as accurately as possible. Unless you have a compelling reason to stick with 200 (ie a client which you can't change expects this), you should return HTTP 202:
202 Accepted
The request has been accepted for processing, but the processing has not been completed.
The only advice for your scenario would be to use thread pool rather than creating a new thread per request. In Java it is very easy, just create pool once during application startup (look at Executors class) and submit new tasks to it each time you need to perform some asynchronous operation. For your scenario this tasks will perform resource intensive operations and second REST call from within a different thread, long after the original request was served with 200.
In a Java EE container, the recommended way is to use the WorkManager API (JSR-237). Check this article for an overview.
A J2EE container can serve multiple
users at the same time (in parallel)
by managing a pool of threads, but for
various reasons opening independent
threads within a single J2EE container
is not recommended. Some containers'
security managers will not allow user
programs to open threads. Moreover, if
some containers allowed opening
threads, then they would not manage
those threads and therefore no
container-managed services would be
available for the threads.
I am writing a web service in Java which needs to handle a large number of requests / second. The general flow will be:
Web service receives a request from client
Returns a 'keep polling me' response to client
Calls another web service (or
services), and waits for them to
respond (with a timeout)
Client polls our web service, until
it receives a response (with a
timeout)
Researching on the Internet, I have found two general approaches to writing web services:
Spawn a thread for each request
Use the Reactor pattern (central dispatcher thread responds to IO events)
Do you have a recommendation for which approach is generally better, and what are the pros/cons of each approach? I would also appreciate pointers to examples.
Don't think multi-threading. Think asynchronously. I happened to have just coded an async handler that ran 2,000 RPS with <10 threads in IIS. Not sure how java works since I'm a .net guy but I gotta believe they have similar BeginXXX/EndXXX methods. If you ever spawn a thread then you're not considering all the places your code can block: data base IO, File I/O, web services, etc. These are the places your performance will cause your site to be slow.
Async, Async, Async.
Chant and repeat.
In addition to "No Refunds No Returns" response, I'd say yeah "Think Asynchronously" as you should be allowing your container to manage the multi-threading/scalability and high-availability issues of the web services it has deployed, this allows you to set-up things like clustering and so forth using your application container.
EDIT: So in conclusion, there isn't a pattern as such, maybe you should explore the scalability/availability features of your application container...
Asynchronism is indeed the right approach but don't manage this yourself, use something that supports asynchronous web service invocation like JAX-WS 2.0 (which uses the Future interface and/or the Executor framework from java.util.concurrent). See Asynchronous Web Service Invocation with JAX-WS 2.0.