I got this servlet which return a pdf file to the client web browser.
We do not want to risk any chance that when the number of request is too much, the server is paralyzed.
We would like to make an application level (program) way to set a limit in the number of concurrent request, and return a error message to the browser when the limit is reached. We need to do it in applicantion level because we have different servlet container in development level(tomcat) and production level(websphere).
I must emphasize that I want to control the maximum number of request instead of session. A user can send multiple request over the server with the same session.
Any idea?
I've thought about using a static counter to keep track of the number of request, but it would raise a problem of race condition.
I'd suggest writing a simple servlet Filter. Configure it in your web.xml to apply to the path that you want to limit the number of concurrent requests. The code would look something like this:
public class LimitFilter implements Filter {
private int limit = 5;
private int count;
private Object lock = new Object();
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain) throws IOException, ServletException {
try {
boolean ok;
synchronized (lock) {
ok = count++ < limit;
}
if (ok) {
// let the request through and process as usual
chain.doFilter(request, response);
} else {
// handle limit case, e.g. return status code 429 (Too Many Requests)
// see https://www.rfc-editor.org/rfc/rfc6585#page-3
}
} finally {
synchronized (lock) {
count--;
}
}
}
}
Or alternatively you could just put this logic into your HttpServlet. It's just a bit cleaner and more reusable as a Filter. You might want to make the limit configurable through the web.xml rather than hard coding it.
Ref.:
Check definition of HTTP status code 429.
You can use RateLimiter. See this article for explanation.
You might want to have a look on Semaphore.
Semaphores are often used to restrict the number of threads than can access some (physical or logical) resource.
Or even better try to figure it out with the server settings. That would of course be server-dependant.
I've thought about using a static counter to keep track of the number of request, but it would raise a problem of race condition.
If you use a AtomicInteger for the counter, you will not have the problem of race conditions.
An other way would be using the Java Executor Framework (comes with Java 1.5). There you are able to limit the number of running threads, and block new once until there is a new free thread.
But I think the counter would work and be the easyest solution.
Attention: put the counter relese in a finally block!
//psydo code
final AtomicInteger counter;
...
while(true) {
int v = counter.getValue()
if (v > max) return FAILURE;
if(counter.compareAndSet(v, v+1)) break;
}
try{
doStuff();
} finally{
counter.decrementAndGet();
}
If you are serving static files, it's unlikely that the server will crash. The bottleneck would be the network throughput, and it degrades gracefully - when more requests come in, each still get served, just a little bit slower.
If you set a hard limit on total requests, remember to set a limit on requests per IP. Otherwise, it's easy for one bad guy to issue N requests, deliberately read the responses very slowly, and totally clog your service. This works even if he's on a dialup and your server network has a vast throughput.
Related
I have a route defined in Camel that goes something like this: GET request comes in, a file gets created in the file system. File consumer picks it up, fetches data from external web services, and sends the resulting message by POST to other web services.
Simplified code below:
// Update request goes on queue:
from("restlet:http://localhost:9191/update?restletMethod=post")
.routeId("Update via POST")
[...some magic that defines a directory and file name based on request headers...]
.to("file://cameldest/queue?allowNullBody=true&fileExist=Ignore")
// Update gets processed
from("file://cameldest/queue?delay=500&recursive=true&maxDepth=2&sortBy=file:parent;file:modified&preMove=inprogress&delete=true")
.routeId("Update main route")
.streamCaching() //otherwise stuff can't be sent to multiple endpoints
[...enrich message from some web service using http4 component...]
.multicast()
.stopOnException()
.to("direct:sendUpdate", "direct:dependencyCheck", "direct:saveXML")
.end();
The three endpoints in the multicast are simply POSTing the resulting message to other web services.
This all works rather well when the queue (i.e. the file directory cameldest) is fairly empty. Files are being created in cameldest/<subdir>, picked up by the file consumer and moved into cameldest/<subdir>/inprogress, and stuff is being sent to the three outgoing POSTs no problem.
However, once the incoming requests pile up to about 300,000 files progress slows down and eventually the pipeline fails due to out-of-memory errors (GC overhead limit exceeded).
By increasing logging I can see that the file consumer polling basically never runs, because it appears to take responsibility for all files it sees at each time, waits for them to be done processing, and only then starts another poll round. Besides (I assume) causing the resources bottleneck, this also interferes with my sorting requirements: Once the queue is jammed with thousands of messages waiting to be processed, new messages that would naively be sorted higher up are -if they even still get picked up- still waiting behind those that are already "started".
Now, I've tried the maxMessagesPerPoll and eagerMaxMessagesPerPoll options. They seem to alleviate the problem at first, but after a number of poll rounds I still end up with thousands of files in "started" limbo.
The only thing that sort of worked was making the bottle neck of delay and maxMessages... so narrow that the processing on average would finish faster than the file polling cycle.
Clearly, that is not what I want. I would like my pipeline to process files as fast as possible, but not faster. I was expecting the file consumer to wait when the route is busy.
Am I making an obvious mistake?
(I'm running a somewhat older Camel 2.14.0 on a Redhat 7 machine with XFS, if that is part of the problem.)
Try set maxMessagesPerPoll to a low value on the from file endpoint to only pickup at most X files per poll which also limits the total number of inflight messages you will have in your Camel application.
You can find more information about that option in the Camel documentation for the file component
The short answer is that there is no answer: The sortBy option of Camel's file component is simply too memory-inefficient to accomodate my use-case:
Uniqueness: I don't want to put a file on queue if it's already there.
Priority: Files flagged as high priority should be processed first.
Performance: Having a few hundred thousands of files, or maybe even a few million, should be no problem.
FIFO: (Bonus) Oldest files (by priority) should be picked up first.
The problem appears to be, if I read the source code and the documentation correctly, that all file details are in memory to perform the sorting, no matter whether the built-in language or a custom pluggable sorter is used. The file component always creates a list of objects containing all details, and that apparently causes an insane amount of garbage collection overhead when polling many files often.
I got my use case to work, mostly, without having to resort to using a database or writing a custom component, using the following steps:
Move from one file consumer on the parent directory cameldest/queue that sorts recursively the files in the subdirectories (cameldest/queue/high/ before cameldest/queue/low/) to two consumers, one for each directory, with no sorting at all.
Set up only the consumer from /cameldest/queue/high/ to process files through my actual business logic.
Set up the consumer from /cameldest/queue/low to simply promote files from "low" to "high" (copying them over, i.e. .to("file://cameldest/queue/high");)
Crucially, in order to only promote from "low" to "high" when high is not busy, attach a route policy to "high" that throttles the other route, i.e. "low" if there are any messages in-flight in "high"
Additionally, I added a ThrottlingInflightRoutePolicy to "high" to prevent it from inflighting too many exchanges at once.
Imagine this like at check-in at the airport, where tourist travellers are invited over into the business class lane if that is empty.
This worked like a charm under load, and even while hundreds of thousands of files were on queue in "low", new messages (files) dropped directly into "high" got processed within seconds.
The only requirement that this solution doesn't cover, is the orderedness: There is no guarantee that older files are picked up first, rather they are picked up randomly. One could imagine a situation where a steady stream of incoming files could result in one particular file X just always being unlucky and never being picked up. The chance of that happening, though, is very low.
Possible improvement: Currently the threshold for allowing / suspending the promotion of files from "low" to "high" is set to 0 messages inflight in "high". On the one hand, this guarantees that files dropped into "high" will be processed before another promotion from "low" is performed, on the other hand it leads to a bit of a stop-start-pattern, especially in a multi-threaded scenario. Not a real problem though, the performance as-is was impressive.
Source:
My route definitions:
ThrottlingInflightRoutePolicy trp = new ThrottlingInflightRoutePolicy();
trp.setMaxInflightExchanges(50);
SuspendOtherRoutePolicy sorp = new SuspendOtherRoutePolicy("lowPriority");
from("file://cameldest/queue/low?delay=500&maxMessagesPerPoll=25&preMove=inprogress&delete=true")
.routeId("lowPriority")
.log("Copying over to high priority: ${in.headers."+Exchange.FILE_PATH+"}")
.to("file://cameldest/queue/high");
from("file://cameldest/queue/high?delay=500&maxMessagesPerPoll=25&preMove=inprogress&delete=true")
.routeId("highPriority")
.routePolicy(trp)
.routePolicy(sorp)
.threads(20)
.log("Before: ${in.headers."+Exchange.FILE_PATH+"}")
.delay(2000) // This is where business logic would happen
.log("After: ${in.headers."+Exchange.FILE_PATH+"}")
.stop();
My SuspendOtherRoutePolicy, loosely built like ThrottlingInflightRoutePolicy
public class SuspendOtherRoutePolicy extends RoutePolicySupport implements CamelContextAware {
private CamelContext camelContext;
private final Lock lock = new ReentrantLock();
private String otherRouteId;
public SuspendOtherRoutePolicy(String otherRouteId) {
super();
this.otherRouteId = otherRouteId;
}
#Override
public CamelContext getCamelContext() {
return camelContext;
}
#Override
public void onStart(Route route) {
super.onStart(route);
if (camelContext.getRoute(otherRouteId) == null) {
throw new IllegalArgumentException("There is no route with the id '" + otherRouteId + "'");
}
}
#Override
public void setCamelContext(CamelContext context) {
camelContext = context;
}
#Override
public void onExchangeDone(Route route, Exchange exchange) {
//log.info("Exchange done on route " + route);
Route otherRoute = camelContext.getRoute(otherRouteId);
//log.info("Other route: " + otherRoute);
throttle(route, otherRoute, exchange);
}
protected void throttle(Route route, Route otherRoute, Exchange exchange) {
// this works the best when this logic is executed when the exchange is done
Consumer consumer = otherRoute.getConsumer();
int size = getSize(route, exchange);
boolean stop = size > 0;
if (stop) {
try {
lock.lock();
stopConsumer(size, consumer);
} catch (Exception e) {
handleException(e);
} finally {
lock.unlock();
}
}
// reload size in case a race condition with too many at once being invoked
// so we need to ensure that we read the most current size and start the consumer if we are already to low
size = getSize(route, exchange);
boolean start = size == 0;
if (start) {
try {
lock.lock();
startConsumer(size, consumer);
} catch (Exception e) {
handleException(e);
} finally {
lock.unlock();
}
}
}
private int getSize(Route route, Exchange exchange) {
return exchange.getContext().getInflightRepository().size(route.getId());
}
private void startConsumer(int size, Consumer consumer) throws Exception {
boolean started = super.startConsumer(consumer);
if (started) {
log.info("Resuming the other consumer " + consumer);
}
}
private void stopConsumer(int size, Consumer consumer) throws Exception {
boolean stopped = super.stopConsumer(consumer);
if (stopped) {
log.info("Suspending the other consumer " + consumer);
}
}
}
I would propose an alternative solution unless you really need to save the data as files.
From your restlet consumer, send each request to a message queuing app such as activemq or rabbitmq or something similar. You will quickly end up with lots of messages on that queue but that is ok.
Then replace your file consumer with a queue consumer. It will take some time but the each message should be processed separately and sent to wherever you want. I have tested rabbitmq with about 500 000 messages and that has worked fine. This should reduce the load on the consumer as well.
I have an Akka actor that owns an AsyncHttpClient. This actor must handles a lot of asynchronous requests. Because my system cannot handle thousands of requests simultaneously, I need to limit the number of concurrent requests.
Right now, I'm doing this :
AsyncHttpClientConfig config = new AsyncHttpClientConfig.Builder().setAllowPoolingConnection(true)
.addRequestFilter(new ThrottleRequestFilter(32))
.setMaximumConnectionsPerHost(16)
.setMaxRequestRetry(5)
.build();
final AsyncHttpClient httpClient = new AsyncHttpClient(new NettyAsyncHttpProvider(config));
When my actor receives a message, I use the client like this :
Future<Integer> f = httpClient.prepareGet(url).execute(
new AsyncCompletionHandler<Integer>() {
#Override
public Integer onCompleted(Response response) throws Exception {
// handle successful request
}
#Override
public void onThrowable(Throwable t){
// handle failed request
}
}
);
The problem is that requests are never put in the client queue and are all processed like the configuration doesn't matter. Why doesn't this work as it should?
From the maintainer:
setMaxConnectionsPerHost only caps the number of connections that can be open to a given host. There's no built-in queuing mechanism for requests that might need a connection while there's none available.
So basically, it's a hard limit. Also, in versions of the library prior to, I believe, 1.9.10, the maximumConnectionsPerHost field was not being properly utilized by the code to limit the number of concurrent connections per host. Instead, there was a bug where the client only looked at the maximumConnectionsTotal field.
Link to issue referenced on GitHub
My deployed server has sometimes long response times, while working and developing at localhost all calls are really fast.
This has made my application enter unexpected behaviour once deployed a few times due to problems with resource loading taking too long.
I'd like to simulate in my local tests the bad connection with my real server, therefore I want to add a random delay to every request-response and my first thought was to use Thread.sleep in the servlet:
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
//add delay before processing request
if (DELAY > 0){
int delay;
if (RANDOMIZE){
delay = Random.nextInt(DELAY);
} else {
delay = DELAY;
}
try {
Thread.sleep(delay);
} catch (InterruptedException e1) {
logger.error(e1);
}
}
...
However I have read that one should not use Thread.sleep() inside a servlet, but the context of such discouragement and their solutions are drastically different from my case, can I use thread.sleep() in this context?
EDIT: This is of course only for local and for the client to be strained a bit in the local tests... I just want to simulate the bad network I've encountered in reality!
I think this whole approach is flawed. I wouldn't introduce a random delay (how are you going to repeat test cases?). You can introduce a Thread.sleep(), but I wouldn't. Would this be in your production code ? Is it configurable ? What happens if it's accidentlally turned on in production ?
I would rather set up a test server with the exact characteristics of your production environment. That way you can not only debug effectively, but build a regression test suite that will allow you to develop effectively, knowing how the application will perform in production.
Perhaps the one concession to the above is to introduce network delays (as appropriate) between client and server if your users are geographically diverse. That's often done using a hardware device on the network and wouldn't affect your code or configuration.
I did this to get delay :
response.setContentType("text/html;charset=UTF-8");
try (PrintWriter out = response.getWriter()) {
out.println("<meta http-equiv=\"Refresh\" content=\"3;url=home.jsp/\">");
}
Remember that in content=\"3;url=home.jsp/\", 3 is the delay seconds and home.jsp is the page you want to go to after the given seconds.
I'm trying to run a Jetty Server that can have a number of people connect to the server and see a list of print outs. I want everybody who connects to see the same values printed out.
For instance, if I have a single list keeping track of the time and I want 5 or so people to be able to go to my website (e.g. localhost:8080/time) and have them all see what time it is every 30 seconds, how would i set that up?
What I have:
I am using Jetty.
I created a single server bound to port 8080.
I created my own handler that extends AbstractHandler
this writes to the screen saying when an event has transpired (i.e. 30 seconds have passed)
If two people connect to this page, they each see a print out every minute (that is it switches back and forth letting each person know when every other event has transpired)
If 3 people connect, only two can stay connected and the third just spins getting no output to the screen
I have not set up an Connectors of my own since my attempts to do so have been unsuccessful and i'm not sure how I understand if that is the solution to my problem.
Any help would be much appreciated and if anybody has some idea but needs some clarification on what I am doing I would be glad to give more details.
Thanks!
Handler code:
#Override
public void handle(String target, Request request, HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) throws IOException, ServletException
{
httpServletResponse.setContentType("text/html;charset=utf-8");
httpServletResponse.setStatus(HttpServletResponse.SC_OK);
request.setContextPath("/time");
request.setHandled(true);
while (true) {
synchronized(Main.list) {
while (!Main.list.isEmpty()) {
Double time = Main.list.get(0);
httpServletResponse.getWriter().println("<h1>The time now is " + time + "</h1>");
httpServletResponse.flushBuffer();
Main.list.remove(0);
}
try {
Main.list.wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
So the list object is a static ArrayList defined in the Main class that I wake up (i.e. notify) every 30 seconds. Hopefully this helps someone understand more what I am talking about as i'm not sure what I could change in the handler...
How are you feeding clients into your handler? Browsers have limits to the number of connections are made to to a particular host, perhaps your seeing that.
there is nothing intrinsically wrong that handler code aside from it being a generally odd thing to see in a handler
I got this servlet which return a pdf file to the client web browser.
We do not want to risk any chance that when the number of request is too much, the server is paralyzed.
We would like to make an application level (program) way to set a limit in the number of concurrent request, and return a error message to the browser when the limit is reached. We need to do it in applicantion level because we have different servlet container in development level(tomcat) and production level(websphere).
I must emphasize that I want to control the maximum number of request instead of session. A user can send multiple request over the server with the same session.
Any idea?
I've thought about using a static counter to keep track of the number of request, but it would raise a problem of race condition.
I'd suggest writing a simple servlet Filter. Configure it in your web.xml to apply to the path that you want to limit the number of concurrent requests. The code would look something like this:
public class LimitFilter implements Filter {
private int limit = 5;
private int count;
private Object lock = new Object();
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain) throws IOException, ServletException {
try {
boolean ok;
synchronized (lock) {
ok = count++ < limit;
}
if (ok) {
// let the request through and process as usual
chain.doFilter(request, response);
} else {
// handle limit case, e.g. return status code 429 (Too Many Requests)
// see https://www.rfc-editor.org/rfc/rfc6585#page-3
}
} finally {
synchronized (lock) {
count--;
}
}
}
}
Or alternatively you could just put this logic into your HttpServlet. It's just a bit cleaner and more reusable as a Filter. You might want to make the limit configurable through the web.xml rather than hard coding it.
Ref.:
Check definition of HTTP status code 429.
You can use RateLimiter. See this article for explanation.
You might want to have a look on Semaphore.
Semaphores are often used to restrict the number of threads than can access some (physical or logical) resource.
Or even better try to figure it out with the server settings. That would of course be server-dependant.
I've thought about using a static counter to keep track of the number of request, but it would raise a problem of race condition.
If you use a AtomicInteger for the counter, you will not have the problem of race conditions.
An other way would be using the Java Executor Framework (comes with Java 1.5). There you are able to limit the number of running threads, and block new once until there is a new free thread.
But I think the counter would work and be the easyest solution.
Attention: put the counter relese in a finally block!
//psydo code
final AtomicInteger counter;
...
while(true) {
int v = counter.getValue()
if (v > max) return FAILURE;
if(counter.compareAndSet(v, v+1)) break;
}
try{
doStuff();
} finally{
counter.decrementAndGet();
}
If you are serving static files, it's unlikely that the server will crash. The bottleneck would be the network throughput, and it degrades gracefully - when more requests come in, each still get served, just a little bit slower.
If you set a hard limit on total requests, remember to set a limit on requests per IP. Otherwise, it's easy for one bad guy to issue N requests, deliberately read the responses very slowly, and totally clog your service. This works even if he's on a dialup and your server network has a vast throughput.