My Servler spends quite some time in reading request.getInputStream() and writing to response.getOutputStream(). In the long run, this can be a problem as its blocking a thread for nothing but reading/writing literally a few bytes per second. (*)
I'm never interested in a partial request data, the processing should not start before the request is completely available. Similarly for the response.
I guess, asynchronous IO would solve it, but I wonder what's the proper way. Maybe a servlet Filter replacing the ServletInputStream by a wrapped ByteArrayInputStream, using request.startAsync and calling the chained servlet after having collected the whole input?
Is there already such a filter?
Should I write one or should I use a different approach?
Note that what I mean is to avoid wasting threads on slow servlet streams. This isn't the same as startAsync which avoids wasting threads just waiting for some event.
And yes, at the moment it'd be a premature optimization.
My read loop as requested
There's nothing interesting in my current input stream reading method, but here you are:
private byte[] getInputBytes() throws IOException {
ServletInputStream inputStream = request.getInputStream();
final int len = request.getContentLength();
if (len >= 0) {
final byte[] result = new byte[len];
ByteStreams.readFully(inputStream, result);
return result;
} else {
return ByteStreams.toByteArray(inputStream);
}
}
That's all and it blocks when data aren't available; ByteStreams come from Guava.
Summary of my understanding so far
As the answers clearly state, it's impossible to work with servlet streams without wasting a thread on them. Neither the servlet architecture nor the common implementation expose anything allowing to say "buffer the whole data and call me only when you collected everything", albeit they use NIO and could do it.
The reason may be that usually a reverse proxy like nginx gets used, which can do it. nginx does this buffering by default and it couldn't be even switched off until two years ago.
Actually a supported case???
Given that many negative answer, I'm not sure, but it looks like my goal
to avoid wasting threads on slow servlet streams
is actually fully supported: Since 3.1, there's ServletInputStream.html#setReadListener which seems to be meant exactly for this. The thread allocated for processing Servlet#Service initially calls request.startAsync(), attaches the listener and gets returned to the pool by simply returning from service. The listener implements onDataAvailable(), which gets called when it's possible to read without blocking, adds a piece of data and returns. In onAllDataRead(), I can do the whole processing of the collected data.
There's an example, how it can be done with Jetty. It seems to cover non-blocking output as well.
(*) In the logfiles, I can see requests taking up to eight seconds which get spend on reading the input (100 bytes header + 100 bytes data). Such cases are rare, but they do happen, although the server is mostly idle. So I guess, it's a mobile client on a very bad connection (some users of ours connect from places having such bad connectivity).
HttpServletRequest#startAsync() isn't useful for this. That's only useful for push things like web sockets and the good 'ol SSE. Moreover, JSR356 Web Socket API is built on top of it.
Your concrete problem is understood, but this definitely can't be solved from the servlet on. You'd only end up wasting yet more threads for the very simple reason because the container has already dedicated the current thread to the servlet request until the request body is read fully up to the last bit, even if it's ultimately read by a newly spawned async thread.
To save threads, you actually need a servletcontainer which supports NIO and if necessary turn on that feature. With NIO, a single thread can handle as many TCP connections as the available heap memory allows it, instead of that a single thread is allocated per TCP connection. Then, in your servlet you don't at all need to worry about this delicate I/O task.
Almost all modern servletcontainers support it: Undertow (WildFly), Grizzly (GlassFish/Payara), Tomcat, Jetty, etc. Some have it by default enabled, others require extra configuration. Just refer their documentation using the keyword "NIO".
If you'd actually also want to save the servlet request thread itself, then you'd basically need to go a step back, drop servlets and implement a custom NIO based service on top of an existing NIO connector (Undertow, Grizzly, Jetty, etc).
You can't. The Servlet container allocates the thread to the request, and that's the end of it, it's allocated. That's the model. If you don't like that, you will have to stop using Servlets.
Even if you could solve (1), you can't start async I/O on an input stream.
The way to handle slow requests is to time them out, by setting the appropriate setting for whatever container you're using ... if you actually have a problem, and it's far from clear that you really do, with a mostly idle server and this only happening rarely.
Your read loop makes a distinction without a difference. Just read the request input stream to its end. The servlet container already ensures that end of stream happens at the content-length if provided.
There's a class called org.apache.catalina.connector.CoyoteAdapter, which is the class that receives the marshaled request from TCP worker thread. It has a method called "service" which does the bulk of the heavy lifting. This method is called by another class: org.apache.coyote.http11.Http11Processor which also has a method of the same name.
I find it interesting that I see so many hooks in the code to handle async io, which makes me wonder if this is not a built in feature of the container already? Anyway, with my limited knowledge, the best way that I can think of to implement the feature you are talking about, would be to create a class:
public class MyAsyncReqHandlingAdapter extends CoyoteAdapter and #Override service() method and roll your own... I don't have the time to devote to doing this now, but I may revisit in the future.
In this method you would need a way to identify slow requests and handle them, by handing them off to a single threaded nio processor and "complete" the request at that level, which, given the source code:
https://github.com/apache/tomcat/blob/075920d486ca37e0286586a9f017b4159ac63d65/java/org/apache/coyote/http11/Http11Processor.java
https://github.com/apache/tomcat/blob/3361b1321201431e65d59d168254cff4f8f8dc55/java/org/apache/catalina/connector/CoyoteAdapter.java
You should be able to figure out how to do. Interesting question and yes it can be done. Nothing I see in the spec says that it cannot...
Related
I am using an http library to fetch data that is 200 mb in size. Each line in the data is then processed. To save memory I would like to process the data line by line as the data is streamed in rather than waiting for all 200 mb to be downloaded first.
The http library I am using exposes a method that looks something like OnCharReceived(CharBuffer buffer) that can be overridden so that I can in effect process each chunk of data as it comes in.
I would like to expose this data as an InputStream. My first thought was to use a PipedInputStream and PipedOutputStream pair where in OnCharReceived() I would write to the PipedOutputStream and in my thread read from the PipedInputStream. However, this seems to have the problem that the underlying buffer of the pipe could get full requiring the writing thread to block in OnCharReceived until my thread gets around to processing data. But blocking in OnCharReceived would probably be blocking in the http library's IO thread and would be very bad.
Are there Java classes out there that handle the abstract problem I need to solve here without me having to roll my own custom implementation. I know of things like BlockingQueue that could be used as part of a larger solution. But are there any simple solutions.
For reasons of legacy code I really need the data exposed as an InputStream.
Edit: To be more precise I am basing my code on the following example from the apache http async library
https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchangeStreaming.java
If there's a simpler solution I would not get near Piped[In/Out]putStream. It introduces unnecessary complicated threading concerns as you pointed out. Keep in mind you can always write to a temp file and then read from the file as an InputStream. This also has the advantage of closing the HTTP connection as fast as possible and avoid timeouts.
There might be other solutions depending on the API you are using but I think the proposed solution still makes sense for the reasons above.
I am building a server in java that communicates with several clients at the same time, the initial approach we had is the the server listens to connections from the clients, once a connection is received and a socket is created, a new thread is spawned to handle the communication with each client, that is read the request with an ObjectInputStream, do the desired operation (fetch data from the DB, update it, etc.), and send back a response to the client (if needed). While the server itself goes back to listen to more connections.
This works fine for the time being, however this approach is not really scalable, it works great for a small amount of clients connected at the same time, however since every client spawns another thread, what will happen when there are a too many clients connected at once?
So my next idea was to maintain a list of sorts that will hold all connected clients (the socket object and some extra info), use a ThreadPool for to iterate through them and read anything they sent, if a message was received then put it in a queue for execution by another ThreadPool of worker threads, and once the worker has finished with its task if a response is required then send it.
The 2 latter steps are pretty trivial to implement, the problem is that with the original thread per client implementation, I use ObjectInputStream.readObject() to read the message, and this method blocks until there is something to read, which is fine for this approach, but I can't use the same thing for the new approach, since if I block on every socket, I will never get to the ones that are further down the list.
So I need a way to check if I have anything to read before I call readObject(), so far I tried the following solutions:
Solution 1:
use ObjectInputStream.available() to check if there is anything available to read, this approach failed since this method seems to always return 0, regardless of whether there is an object in the stream or not. So this does not help at all.
Solution 2:
Use PushbackInputStream to check for the existence of the first unread byte in the stream, if it exists then push it back and read the object using the ObjectInputStream, and if it doesn't move on:
boolean available;
int b = pushbackinput.read();
if (b==-1)
available = false;
else
{
pushbackinput.unread(b);
available = true;
}
if (available)
{
Object message= objectinput.readObject();
// continue with what you need to do with that object
}
This turned out to be useless too, since read() blocks also if there is no input to read. It seems to only return the -1 option if the stream was closed. If the stream is still open but empty it just blocks, so this is no different than simply using ObjectInputStream.readObject();
Can anyone suggest an approach that will actually work?
This is a good question, and you've done some homework.... but it involves going through some history to get things right. Note, your issue is actually more to do with the socket-level communication rather than the ObjectInputStream:
The easiest way to do things in the past was to have a separate thread per socket. This was scalable to a point but threads were expensive and slow to create.
In response, for large systems, people created thread pools and would service the sockets on threads when there was work to do. This was complicated.
The Java language was then changed with the java.nio package which introduced the Selector together with non-blocking IO. This created a reliable (although sometimes confusing) way to service multiple sockets with fewer threads. In your case through, it would not help fully/much because you want to know when a full Object is ready to be read, not when there's just 'some' object.
In the interim the 'landscape' changed, and Java is now able to more efficiently create and manage threads. 'Current' thinking is that it is better/faster and easier to allocate a single thread per socket again.... see Java thread per connection model vs NIO
In your case, I would suggest that you stick with the thread-per-socket model, and you'll be fine. Java can scale and handle more threads than sockets, so you'll be fine.
I'm the main developer of an online game.
Players use a specific client software that connects to the game server with TCP/IP (TCP, not UDP)
At the moment, the architecture of the server is a classic multithreaded server with one thread per connection.
But in peak hours, when there are often 300 or 400 connected people, the server is getting more and more laggy.
I was wondering, if by switching to a java.nio.* asynchronous I/O model with few threads managing many connections, if the performances would be better.
Finding example codes on the web that cover the basics of such a server architecture is very easy. However, after hours of googling, I didn't find the answers to some more advanced questions:
1 - The protocol is text-based, not binary-based. The clients and the server exchanges lines of text encoded in UTF-8. A single line of text represents a single command, each lines are properly terminated by \n or \r\n.
For the classic multithreaded server, I have that kind of code :
public Connection (Socket sock) {
this.in = new BufferedReader( new InputStreamReader( sock.getInputStream(), "UTF-8" ));
this.out = new BufferedWriter( new OutputStreamWriter(sock.getOutputStream(), "UTF-8"));
new Thread(this) .start();
}
And then in run, data are read line by line with readLine.
In the doc, I found an utilitiy class Channels that can create a Reader out of a SocketChannel. But it is said that the produced Reader wont work if the Channel is in non-blocking mode, what contradicts the fact that non-blocking mode is mandatory to use the highly performant channel selection API I'm willing to use. So, I suspect that it isn't the right solution for what I would like to do.
The first question is therefore the following: if I can't use that, how to efficiently and properly take care of breaking lines and converting native java strings from/to UTF-8 encoded data in the nio API, with buffers and channels?
Do I have to play with get/put or inside the wrapped byte array by hand? How to go from ByteBuffer to strings encoded in UTF-8 ? I admit to don't understand very well how to use classes in the charset package and how it works to do that.
2 - In the asynchronous/non-blocking I/O world, what about the handling of consecutive read/write that have by nature to be executed sequencially one after the other?
For example, the login procedure, which is typicly challenge-response-based: the server sends a question (a particular computation), the client sends the response, and then the server checks the response given by the client.
The answer is, I think, certainly not to make a single task to send to worker threads for the whole login process, as it is quite long, with the risk to freeze worker threads for too much time (Imagine that scenario: 10 pool threads, 10 players try to connect at the same time; tasks related to players already online are delayed until one thread is again ready).
3 - What happens if two different threads simultaneously call Channel.write(ByteBuffer) on the same Channel?
Do the client might receive mixed up lines ? For example if a thread sends "aaaaa" and another sends "bbbbb", could the client receive "aaabbbbbaa", or am I ensured that everyting is sent in a consist order? Am I allowed to modify the buffer used right after the call returned?
Or asked differently, do I need additional synchronization to avoid this sort of situation?
If I need additionnal synchronization, how to know when release locks and so on, upon write finishes?
I'm afraid that the answer isn't as simple as registering for OP_WRITE in the selector. By trying that, I noticed that I get the write-ready event all the time and always for all clients, exiting Selector.select early mostly for nothing, since there are only 3 or 4 messages to send pers second per client, while the selection loop is performed hundreds of times per second. So, potentially, active wait in perspective, what is very bad.
4 - Can multiple threads call Selector.select on the same selector simultaneously without any concurrency problems such as missing an event, scheduling it twice, etc?
5 - In fact, is nio as good as it is said to be ? Would it be interesting to stay to classic multithreaded model, but unstead of creating a thread per connection, use fewer threads and loop over the connections to look for data availability using InputStream.isAvailable ? Is that idea stupid and/or inefficient?
1) Yes. I think that you need to write your own nonblocking readLine method. Note also that a nonblocking read may be signaled when there are several lines in the buffer, or when there is an incomplete line:
Example: (first read)
USER foo
PASS
(second read)
bar
You will need to store (see 2) the data that was not consumed, until enough information is ready to process it.
//channel was select for OP_READ
read data from channel
prepend data from previous read
split complete lines
save incomplete line
execute commands
2) You will need to keep the state of each client.
Map<SocketChannel,State> clients = new HashMap<SocketChannel,State>();
when a channel is connected, put a fresh state into the map
clients.put(channel,new State());
Or store the current state as the attached object of the SelectionKey.
Then, when executing each command, update the state. You may write it as a monolithic method, or do something more fancy such as polymorphic implementations of State, where each state knows how to deal with some commands (e.g. LoginState expects USER and PASS, then you change the state into a new AuthorizedState).
3) I don't recall using NIO with many asynchronous writers per channel, but the documentation says it is thread safe (I won't elaborate, since I have no proof of this). About OP_WRITE, note that it signals when the write buffer is not full. In other words, as said here: OP_WRITE is almost always ready, i.e. except when the socket send buffer is full, so you will just cause your Selector.select() method to spin mindlessly.
4) Yes. Selector.select() performs a blocking selection operation.
5) I think that the most difficult part is switching from a thread-per-client architecture, to a different design where reads and writes are decoupled from processing. Once you have done that, it is easier to work with channels than working your own way with blocking streams.
Because of browser compatibility issues, I have decided to use long polling for a real time syncing and notification system. I use Java on the backend and all of the examples I've found thus far have been PHP. They tend to use while loops and a sleep method. How do I replicate this sort of thing in Java? There is a Thread.sleep() method, which leads me to...should I be using a separate thread for each user issuing a poll? If I don't use a separate thread, will the polling requests be blocking up the server?
[Update]
First of all, yes it is certainly possible to do a straightforward, long polling request handler. The request comes in to the server, then in your handler you loop or block until the information you need is available, then you end the loop and provide the information. Just realize that for each long polling client, yes you will be tying up a thread. This may be fine and perhaps this is the way you should start. However - if your web server is becoming so popular that the sheer number of blocking threads is becoming a performance problem, consider an asynchronous solution where you can keep a large numbers of client requests pending - their request is blocking, that is not responding until there is useful data, without tying up one or more threads per client.
[original]
The servlet 3.0 spec provides a standard for doing this kind asynchronous processing. Google "servlet 3.0 async". Tomcat 7 supports this. I'm guessing Jetty does also, but I have not used it.
Basically in your servlet request handler, when you realize you need to do some "long" polling, you can call a method to create an asynchronous context. Then you can exit the request handler and your thread is freed up, however the client is still blocking on the request. There is no need for any sleep or wait.
The trick is storing the async context somewhere "convenient". Then something happens in your app and you want to push data to the client, you go find that context, get the response object from it, write your content and invoke complete. The response is sent back to the client without you having to tie up a thread for each client.
Not sure this is the best solution for what you want but usually if you want to do this at period intervals in java you use the ScheduleExecutorService. There is a good example at the top of the API document. The TimeUnit is a great enum as you can specify the period time easily and clearly. So you can specify it to run every x minutes, hours etc
Is there a way to immediately print the message received from the client without using an infinite loop to check whether the input stream is empty or not?
Because I found that using infinite loop consumes a lot of system resources, which makes the program running so slow. And we also have to do the same (infinite loop) on the client side to print the message on the screen in real time.
I'm using Java.
You should be dealing with the input stream in a separate Thread - and let it block waiting for input. It will not use any resources while it blocks. If you're seeing excessive resource usage while doing this sort of thing, you're doing it wrong.
I think you can just put your loop in a different thread and have it sleep a bit (maybe for half a second?) between iterations. It would still be an infinite loop, but it would not consume nearly as many resources.
You don't you change your architecture a little bit to accommodate WebSockets. check out Socket.IO . It is a cross browser WebSockets enabler.
You will have to write controllers (servlets for example in java) that push data to the client. This does not follow the request-response architecture.
You can also architect it so that a "push servlet" triggers a "request" from the client to obtain the "response".
Since your question talks about Java, and if you are interested in WebSockets, check this link out.
If you're using Sockets, which you should be for any networking.
Then you can use the socket's DataInputStream which you can get using socket.getInputStream() (i think that's the right method) and do the following:
public DataInputStream streamIn;
public Socket soc;
// initialize socket, etc...
streamIn = soc.getInputStream();
public String getInput() {
return (String) streamIn.readUTF(); // Do some other casting if this doesn't work
}
streamIn.readUTF() blocks until data is available, meaning you don't have to loop, and threading will let you do other processing while you wait for data.
Look here for more information on DataInputStream and what you can do with it: http://docs.oracle.com/javase/6/docs/api/java/io/DataInputStream.html
A method that does not require threads would involve subclassing the input stream and adding a notify type method. When called this method would alert any interested objects (i.e. objects that would have to change state due to the additions to the stream) that changes have been made. These interested objects could then respond in anyway that is desired.
Objects writing to the buffer would do their normal writing, and afterward would call the notify() method on the input stream, informing all interested objects of the change.
Edit: This might require subclassing more than a couple of classes and so could involve a lot of code changes. Without knowing more about your design you would have to decide if the implementation is worth the effort.
There are two approaches that avoid busy loops / sleeps.
Use a thread for each client connection, and simply have each thread call read. This blocks the thread until the client sends some data, but that's no problem because it doesn't block the threads handling other clients.
Use Java NIO channel selectors. These allow a thread to wait until one of set of channels (in this case sockets) has data to be read. There is a section of the Oracle Java Tutorials on this.
Of these two approaches, the second one is most efficient in terms of overall resource usage. (The thread-per-client approach uses a lot of memory on thread stacks, and CPU on thread switching overheads.)
Busy loops that repeatedly call (say) InputStream.available() to see if there is any input are horribly inefficient. You can make them less inefficient by slowing down the polling with Thread.sleep(...) calls, but this has the side effect of making the service less responsive. For instance, if you add a 1 second sleep between each set of polls, the effect that each client will see is that the server typically delays 1 second before processing each request. Assuming that those requests are keystrokes and the responses echo them, the net result is a horribly laggy service.