A little help please.
I am designing a stateless server that will have the following functionality:
Client submits a job to the server.
Client is blocked while the server tries to perform the job.
The server will spawn one or multiple threads to perform the job.
The job either finishes, times out or fails.
The appropriate response (based on the outcome) is created, the client is unblocked and the response is handed off to the client.
Here is what I have thought of so far.
Client submits a job to the server.
The server assigns an ID to the job, places the job on a Queue and then places the Client on an another queue (where it will be blocked).
Have a thread pool that will execute the job, fetch the result and appropriately create the response.
Based on ID, pick the client out of the queue (thereby unblocking it), give it the response and send it off.
Steps 1,3,4 seems quite straight forward however any ideas about how to put the client in a queue and then block it. Also, any pointers that would help me design this puppy would be appreciated.
Cheers
Why do you need to block the client? Seems like it would be easier to return (almost) immediately (after performing initial validation, if any) and give client a unique ID for a given job. Client would then be able to either poll using said ID or, perhaps, provide a callback.
Blocking means you're holding on to a socket which obviously limits the upper number of clients you can serve simultaneously. If that's not a concern for your scenario and you absolutely need to block (perhaps you have no control over client code and can't make them poll?), there's little sense in spawning threads to perform the job unless you can actually separate it into parallel tasks. The only "queue" in that case would be the one held by common thread pool. The workflow would basically be:
Create a thread pool (such as ThreadPoolExecutor)
For each client request:
If you have any parts of the job that you can execute in parallel, delegate them to the pool.
And / or do them in the current thread.
Wait until pooled job parts complete (if applicable).
Return results to client.
Shutdown the thread pool.
No IDs are needed per se; though you may need to use some sort of latch for 2.1 / 2.3 above.
Timeouts may be a tad tricky. If you need them to be more or less precise you'll have to keep your main thread (the one that received client request) free from work and have it signal submitted job parts (by flipping a flag) when timeout is reached and return immediately. You'll have to check said flag periodically and terminate your execution once it's flipped; pool will then reclaim the thread.
How are you communicating to the client?
I recommend you create an object to represent each job which holds job parameters and the socket (or other communication mechanism) to reach the client. The thread pool will then send the response to unblock the client at the end of job processing.
The timeouts will be somewhat tricky, and will have hidden gotcha's but the basic design would seem to be to straightforward, write a class that takes a Socket in the constructor. on socket.accept we just do a new socket processing instantiation, with great foresight and planning on scalability or if this is a bench-test-experiment, then the socket processing class just goes to the data processing stuff and when it returns you have some sort of boolean or numeric for the state or something, handy place for null btw, and ether writes the success to the Output Stream from the socket or informs client of a timeout or whatever your business needs are
If you have to have a scalable, effective design for long-running heavy-haulers, go directly to nio ... hand coded one-off solutions like I describe probably won't scale well but would provide fundamental conceptualizing basis for an nio design of code-correct work.
( sorry folks, I think directly in code - design patterns are then applied to the code after it is working. What does not hold up gets reworked then, not before )
Related
I'm still pretty new to netty so please bare with me. There seems to be plenty of questions asking why a specefic netty implementation is slow and how to make it faster. But my use case is a bit different. I want to avoid low level socket implementations (hence netty) but I also know that blocking the event group is bad. I know I can dynamically manage the pipeline. I'm not sure I know enough about netty to know if this is possible, and I've not tried much that I don't already know is bad (thread.sleep for example). The protocol is HTTP but I also need it to be useful for other protocols.
But what I don't know is, for a single connection on a shared port, how to slow down the response of the server to the client, and vice versa? Or put more aptly: where, and what, would I implement the slowness required? My guess is the encoder for the where; but because of netty's approach, i haven't the foggiest for the what.
You say that you know that Thread.sleep is "bad" but it really depends on what you're trying to achieve and where you put the sleep. I believe that the best way to build this would be to use a DefaultEventExecutorGroup to offload the processing of your slow-down ChannelHandler onto non-event-loop threads and then call Thread.sleep in your handler.
From the ChannelPipeline javadoc, under the "Building a pipeline" section:
https://netty.io/4.1/api/io/netty/channel/ChannelPipeline.html
A user is supposed to have one or more ChannelHandlers in a pipeline to receive I/O events (e.g. read) and to request I/O operations (e.g. write and close). For example, a typical server will have the following handlers in each channel's pipeline, but your mileage may vary depending on the complexity and characteristics of the protocol and business logic:
Protocol Decoder - translates binary data (e.g. ByteBuf) into a Java object.
Protocol Encoder - translates a Java object into binary data.
Business Logic Handler - performs the actual business logic (e.g. database access).
and it could be represented as shown in the following example:
static final EventExecutorGroup group = new DefaultEventExecutorGroup(16);
...
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast("decoder", new MyProtocolDecoder());
pipeline.addLast("encoder", new MyProtocolEncoder());
// Tell the pipeline to run MyBusinessLogicHandler's event handler methods
// in a different thread than an I/O thread so that the I/O thread is not blocked by
// a time-consuming task.
// If your business logic is fully asynchronous or finished very quickly, you don't
// need to specify a group.
pipeline.addLast(group, "handler", new MyBusinessLogicHandler());
Be aware that while using DefaultEventLoopGroup will offload the operation from the EventLoop it will still process tasks in a serial fashion per ChannelHandlerContext and so guarantee ordering. Due the ordering it may still become a bottle-neck. If ordering is not a requirement for your use-case you may want to consider using UnorderedThreadPoolEventExecutor to maximize the parallelism of the task execution.
I hope someone can post a better (more explanative) answer than this but basically all that's needed is to use a ChannelTrafficShapingHandler with some small enough values.
For instance, a 2kb response with read and write limit of 512b, maxTime of 6000ms, and a checkInterval of 1000ms forces the response to take 4000ms with the ChannelTrafficShapingHandler, and 50ms without it when running both client and server locally. I expect those times to increase dramatically when on the network wire.
final ChannelTrafficShapingHandler channelTrafficShapingHandler = new ChannelTrafficShapingHandler(
getRateInBytesPerSecond(), getRateInBytesPerSecond(), getCheckInterval(), getMaxTime());
ch.addLast(channelTrafficShapingHandler);
I am developing a webserver in java that will provide websocket communication to its' clients. I have been proposed to use a thread pool when dealing with many clients because it is a lot more time efficient than to use one thread per client.
My question is simply, will Javas ExecutorService, newFixedThreadPool be able to handle a queue of runnable tasks with thread blocking methods being called inside of them?
In other words i guess i am wondering if this thread pool is asynchronous?
The reason i am asking is that i have tried using a newFixedThreadPool with, lets say, 2 threads. Then when i connect 3 clients to the server, i can only receive commands from the first two. But i guess i could be doing something wrong, thats why i am asking.
The runnable tasks are also in an infinite while loop (only ends when client disconnects).
Well, it depends on your implementation. The easiest case is having clients keeping their thread active until the disconnect (or get kicked out because of a timeout). In this case, your thread pool isn't very efficient. I'll only re-use disconnected users' threads instead of creating new one (which is good, but not really relevant).
The second case would be activating the threads only when needed (let's say when a client sends or receives a messages). In this case, you need to remember the server-side (keeping an id for example), in order to be able to sever the thread connection when they don't need them, and re-establish it when they do. In order to do that, you must keep the sockets somewhere, but unbound to any specific thread.
I actually didn't code that myself but I don't see why it would work as this is the mechanism used for websites (i.e. HTTP protocol)
I am building a server in java that communicates with several clients at the same time, the initial approach we had is the the server listens to connections from the clients, once a connection is received and a socket is created, a new thread is spawned to handle the communication with each client, that is read the request with an ObjectInputStream, do the desired operation (fetch data from the DB, update it, etc.), and send back a response to the client (if needed). While the server itself goes back to listen to more connections.
This works fine for the time being, however this approach is not really scalable, it works great for a small amount of clients connected at the same time, however since every client spawns another thread, what will happen when there are a too many clients connected at once?
So my next idea was to maintain a list of sorts that will hold all connected clients (the socket object and some extra info), use a ThreadPool for to iterate through them and read anything they sent, if a message was received then put it in a queue for execution by another ThreadPool of worker threads, and once the worker has finished with its task if a response is required then send it.
The 2 latter steps are pretty trivial to implement, the problem is that with the original thread per client implementation, I use ObjectInputStream.readObject() to read the message, and this method blocks until there is something to read, which is fine for this approach, but I can't use the same thing for the new approach, since if I block on every socket, I will never get to the ones that are further down the list.
So I need a way to check if I have anything to read before I call readObject(), so far I tried the following solutions:
Solution 1:
use ObjectInputStream.available() to check if there is anything available to read, this approach failed since this method seems to always return 0, regardless of whether there is an object in the stream or not. So this does not help at all.
Solution 2:
Use PushbackInputStream to check for the existence of the first unread byte in the stream, if it exists then push it back and read the object using the ObjectInputStream, and if it doesn't move on:
boolean available;
int b = pushbackinput.read();
if (b==-1)
available = false;
else
{
pushbackinput.unread(b);
available = true;
}
if (available)
{
Object message= objectinput.readObject();
// continue with what you need to do with that object
}
This turned out to be useless too, since read() blocks also if there is no input to read. It seems to only return the -1 option if the stream was closed. If the stream is still open but empty it just blocks, so this is no different than simply using ObjectInputStream.readObject();
Can anyone suggest an approach that will actually work?
This is a good question, and you've done some homework.... but it involves going through some history to get things right. Note, your issue is actually more to do with the socket-level communication rather than the ObjectInputStream:
The easiest way to do things in the past was to have a separate thread per socket. This was scalable to a point but threads were expensive and slow to create.
In response, for large systems, people created thread pools and would service the sockets on threads when there was work to do. This was complicated.
The Java language was then changed with the java.nio package which introduced the Selector together with non-blocking IO. This created a reliable (although sometimes confusing) way to service multiple sockets with fewer threads. In your case through, it would not help fully/much because you want to know when a full Object is ready to be read, not when there's just 'some' object.
In the interim the 'landscape' changed, and Java is now able to more efficiently create and manage threads. 'Current' thinking is that it is better/faster and easier to allocate a single thread per socket again.... see Java thread per connection model vs NIO
In your case, I would suggest that you stick with the thread-per-socket model, and you'll be fine. Java can scale and handle more threads than sockets, so you'll be fine.
I'm developing a small client-server program in Java.
The client and the server are connected over one tcp-connection. Most parts of the communication are asynchronous (can happen at any time) but some parts I want to be synchronous (like ACKs for a sent command).
I use a Thread that reads commands from the socket's InputStream and raises an onCommand() event. The Command itself is progressed by the Command-Design-Pattern.
What would be a best-practice approach (Java), to enable waiting for an ACK without missing other, commands that could appear at the same time?
con.sendPacket(new Packet("ABC"));
// wait for ABC_ACK
edit1
Think of it like an FTP-Connection but that both data and control-commands are on the same connection. I want to catch the response to a control-command, while data-flow in the background is running.
edit2
Everything is sent in blocks to enable multiple (different) transmissons over the same TCP-Connection (multiplexing)
Block:
1 byte - block's type
2 byte - block's payload length
n byte - block's paylod
In principle, you need a registry of blocked threads (or better, the locks on which they are waiting), keyed with some identifier which will be sent by the remote side.
For asynchronous operation, you simply sent the message and proceed.
For synchronous operation, after sending the message, your sending thread (or the thread which initiated this) create a lock object, adds this with some key to the registry and then waits on the lock until notified.
The reading thread, when it receives some answer, looks in the registry for the lock object, adds the answer to it, and calls notify(). Then it goes reading the next input.
The hard work here is the proper synchronization to avoid dead locks as well as missing a notification (because it comes back before we added ourself to the registry).
I did something like this when I implemented the remote method calling protocol for our Fencing-applet. In principle RMI works the same way, just without the asynchronous messages.
#Paulo's solution is one I have used before. However, there may be a simpler solution.
Say you don't have a background thread reading results in the connection. What you can do instead do is use the current thread to read any results.
// Asynchronous call
conn.sendMessage("Async-request");
// server sends no reply.
// Synchronous call.
conn.sendMessage("Sync-request");
String reply = conn.readMessage();
I have a java server that handles logins from multiple clients. The server creates a thread for each tcp/ip socket listener. Database access is handled by another thread that the server creates.
At the moment the number of clients I have attaching to the server is quite low (<100) so I have no real performance worries, but I am working out how I should handle more clients in the future. My concern is that with lots of clients my server and database threads will get bogged down by constant calls to their methods from the client threads.
Specifically in relation to the database: At the moment each client thread accesses the public database thread on its server parent and executes a data access method. What I think I should do is have some kind of message queue that a client thread can put its data request on and the database thread will do it when it gets round to it. If there is data to be returned from the data access call then it can put it on a queue for the client thread to pick up. All of this wouldn't hit the main server code or any other client threads.
I therefore think that I want to implement an asynchronous message queue that client threads can put a message on and the database thread will pick up from. Is that the right approach? Any thoughts and links to somewhere I can read up about implementation would be appreciated.
I would not recommend this approach.
JMS was born for this sort of thing. It'll be better than any implementation you'll write from scratch. I'd recommend using a Java EE app server that has JMS built in or something like ActiveMQ or RabbitMQ that you can add to a servlet engine like Tomcat.
I would strongly encourage you to investigate these before writing your own.
What you are describing sounds like an ExecutorCompletionService. This is essentially an asynch task broker that accepts requests (Runnables or Callables) from one thread, returning a "handle" to the forthcoming result in the form of a Future. The request is then executed in a thread pool (which could be a single thread thread pool) and the result of the request is then delivered back to the calling thread through the Future.
In between the time that the request is submitted and response is supplied, your client thread will simply wait on the Future (with an optional timeout).
I would advise, however, that if you're expecting a big increase in the number of clients (and therefore client threads), you should evaluate some of the Java NIO Server frameworks out there. This will allow you to avoid allocating one thread per client, especially since you expect all these threads to spend some time waiting on DB requests. If this is the case, I would suggest looking at MINA or Netty.
Cheers.
//Nicholas
It sounds like what you want to do is limit the number of concurrent requests to the database you want to allow. (To stop it being overloaded)
I suggest you have a limited size connection pool. When too many threads want to use the database they will have to wait until a connection is free. A simple way to do this is with a BlockingQueue with all the connections created in advance.
private final BlockingQueue<Connection> connections = new ArrayBlockingQueue<Connection>(40); {
// create connections
}
// to perform a query.
Connection conn = connections.get();
try {
// do something
} finally {
connections.add(conn);
}
This way you can keep your thread design much the same as it is and limit the number of concurrent queries to the database. With some tweaking you can create the connections as needed and provide a time out if a database connection cannot be obtained quickly.