I have a list of client sockets, usually of size around 2000. These clients are dynamic, they come and go.
I have an ExecutorService with a fixed thread pool of 32 threads handling these threads. This executor service is responsible for decoding and sending the messages to be sent to these 2000 clients.
I want to prevent that two (or more) threads of the executor service are processing the same client at the same time.
One approach could be to introduce another bookkeeping thread (so I end up with 32 + 1 threads) which is responsible for calling ExecutorService.submit(mesage) when the previous message corresponding to the same client is done. But I am not sure if this will introduce a bottleneck, meaning that this newly introduced bookkeeping thread cannot keep up submitting messages.
Ideally, I don't want to pre-allocate a thread to a set of clients in advance, as the message load is not evenly distributed between the clients. It is also not known in advance.
What are approaches for this? Are they offered by java.util.concurrent functionalities?
update
This is a quick summary, as the comments pointed out that there were some misunderstandings:
I don't want a single thread per client, as I would end up with 2000 threads.
Ideally, I don't want to pre-allocate a thread to a set of clients, because message rate is not evenly distributed between all clients and not known in advance.
Message order must be preserved.
I believe it would not be good that thread A is waiting for thread B because B is already sending a message to the same client. In other words, at all times only one thread is processing one client.
When a thread (A) begins processing a message (#1), it needs to register the client id with a shared manager object. For each registered client, there is a queue.
When another thread (B) begins processing a message (#2) for the same client, the registration will detect that thread A is already processing, and will add message #2 to the queue for client. Thread B will then stop and process the next message.
When thread A is done with message #1, it will try to unregister, but since message #2 is queue, thread A will instead begin processing that message. After that, when it tries to unregister again, there are no queued messages and the thread will stop and process the next message.
It is up to the manager code to correctly synchronize access, so a second message is either processed by thread B, or handed off to thread A, without getting lost.
The above logic ensures that thread B will not wait for thread A, i.e. no idle time, and that message #2 is processed as soon as possible, i.e. with minimal delay, without processing two messages for the same client as the same time.
Message order for each client is retained. Globally, message order is of course not retained, because the processing of message #2 is delayed.
Note, there will be only one queue for each thread, so only 32 queues, and only "duplicate" messages are queue, so all queue will usually remain empty.
UPDATE
Example: For identification here, messages are named clientId.messageId where messageId is global.
Messages are submitted to the Executor (3 threads) in this order:
1.1, 2.2, 1.3, 2.4, 3.5, 1.6
Thread A picks up 1.1 and starts processing.
Thread B picks up 2.2 and starts processing.
Thread C picks up 1.3, adds it to thread A's queue, then returns.
Thread C picks up 2.4, adds it to thread B's queue, then returns.
Thread C picks up 3.5 and starts processing.
Thread A is done with message 1.1 and starts processing 1.3.
Thread C is done with message 3.5 and returns.
Thread C picks up 1.6, adds it to thread A's queue, then returns.
Thread C is now idle.
Thread B is done with message 2.2 and starts processing 2.4.
Thread A is done with message 1.3 and starts processing 1.6.
Thread B is done with message 2.4 and returns.
Thread B is now idle.
Thread A is done with message 1.6 and returns.
Thread A is now idle.
Have each thread service it's own queue. Number the sockets. Put each request on queue[socket num % num of threads].
This will ensures requests from a particular socket are handled in series and in sequence.
Unfortunately you won't get load balancing this way.
Alternatively, use a concurrenthashmap to store the sockets being served. If a thread services a request of a socket currently being processed, just put the request back in the queue.
You want to process messages for each client sequentially and at the same time you do not want to allocate separate thread for each client. This is the exact use case to employ Actor model. Actors are like lightweight threads. They are not as powerful as the usual threads but fit ideally for repeatable tasks like yours.
If you find java actor libraries found by Google too heavyweight, you can use the most compact actor implementation at my Github repository, or look at the extended actor implementation included in my asynchronous library df4j.
Related
I want to build a process flow. I have multiple requests to process in a queue. One thread (call it T1) takes the first request start processing it and then passes it to some other thread (from a pool of threads lets call it T2) at a point when it has to do some blocking database access. T1 should get free now to process another request from queue. The blocking database access is done by a thread from T2 pool. Then after the database operation is completed, thread from T2 passes it to a thread T3 which return the processed result of the request and after that gets free to return another result processed by T2.
I want to do this to avoid one thread for one request model as it would bring a lot of context switching overheads and all threads will eventually block on database access and CPU resources will be wasted at that time. The T1 and T3 threads can be considered as thread pools of size of limited size depending on the cores in the CPU.
I thought about above approach after getting to know about async servelet as after getting the request it does not block the thread and instead a different thread does the job and returns the response later.
Let me know if the process flow I need to build is feasible in java and some resources on how can it be achieved.
Splitting a single request seems like a nice idea, but to optimize use of resources, I would look for using semaphores and have each request be handled by a different thread.
Try limiting the number of requests with semaphores, and limit the access to resources that can be accessed only one at a time also with semaphores.
Splitting a single request can be a good idea, but I think it is a good idea mostly in the case scenario that you want to save the data inside some files, to lower the memory usage of the threads.
I will try adding some Java code later if I can find my old projects...
Im new to Java and have been stuck on an issue with respect to thread message passing.
What i mean here is- I have 4 threads, one thread reads msg from network and based on type of msg passes on the msg to either parser thread or database thread . Database thread performs some operation and has to send msg back to the first network thread which puts it into socket. Similarly, the parser thread also performs some action and based on result either has to send msg back to network thread or database thread.
Things i have tried-
I have read about notify() wait() for thread communication which does not help in my case as i need one to one msg passing its not braodcast all
I have read about concurrentqueues blockingqueues - Since this is not an ideal producer consumer problem where one thread is producing msgs and other threads reading from it- i cannot use this.
Using this would be like i need to have 5 queues for each communication channel
network->db,
db->network,
parser->network,
parser->db
Is this efficient to go about?
In c++ i was using msging mechanism where i used to just post msg(windows msg) to corresponding thread's msg pool and that thread in its msging pool, would fetch it
Is there any mechanism like message passing in java which i could use?
one thread reads msg from network and based on type of msg passes on the msg to ...database thread. Database thread performs some operation and has to send msg back to the first network thread which puts it into socket.
You're making the "network" thread responsible to wait for messages from the network, and also, to wait for messages from the "database" thread. That's awkward. You may find it somewhere between mildly difficut and impossible to make that happen in a clean, satisfying way.
My personal opinion is that each long-lived thread in a multi-threaded program should wait for only one thing.
What is the reason for having the database thread "send msg back to the first network thread [to be put] into socket?" Why can't the database thread itself put the message into the socket?
If there's a good reason for the database not to send out the message, then why can't "put the message into the socket" be a task that your database thread submits to a thread pool?
I have read about notify() wait() for thread communication which does not help in my case
Would a BlockingQueue help?
I have read about concurrentqueues blockingqueues - Since this is not an ideal producer consumer problem where one thread is producing msgs and other threads reading from it- i cannot use this. Using this would be like i need to have 5 queues for each communication channel.
And? If adding more queues or more threads to a program makes the work that those threads do simpler or makes the explanation of what those queues are for easier to understand, would that be a Bad Thing?
Note about wait() and notify(). Those are low-level methods that are meant to be used in a very specific way to build higher-level mechanisms. I don't know whether the standard Java BlockingQueue implementations actually does use wait() and notify() but it would not be hard to implement a BlockingQueue that actually did use that mechanism. So, if BlockingQueue solves your problem, then that means wait() and notify() solve your problem. You just didn't see the solution.
In fact, I would be willing to bet that wait() and notify() can be used to solve any problem that requires one thread to wait for another. It's just a matter of seeing what else you need to build around them.
Should I accept connections and monitoring clients on a listener thread and then let workers handle the request and answer to the client, or should I do everything on one thread?
Neither.
Ideally, for an NIO-based server, you create a thread pool using something like Executors.newFixedThreadPool(), which you will use to perform all the processing for handling your requests.
But, there should be no assignment of requests to specific threads, because the rest of your system should be asynchronous as well. That means that when a request handler needs to perform some lengthy I/O work or similar, instead of blocking the thread and waiting for it to finish, it starts it asynchronously and arranges for processing to continue when the work is finished by submitting a new task to the thread pool. There's no telling which thread will pick up the work at that point, so the processing for a request could end up being spread across many threads.
You should usually coordinate your asynchronous processing using CompletableFuture the same way that Promise is used in node. Have a look at my answer over here, which tries to explain how to do that: decoupled design for async http request
If your request handling is 100% asynchronous, that is you never wait for anything during request handling and you're on a single-core system, then it might be slightly better to do everything in the same thread.
If you have a multi-core system or you wait on I/O during request processing, then you should use a thread pool instead.
I want to know what happens when we receive ACK. Do we receive ACKs in a single thread or in many threads?
Do handleAck and handleNack methods are used by a single thread or many threads? If they are used by a single thread then it is OK. But if they are used by several threads then we have to construct our code in the thread safe manner.
You shouldn't need to write your ConfirmListener code thread-safe but not because the ack and nack methods won't be called from multiple threads but because you shouldn't share a Channel between threads to begin with.
The documentation specifically calls this out:
While some operations on channels are safe to invoke concurrently,
some are not and will result in incorrect frame interleaving on the
wire. Sharing channels between threads will also interfere with *
Publisher Confirms.
When you are publishing to the broker just don't share the Channel. Channels are lightweight and not that expensive to create. That way you don't need to worry about the confirms either.
If you do share the Channel your confirms will be interfered with as per the above quote.
I know that threads have a message queue and handlers are able to push runnables or messages to them, but when I profile my android application using Android Studio tools, there is a strange process:
android.os.MessageQueue.nativePollOnce
It uses the CPU more than all the other processes. What is it and how can I reduce the time that the CPU spends on it?
You can find the profiler result below.
Short answer:
The nativePollOnce method is used to "wait" till the next Message becomes available. If the time spent during this call is long, your main (UI) thread has no real work to do and waits for next events to process. There's no need to worry about that.
Explanation:
Because the "main" thread is responsible for drawing UI and handling various events, it's Runnable has a loop which processes all these events.
The loop is managed by a Looper and its job is quite straightforward: it processes all Messages in the MessageQueue.
A Message is added to the queue for example in response to input events, as frame rendering callback or even your own Handler.post calls. Sometimes the main thread has no work to do (that is, no messages in the queue), which may happen e.g. just after finishing rendering single frame (the thread has just drawn one frame and is ready for the next one, just waits for a proper time). Two Java methods in the MessageQueue class are interesting to us: Message next() and boolean enqueueMessage(Message, long). Message next(), as its name suggest, takes and returns the next Message from the queue. If the queue is empty (and there's nothing to return), the method calls native void nativePollOnce(long, int) which blocks until a new message is added. At this point you might ask how does nativePollOnce know when to wake up. That's a very good question. When a Message is added to the queue, the framework calls the enqueueMessage method, which not only inserts the message into the queue, but also calls native static void nativeWake(long), if there's need to wake up the queue. The core magic of nativePollOnce and nativeWake happens in the native (actually, C++) code. Native MessageQueue utilizes a Linux system call named epoll, which allows to monitor a file descriptor for IO events. nativePollOnce calls epoll_wait on a certain file descriptor, whereas nativeWake writes to the descriptor, which is one of the IO operations, epoll_wait waits for. The kernel then takes out the epoll-waiting thread from the waiting state and the thread proceeds with handling the new message. If you're familiar with Java's Object.wait() and Object.notify() methods, you can imagine that nativePollOnce is a rough equivalent for Object.wait() and nativeWake for Object.notify(), except they're implemented completely differently: nativePollOnce uses epoll and Object.wait() uses futex Linux call. It's worth noticing that neither nativePollOnce nor Object.wait() waste CPU cycles, as when a thread enters either method, it becomes disabled for thread scheduling purposes (quoting the javadoc for the Object class). However, some profilers may mistakenly recognize epoll-waiting (or even Object-waiting) threads as running and consuming CPU time, which is incorrect. If those methods actually wasted CPU cycles, all idle apps would use 100% of the CPU, heating and slowing down the device.
Conclusion:
You shouldn't worry about nativePollOnce. It just indicates that processing of all Messages has been finished and the thread waits for the next one. Well, that simply means you don't give too much work to your main thread ;)