a jk_connector worker is basically a tomcat instance waiting to process requests from a web server.
The apache docs tell you that you should have multiple workers if you have multiple apps, but doesnt really explain why.
What are the pros/cons of having a worker per web app vs 1 worker for multiple apps?
Processor affinity for one. If the workset is bound to one executional unit its built in cache be utilized more effectively. The more applications to share the space the more contention.
Most systems today are based on multiple cpu cores where threads can execute independently on each core. This means that a busy server can better utilize system resources if there are more threads (e.g., 1 thread/cpu), both for multicore (SMP) and multithreading (SMT) systems. A common way for servers is to provide a process/thread pool of workers which can be used and reused to serve multiple simultaneous requests.
Related
Assume we have a computer with four physical cores, and we want to decrease latency of a task and best number of threads to do that is 4.
But if we are in a web application, and we use an application server(or servlet container) like tomcat, jetty, netty, I think for doing throughput performance issue, that application server uses 4 threads.
At this point, if we want to use 4 threads to decrease latency of a task, Given that 4 threads is used by application server. With using multi threading, we can not get the most or great benefits. Is that true for web applications?
Thank you so much in advance.
When a single user is accessing an application, multiple threads can be used, and they can run parallel if multiple cores are present. If only one processor exists, then threads will run one after another.
When multiple users are accessing an application, how are the threads handled?
I can talk from Java perspective, so your question is "when multiple users are accessing an application, how are the threads handled?".
The answer is it all depends on how you programmed it, if you are using some web/app container they provide thread pool mechanism where you can have more than one threads to server user reuqests, Per user there is one request initiated and which in turn is handled by one thread, so if there are 10 simultaneous users there will be 10 threads to handle the 10 requests simultaneously, now we do have Non-blocking IO now a days where the request processing can be off loaded to other threads so allowing less than 10 threads to handle 10 users.
Now if you want to know how exactly thread scheduling done around CPU core, it again depends on the OS. One thing common though 'thread is the basic unit of allocation to a CPU'. Start with green threads here, and you will understand it better.
The incorrect assuption is
If only one processor exists, then threads will run one after another.
How threads are being executed is up to the runtime environment.
With java there are some definitions that certain parts of your code will not be causing synchronisation with other threads and thus will not cause (potential) rescheduling of threads.
In general, the OS will be in charge of scheduling units-of-execution. In former days mostly such entities have been processes. Now there may by processes and threads (some do scheduling only at thread level). For simplicity let ssume OS is dealing with threads only.
The OS then may allow a thread to run until it reaches a point where it can't continue, e.g. waiting for an I/O operation to cpmplete. This is good for the thread as it can use CPU for max. This is bad for all the other threads that want to get some CPU cycles on their own. (In general there always will be more threads than available CPUs.So, the problem is independent of number of CPUs.) To improve interactive behaviour an OS might use time slices that allow a thread to run for a certain time. After the time slice is expired the thread is forcible removed from the CPU and the OS selects a new thread for being run (could even be the one just interrupted).
This will allow each thread to make some progress (adding some overhead for scheduling). This way, even on a single processor system, threads my (seem) to run in parallel.
So for the OS it is not at all important whether a set of thread is resulting from a single user (or even from a single call to a web application) or has been created by a number of users and web calls.
You need understand about thread scheduler.
In fact, in a single core, CPU divides its time among multiple threads (the process is not exactly sequential). In a multiple core, two (or more) threads can run simultaneously.
Read thread article in wikipedia.
I recommend Tanenbaum's OS book.
Tomcat uses Java multi-threading support to serve http requests.
To serve an http request tomcat starts a thread from the thread pool. Pool is maintained for efficiency as creation of thread is expensive.
Refer to java documentation about concurrency to read more https://docs.oracle.com/javase/tutorial/essential/concurrency/
Please see tomcat thread pool configuration for more information https://tomcat.apache.org/tomcat-8.0-doc/config/executor.html
There are two points to answer to your question : Thread Scheduling & Thread Communication
Thread Scheduling implementation is specific to Operating System. Programmer does not have any control in this regard except setting priority for a Thread.
Thread Communication is driven by program/programmer.
Assume that you have multiple processors and multiple threads. Multiple threads can run in parallel with multiple processors. But how the data is shared and accessed is specific to program.
You can run your threads in parallel Or you can wait for threads to complete the execution before proceeding further (join, invokeAll, CountDownLatch etc.). Programmer has full control over Thread life cycle management.
There is no difference if you have one user or several. Threads work depending the logic of your program. The processor runs every thread for a certain ammount of time and then follows to the next one. The time is very short, so if there are not too much threads (or different processes) working, the user won't notice it. If the processor uses a 20 ms unit, and there are 1000 threads, then every thread will have to wait for two seconds for its next turn. Fortunately, current processors, even with just one core, have two process units which can be used for parallel threads.
In "classic" implementations, all web requests arriving to the same port are first serviced by the same single thread. However as soon as request is received (Socket.accept returns), almost all servers would immediately fork or reuse another thread to complete the request. Some specialized single user servers and also some advanced next generation servers like Netty may not.
The simple (and common) approach would be to pick or reuse a new thread for the whole duration of the single web request (GET, POST, etc). After the request has been served, the thread likely will be reused for another request that may belong to the same or different user.
However it is fully possible to write the custom code for the server that binds and then reuses particular thread to the web request of the logged in user, or IP address. This may be difficult to scale. I think standard simple servers like Tomcat typically do not do this.
We are running same Jetty service on two servers but are seeing different number of threads created by both services (50 vs ~100 threads).
Both servers are running identical Java code on RedHat5 (they do have slightly different kernels). Yet Jetty on one of the servers creates more threads than the other one. How is it possible?
Thread counts are dynamic, depends on many many factors.
The number of threads that you see at any one point can vary greatly, based on hardware differences (number of cpu cores, number of network interfaces, etc), kernel differences, java differences, load differences, active user counts, active connection counts, transactions per second, if there are external dependencies (like databases), how async processing is done, how async I/O is done, use of http/2 vs http/1, use of websocket, and even ${jetty.base} configuration differences.
As for the counts you are seeing, 50 vs 100, that's positively tiny for a production server. Many production servers on moderately busy systems can use 500 (java) threads, and on very busy commodity systems its can be in the 5,000+ range. Even on specialized hardware (like an Azul systems devices) its not unheard of to be in the 90,000+ thread range with multiple active network interfaces.
Goal
I want to understand how to handle two thread pools simultaneously in java?
Consider a client server system in which clients are sending blocking I/O requests to the server (for example a file server ). There is a single ThreadPoolExecutor instance running on the server. Some types of client’s requests take much longer to process than other requests. These requests are called high I/O intensity requests. These high I/O intensity requests hog all threads and bring down entire application.
I want to solve this problem by two separate ThreadPoolExecutor.
I create two ThreadPoolExecutor instances ,one for high I/o intensity requests and another for low I/o intensity requests, and through offline workload procedure I create a lookup table to classify requests and when a request arrive I first search its class in the lookup table so that I can handover it to its corresponding thread pool.
Real Problem.
How to share processors equally to these two thread pools. Will this task be handled by JVM itself or I have to handle it by myself on application level ?
Should I make use of cluster and use another machine that run an instance of ThreadPoolExecutor to handle high I/O intensity requests?
Kindly give me proper design suggestions.
Generally is up to the system CPU scheduler how to distribute time between threads. Thread pool has nothing to do with thread scheduling. It can manage some threads reusing or synchronization between them.
The only advantage of creating 2 pools instead of 1 is that one pool can use ThreadFactory different than standard Executors.defaultThreadFactory(). You can give different priority for your demanding clients. Prority is a information that scheBut they would suffer even more then if you make them less important or vice versa ;)
Maybe you could rather do something like tuning it's priority when someone uses too much resources.
Here is some reference how does Microsoft uses priorities to tune threads CPU consumption.
No the JVM cannot route the ThreadPoolExecutor for you. You may implement a external watcher thread that monitor your threads and apply the appropriate policy to them (priority, exception handling and so on).
Take a look at this example:
http://tutorials.jenkov.com/java-multithreaded-servers/thread-pooled-server.html
I noticed that some web frameworks such as Play Framework allows you to configure multiple thread-pools with different sizes (num of threads within it). Let's say we run this play within a single machine with single core. Wouldn't there be a huge overhead by having multiple thread-pools?
For example, smaller thread pool is assuming asynchronous operations vs large thread-pool indicate a lot of blocking calls so threads can context-switch. Both cases is assuming that parallelism factor based on number of cores are in machine. My concern is that processor is further shared.
How does this work?
Thanks!
Play certainly allows you to configure multiple execution contexts (the equivalent of a thread pool), but that does not mean that you should do it, especially if you have a machine with a single core. By default the configuration should be kept low (close to the number of cores) for high-throughput - assuming, of course, that the operations are all non-blocking. If you have blocking operations the idea is to have them run on a separate execution context, as they otherwise lead to the exhaustion of the default request processing ExecutionContext (the request processing pipeline in Play runs on the default ExecutionContext, which is by default limited to a small number of threads).
As to what happens when you have more threads than cores and what happens when you do so highly depends on the operations you're running (in regards to I/O, etc.). One thread per core is supposedly optimal if you only do CPU-bound operations. See also this question.