We have the scenario, that we want to deploy ~300 java-applications for a in-house use-case in a kubernetes cluster. A lot of them are just used 4 times a year - and the rest of the year they are just wasting RAM.
To reduce the memory footprint we're currently discussing the following options:
Using a kubernetes-"buildt-in" mechanism, which starts the container when a request will arrive. After a timeout of (f.e. 10 hours) the container will be suspended/hibernated.
Offloading the RAM to disc (for specific containers) is allowed too.
Starting the containers by a "Proxy-Webpage": First, the user have to login to a web-app, where he is searching for and selecting the desired application. OnDemand (perhaps by a kubectl command in background etc) the application will be started.
Does someone have this special use-case, too?
We're starting this roject right now. So other options are helpful too. Just Java as development language is fixed.
Is there a built-in solution in kubernetes, to reduce the memory footprint?
Is our option #3 really a "good" solution?
We are developing a site that will allow users to send semi-real-time events to other users. The UI will display an icon when there is a new event for a user (pretty standard stuff).
I have read that periodic short polling does not scale as well as websockets because it puts more pressure on the web server. I am not quite sure why this would be the case?
We are using tomcat NIO (which does not have a one-to-one connection per thread ratio). As I understand it, Tomcat NIO is pretty good at handling longer HTTP connection timeouts with a small number of threads.
So, if the periodic polling time is less than the connection timeout, then the polling should not have to create another TCP handshake, as it will just reuse an existing HTTP 1.1 connection.
Thus, the above does not seem like it would create too much pressure on the server. It may not be as real-time as long polling or websockets, but I do not see why it should not scale (assuming that the server can quickly respond with a response indicating a new event or not – we use an in memory concurrent hashmap, so this should be pretty fast with no necessary DB access).
Am I missing anything?
Thanks,
-Adam
Short polling may not be as trendy as long polling and web sockets but it works and works everywhere.
Trello (backed by some of the same people as SO) normally uses web sockets but when they encountered a crippling bug in their web sockets implementation on launch day they were saved by short polling:
We hit a problem right after launch. Our WebSocket server implementation started behaving very strangely under the sudden and heavy real-world usage of launching at TechCrunch disrupt, and we were glad to be able to revert to plain polling and tune server performance by adjusting the active and idle polling intervals. It allowed us to degrade gracefully as we increased from 300 to 50,000 users in under a week. We’re back on WebSockets now, but having a working short-polling system still seems like a very prudent fallback.
The full story is well worth a read.
I'd particularly highlight,
The use of HAProxy to terminate the client connection. Meaning that internal web servers are shielded from slow and misbehaving clients and the overhead of repeatedly creating connections becomes less of an issue due to HAProxy's scalability/efficiency;
Trello's polling frequency was adjustable meaning that under heavy load they could tell all clients to poll less frequently thus exchanging responsiveness for increased capacity.
In Brazil at least there are many retail trading platforms that use short polling, with very short polling intervals for rapid publication of stock prices, and regularly support thousands of concurrent users.
Unlike long polling and web sockets, short polling doesn't require a persistent connection so with something like HAProxy in the middle your maximum number of "connections" could actually be greater than the number of concurrent sockets supported by your hardware (although at that point you'd probably be seeing some degradation in responsiveness).
In a Java servlet environment, what are the factors that are the bottleneck for number of simultaneous users.
Number of HTTP connections the server can allow per port
Number of HTTP connections the server can allow across several ports (I can have multiple WAS profiles on several HTTP ports)
Number of servlets in pool
Number of threads configured for WAS to use to service connections
RAM available to server (is there any any correletation between number of service threads assuming 0-memory leak in application)
Are there any other factors?
Edited:
To leave business logic out of the picture, assume have only one servlet printing one line on Log4j.
Can my Tomcat server handle 6000 simultaneous HTTP connections? Why
not (file handles? CPU time per request?)?
Can I have thread pool size as 5000 (do idle threads cost CPU/RAM)?
Can I have oracle connection pool size as 500 connections (do idle
connections cost CPU/RAM)?
Is the amount of garbage that is generated for each connection have an impact? For example, if for each HTTP connection 20KB of objects are created and left behind by Tomcat.. then by the time 2500 requests are processed 100MB heap would be used and this may trigger a GC pause of 300ms.
Can we say something like this: if Tomcat uses 0.2 sec of CPU time for processing a single HTTP request, then it would be able to handle roughly 500 http connections in a second. So, 6000 connections would need 5 seconds.
Interesting question, If we leave apart all the performance deciding attributes finally it boils down to how much work you are doing in the servlet or how much time it takes if it has highest I/O, CPU and memory. Now lets move down with you list with the above statement in mind;-
Number of HTTP connections the server can allow per port
There are limit for file descriptors but that again gets triggered by how much time a servlet is taking complete a request or how much time it takes from request first byte receive to finish sending the entire response. Because if it take only 1ms and you are using Netty and persistent connection, you can reach a really high >> 6000.
Number of servlets in pool
Theoretically >> 6000. But how many thread are processing your requests? Is there a thread pool that is burning your requests ? So you want to increase threads, but how much lets say 2000 concurrent threads. Is your CPU behaving poor with context switching ? Is it I/O bound? if yes it makes sense to context switch but then you will be hitting those network limits because a lot of thread waiting on network I/O, so ultimately how much time you spent on a piece of work.
DB
If it oracle, bless you with connection management, you definitely need rigorous monitoring here. Now this is just another limiting factor and can be considered as an just another blocking I/O. By definition of I/O, latency/throughput matters and becomes a bottleneck the moment it becomes the bigger than the smallest piece of work.
So, finally, you need to break down following or more attributes for all the servlets
Is it CPU bound? If yes, how much cycles it takes or can it be converted safely to some time unit. e.g. 1ms for just the compute piece of work.
Is it I/O bound, If yes similarly find the unit.
and others
A long list of what you have, e.g. CPU, Memory, GB/s
Now you know how much work needs to be done and all you do is divide by what you have and keep tuning , so that you find out the optimal and also find out what else attribute you have not considered and consider them.
The biggest bottleneck I have experienced is the time it takes to process the request.
The faster you can service a request, the more connections you can handle.
It's a difficult question to answer due to every application being different.
To figure this out for an application I support, I created a unit test that spawns many threads and I watch the memory usage in VisualVM in eclipse.
You can see how your memory consumption changes with the number of threads in use.
And you should be able to get a thread dump and see how much memory the thread is using.
You can extrapolate an average out to understand how much RAM you might need for N number of users.
The bottleneck will be a moving target since you'll optimize one area until you can scale larger, then another area will become your bottleneck.
If the response time of the servlet is a bottleneck, you'll could use some queuing mathematics to determine how many requests can be queued optimally based on the avg response time.
http://www4.ncsu.edu/~hp/SSME_QueueingTheory.pdf
Hope this helps.
Updated to address your additional questions:
Can my Tomcat server handle 6000 simultaneous HTTP connections? Why not (file handles? CPU time per request?)?
It's possible but probably not. Also you should probably add a web layer in front of the application server if you plan on doing high volume.
Suppose you have 6000 users all pounding away on your application. Each request a user sends only exists on the server for a moment [hopefully], and your peak thread count may have never reached over 20.
I'd recommend setting up some monitoring to understand how your application performs under real use cases. Check out http://Hawt.io which uses Jolokia to grab JMX metrics via http.
If your serious about analytics I'd recommend using something like Graphite to aggregate your JMX metrics. https://github.com/graphite-project/graphite-web
I've written a collector for Jolokia to send metrics to Carbon/Graphite, and may be able to open-source it with approval from my management. Let me know if you are interested.
Can I have thread pool size as 5000 (do idle threads cost CPU/RAM)?
Idle threads are not much to worry about, though setting your thread pool too high could allow your application server to receive too many requests. If this happens you may end up flooding your DB with connections it cant handle, or your memory allocation may not be enough to handle so many requests. This could start overall application performance degradation.
Set too low, and your app server could start queuing request again causing performance degradation.
It's normally to have some queuing during spikes or high volume times, but you don't want to overload your application server. Check out queuing theory to understand more about this.
Also, this is where having a web server in front of the app server could help you. If you have Apache serve your static content, only dynamic requests will reach the application servers in most cases.
Tuning is very specific to your individual application. I'd recommend staying with the defaults and just optimize your code until you can gather enough data to know which knob should be turned.
Can I have oracle connection pool size as 500 connections (do idle connections cost CPU/RAM)?
Same situation as the application thread pool size. Though your pool size for DB should be much smaller than the app thread count.
500 would be too high for most web applications unless you have very high volume, in which case you may need a DB cluster environment like Oracle RAC.
If the pool is set too high and you start using a lot of connections, your DB hardware will not be able to keep up and you will end up with performance problem on the database server.
The time it takes for a query to return may increase, in turn causing your application response time to increase. The "log jam" effect.
Use profiling or metrics to determine the avg number of active DB connections under normal use, and use that as a baseline for determining the max allowed.
Is the amount of garbage that is generated for each connection have an impact? For example, if for each HTTP connection 20KB of objects are created and left behind by Tomcat.. then by the time 2500 requests are processed 100MB heap would be used and this may trigger a GC pause of 300ms.
The numbers would be different, but yes. Also remember the Full GC are more concern. The incremental GCs will not pause your application. Check out "concurrent mark and sweep" and "Garbage first".
Can we say something like this: if Tomcat uses 0.2 sec of CPU time for processing a single HTTP request, then it would be able to handle roughly 500 http connections in a second. So, 6000 connections would need 5 seconds.
It's not quite that easy as each request is coming in, there are also some being processed and completed. Check out queuing theory to understand this better.
http://www4.ncsu.edu/~hp/SSME_QueueingTheory.pdf
There is another common bottleneck : the size of the database connection pool. But I have an additional remark : when you exhaust the number of allowed HTTP connections, of the number of threads allowed to serve request, you will only reject some requests. But when you exhaust memory (too much sessions with too much data for example), you can crash the whole application.
The difference is that in the case of heavy load for a short time, when load later falls down :
in first case, the application is up and can serve requests normally
in second case the application is down and must be restarted
EDIT :
I forgot to remember real use cases. The biggest problem I ever found for serving numerous concurrent connections is the quality of the database requests (assuming you use a database). There is not a direct impact since there is no maximum number, but you can easily hog all database server resources. Common examples of poor database requests :
no index on a table with a large number of rows
a request (on a big table) that makes no use of any index
the n+1 syndrome : with a ORM when you map a one to many relation to a collection no eagerly when you always need data from the collection
the load full database syndrome : with a ORM when you map all relations as eager, any single request ends in loading a high quantity of dependent data.
What is worse with those problems, is that they can cause no harm in tests when the database is young because there are not that many rows, but with time and increasing number of rows performances fall giving a unusable application over few users.
Number of HTTP connections the server can allow per port
Unlimited except by kernel resources, e.g. FDs, socket buffer soace, etc.
Number of HTTP connections the server can allow across several ports (I can have multiple WAS profiles on several HTTP ports)
As the number of connections per port is unlimited, this irrelevant.
Number of servlets in pool
Irrelevant except insofar as it increases the rate of incoming requests.
Number of threads configured for WAS to use to service connections
Relevant in an indirect way, see below.
RAM available to server (is there any any correletation between number of service threads assuming 0-memory leak in application)
Relevant if it limits the number of threads below the configured number of threads mentioned above.
The fundamental limitation is request service time. The shorter, the better. The longer it is, the longer the thread is tied up in that request, the longer wait queues get, ... Queuing theory dictates that the 'sweet spot' is no more than 70% server utilization. Beyond that, wait times grow rapidly with increasing utilization.
So anything that contributes to request service time is significant: for example, thread pool size, connection pool size, concurrency bottlenecks, ...
You should also consider that the use case itself is limiting the amount of concurrency. Imagine a collaborative environment where the order of actions matters. This forces you to synchronize actions - even if you would have been able to process all of them at once.
In java land this could be a simple thing as sharing a single resource which is using blocking access. (e.g. shared Random number generators (not per thread), shared Vectors, concurrent structures like ConcurrentHashMap etc.).
The more synchronization the less you will be able to fully utilize your server hardware.
So apart from running out of memory or saturating the CPU or hitting the garbage collection limit this synchronization might be a problem which does not only need to be solved in your code but maybe even requires you to soften some requirements of the high level workflow.
Seeing point 6, you can use these tools to see if your hardware is being the bottleneck: Assuming that you're on linux, you can use VmStat to see some statistics on your RAM usage, top or atop (depending on your distro) to see processes taking a toll in your CPU and RAM, nload and iftop to see what is consuming network bandwith, and iotop to see what is reading and writing to your disk.
On 32 bit systems, JVM has a memory limit of 1.5 to 2 GB. What is a good value of JVM memory on 64 bit Linux ? How that can be mapped to maximum number of threads and maximum requests in tomcat ?
I am using JDK 6+ and tomcat 7. RAM available will be 12 GB on a quad core processor.
MRD
I don't think there's an out of the box answer to this question. This depends heavily on what kind of applications you are going to host and how much is it going to be on your system. I administer a small server with 3-4 applöications on a 64bit linux system. Using 4GB is more than enough for me.
My advise is make a wild guess how much ram is required for your applications. Then startup tomcat with a monitor tool then watch how much load is there on your tomcat. You might have allocated too much resource for tomcat. Maybe too few. you never know
Please read this article on Simultaneous users, and also the article about load balancing in tomcat
Basically you have to differentiate between users and requests. you might have 5000 users browsing your site, but only 100 making requests for a new page at one moment. By default tomcat supports 50 concurrent requests (not 100% sure though). But this number can be changed in your tomcat configuration. Obviously you might need more hardware. In the second article, max 200 requests per tomcat instance is recommended. only simple calculation rules as mentioned in the second article and doing some monitoring can help.
There's even a load balancer manager for tomcat. Check it out
load balancer for tomcat
One more thing to think of, is although you have the hardware and the right load balancing to support 5000 users, you also need enough bandwidth to do so. Again explained in the second article "load balancing in tomcat"
Good luck
It depends on how many users will visit you application simultaneously.
Sometimes, the app will run very slowly at a particular time point,
For instance, at 8:00 AM, login action causes the app can't stand.
I suggest you to estimate average memory per user, according the “total number of users",
Then you may get a nearly almost RIGHT memory setting.
I'm very much a Tomcat newbie, so I'm guessing that the answer to this is pretty straightforward, but Google is not being friendly to me today.
I have a Java web application mounted on Apache Tomcat. Whilst the application has a front page (for diagnostic purposes), the application is really all about a SOAP interface. No client will ever need to look up the server's web page. The clients send SOAP requests to the server, which parses the requests and then looks up results in a database. The results are then passed back to the clients, again over SOAP.
In its default configuration, Tomcat appears to queue requests. My experiment consisted of installing the client on two separate machines pointing at the same server and running a search at exactly the same time (well, one was 0.11 seconds after the other, but you get the picture).
How do I configure the number of concurrent request threads?
My ideal configuration would be to have X request threads, each of which recycles itself (i.e. calls destructor and constructor and recycles its memory allocation) every Y minutes, or after Z requests, whichever is the sooner. I'm told that one can configure IIS to do this (although I also have no experience with IIS), but how would you do this with Tomcat?
I'd like to be able to recycle threads because Tomcat seems to be grabbing memory when a request comes in and not releasing it, which means that I get occasional (but not consistent) Java Heap Space errors when we are approaching the memory limit (which I have already configured to be 1GB on a 2GB server). I'm not 100% sure if this is due to a memory leak in my application, or just that the tools that I'm using use a lot of memory.
Any advice would be gratefully appreciated.
Thanks,
Rik
Tomcat, by default, can handle up to 150 concurrent HTTP requests - this is totally configurable and obviously varies depending on your server spec and application.
However, if your app has to handle 'bursts' of connections, I'd recommend looking into Tomcat's min and max "spare" threads. These are threads actively waiting for a connection. If there aren't enough waiting threads, Tomcat has to allocate more (which incurs a slight overhead), so you might see a delay.
Also, have a look at my answer to this question which covers how to configure the connector:
Tomcat HTTP Connector Threads
In addition, look at basic JVM tuning - especially in relation to heap allocation overhead and GC pause times.