I am using JMeter to do stress testing to my project.
I set the Constant Throughout Timer as 20/s, and last for 100s. But the actual is: the TPS most get 7/s, and never get 20/s. I don't know why is this.
I want to ask 2 questions:
1、As my understanding, the JMeter can simulate the stress so it can get TPS at 20/s, is there any config wrong of my JMeter?
2、If for other case, the Jmeter can simulate the stress only depend on the server, is it mean there is problem with my server lead to the TPS can't be improved?
Constant Throughput Timer can only pause the threads in order to limit JMeter to the defined number of requests per minute. Moreover, it's more or less precise only on "minute" level, if your test lasts less - you may not see the impact.
Another factor is application response time as JMeter waits for the previous response before sending the next request therefore if the application doesn't respond fast enough you will not be able to reach the desired number of requests per second
And last but not the least is that JMeter should have enough resources in order to send required number of requests per second because if JMeter is not properly configured or the machine where JMeter is running is overloaded - you will not be able to send requests fast enough even the application can handle more load.
So recommendations are:
Make sure you provide enough threads (virtual users) to conduct the required load in the Thread Group
Make sure to follow JMeter Best Practices
Make sure to monitor the baseline health metrics of the machine where JMeter is running (CPU, RAM, Network, Disk, etc.), it can be done using JMeter PerfMon Plugin. If you detect the lack of resources you will need to find another machine and go for Distributed Testing
You can consider Concurrency Thread Group and Throughput Shaping Timer combination instead of your current setup, they can be connected via Feedback Function so JMeter will be able to kick off more threads if current amount is not enough to create the necessary load
Related
TLDR:
JMeter crawls with just 1 test user if I select "Retrieve all embedded resources".
JMeter throws Uncaught Exception java.lang.OutOfMemoryError: Java heap space in thread Thread if I test with 100 users
How to eventually scale to test with thousands of users?
More details:
I'm recording the jmx script using the BlazeMeter Chrome extension. The user logs in, and completes a course and a test. There are many ajax requests being made along the way. Around 350 - 400 steps for the script. Mostly ajax POST request that have a json response.
When I click through manually in Chrome, the site loads quickly. Maybe a page load takes 2 seconds tops.
But when I import that script into JMeter with the "Retrieve all embedded resources" and "Parallel downloads" set to 6, and run it (in the GUI initially with just 1 user), it will get through, say, 7 steps quickly, and then just hang, sometimes for 10+ minutes before advancing to the next step. This doesn't happen if I uncheck "Retrieve all embedded resources", but I don't want to do that since that wouldn't be a realistic test.
If I take that same test and run it with 100 users (from the command line using JVM_ARGS='-Xms4096m -Xmx4096m' sh jmeter -n -t myfolder/mytest.jmx -l myfolder/testresults.jtl), I get Uncaught Exception java.lang.OutOfMemoryError: Java heap space in thread Thread and my computer fan goes nuts.
I have an HTTP Cache Manager configured with only "Use Cache-Control/Expires header when processing GET requests" checked and I've lowered the "Max Number of elements in cache down to 10 since that's the only way I can get the test running at all.
Ideally, I'd like to test thousands of users at once, but if I can't reliably get 100 users tested, I don't understand how I'm supposed to scale to thousands.
I see that there are other cloud-based testing options, and I've tried a few out now, however I always come to a halt when it comes to configuring how to test logged in users. Seems like most solutions don't support this.
Feels like the kind of thing that lots of people should have come across in the past, but I find almost no one having these issues. What's the right way to load test thousands of logged-in users on a web application?
If JMeter "hangs" for 10 minutes retrieving an embedded resource there is a problem with this specific resource, you can see the response times for each resource using i.e. View Results Tree listener and either raise a bug for this particular "slow" component or exclude it from the scope using HTTP Request Defaults. There you can also specify the timeout, if the request won't return the response within the given timeframe - it will be marked as failed so you won't have to wait for 10 minutes:
Your way of increasing the heap is correct, it looks like 4Gb is not sufficient you will either have to provide more or consider switching to distributed testing
Make sure to follow JMeter Best Practices in any case. If your "computer fan goes nuts" it means that CPU usage is very high therefore JMeter most probably won't be able to send requests fast enough so you will get false-negative results.
It's difficult to simulate a real stress scenario when every micro-service is under stress. Generally in load test, there's only few micro-service which are under stress and not all the micro-services. Is there any way we can increase the response time of all the outgoing calls from my service, so that it's easy to estimate the actual response time of my service during a load test or any way how to implement that.
I havent tried anything because i dont have any clue about it.
If you have a complex application built on top of microservice architecture pattern be aware that it will always act on speed of its slowest component.
So your goal is not to stress all the microservices, but to determine what is the slowest one. Once done you can investigate the reason, suggest and/or implement the fix and repeat the test until you will be satisfied with the result.
The best approach would be going for stress testing of the whole integrated system, starting from 1 virtual user and increasing the load until transport starts queuing up at ingress of one of the microservices.
Once you figure out which is the slowest microservice you should be able to stress-test it separately and use monitoring or profiling tools in order to determine the bottleneck.
You could consider two of these options.
Gatling - https://www.baeldung.com/load-test-a-website-with-gatling
JMeter - https://octoperf.com/blog/2018/04/23/jmeter-rest-api-testing/
Our multi-threaded Java application is using the Java XCC library. Over MarkLogic admin console under status tab only 2 threads are shown as active while the application is running, that is the most probable reason of bottleneck in our project. Please advise what is wrong here?
To effectively run xcc requests in parallel you need to make sure you are using separate Sessions for each thread. See:
https://docs.marklogic.com/javadoc/xcc/com/marklogic/xcc/Session.html
Having only 2 active threads running is not necessarily a sign of a problem, its possible that your requests are being processed as fast as you issue them and read the response. If your queries are fast enough there is no need for more threads. Without more information about your queryies, response times and server load its not possible to say if there is a bottleneck or not. How many threads are you running ? Compare the response time as you increase threads. Check that you have sufficient network IO so that your requests are not bottlenecked in the network layer.
I suggest profiling your queries and using the Performance History console to see if the server is running at high utilization. Try increasing the number of client threads, possibly running them from different servers.
My Java web application pulls some data from external systems (JSON over HTTP) both live whenever the users of my application request it and batch (nightly updates for cases where no user has requested it). The data changes so caching options are likely exhausted.
The external systems have some throttling in place, the exact parameters of which I don't know, and which likely change depending on system load (e.g., peak times 10 requests per second from one IP address, off-peak times 100 requests per second from open IP address). If the requests are too frequent, they time out or return HTTP 503.
Right now I am attempting the request 5 times with 2000ms delay between each, giving up if an error is received each time. This is not optimal as sometimes at peak-times nearly all requests fail; I could avoid making these requests and perhaps get at least some to succeed instead.
My goals are to have a somewhat simple, reliable design, and enough flexibility so that I could both pull some metrics from the throttler to understand how well the external systems are responding (and thus adjust how often they are invoked), and to auto-adjust the interval with which I call them (individually per system) so that it is optimal both on off-peak and peak hours.
My infrastructure is Java with RabbitMQ over MongoDB over Linux.
I'm thinking of three main options:
Since I already have RabbitMQ used for batch processing, I could just introduce a queue to which the web processes would send the requests they have for external systems, then worker processes would read from that queue, throttle themselves as needed, and return the results. This would allow running multiple parallel worker processes on more servers if needed. My main concern is that it isn't a very simple solution, and how to manage peak-hour throughput being low and thus the web processes waiting for a long while. Also this converts my RabbitMQ into a critical single failure point; if it dies the whole system stops (as opposed to the nightly batch processes just not running any more, which is less critical). I suppose rpc is the correct pattern of RabbitMQ usage, but not sure. Edit - I've posted a related question How to properly implement RabbitMQ RPC from Java servlet web container? on how to implement this.
Introduce nginx (e.g. ngx_http_limit_req_module), HAProxy (link) or other proxy software to the mix (as reverse proxies?), have them take care of the throttling through some configuration magic. The pro is that I don't have to make code changes. The con is that it is more technology used, and one I've not used before, so chances of misconfiguring something are quite high. It would also likely not be easy to do dynamic throttling depending on external server load, or prioritizing live requests over batch requests, or get statistics of how the throttling is doing. Also, most documentation and examples will likely be on throttling incoming requests, not outgoing.
Do a pure-Java solution (e.g., leaky bucket implementation). Would be simple in the sense that it is "just code", but the devil is in the details; debugging all the deadlocks, starvations and race conditions isn't always fun.
What am I missing here?
Which is the best solution in this case?
P.S. Somewhat related question - what's the proper approach to log all the external system invocations, so that statistics are collected as to how often I invoke them, and what the success rate is?
E.g., after every invocation I'd invoke something like .logExternalSystemInvocation(externalSystemName, wasSuccessful, elapsedTimeMills), and then get some aggregate data out of it whenever needed.
Is there a standard library/tool to use, or do I have to roll my own?
If I use option 1. with RabbitMQ, is there a way to organize the flow so that I get this out of the box from the RabbitMQ console? I wouldn't want to send all failed messages to poison queue, it would fill up too quickly though and in most cases there is no need to re-process these failed requests as the user has already sadly moved on.
Perhaps this open source system can help you a little: http://code.google.com/p/valogato/
How to you simulate > 120 concurrent users using a Load Test Framework such as JMeter? Real concurrency, as far as I understand, would be possible if I would use 120 servers or 120 CPU cores.
How did/do you test your web application or service with real conditions?
I have found JMeter to work fine, but I generally use is across 4-5 PC's to get the most accurate results.
Although you may believe 120 concurrent users may be difficult to simulate from a single PC, you have to realise that in a real work scenario, 120 simultaneous users will not be accessing the server at exactly the same time, therefore the CPU threading algorithms are sufficient to simulate the load.
What you need to do is to understand the likely usage of your application from your users, i.e. how many requests per second/minute you will get from a single user and make sure that your test simulates this effectively.
So, our technique is to use JMeter running on 5 PCs running different tests, and to monitor the server performance during the tests.
There are of course many other products available that for a price are able to better simulate web traffic, but I have always found JMeter to be perfectly good for my needs.