I configured like this:
server:
port: 8080
servlet:
context-path: /user
tomcat:
connection-timeout: 3s
threads:
max: 1
min-spare: 1
accept-count: 1
max-connections: 1
But when I tried with Jmeter,
connections are not dropped immediately unitl waiting for 50 seconds.
try this maybe:
server.servlet.session.timeout=60s
In your case, Tomcat will not allow to set the timeout any less than 60 seconds : https://github.com/spring-projects/spring-boot/issues/7383
Related
When the application is started but not yet warmed(Jit need time), it cannot process the expected RPS.
The problem is in the incoming queue. As the IO thread continues to process requests, there are many requests in the queue that the GC cannot clean up. After the overflow of the Survived generation, GC starts perfom major pause, which slows down the execution of requests even more and after some time the application falls on the OOM.
My application have self warmed readnessProbe (3k random request).
I try to configure count of thread and queue size:
application.yml
micronaut:
server:
port: 8080
netty:
parent:
threads: 2
worker:
threads: 2
executors:
io:
n-threads: 1
parallelism: 1
type: FIXED
scheduled:
n-threads: 1
parallelism: 1
corePoolSize: 1
And some props
System.setProperty("io.netty.eventLoop.maxPendingTasks", "16")
System.setProperty("io.netty.eventexecutor.maxPendingTasks", "16")
System.setProperty("io.netty.eventLoopThreads", "1")
But the queue keeps filling up:
i want to find somw way to restriction input queu size at Micronaut, so that the application does not failed under high load
1、environment
OS version and kernel: CentOS Linux release 7.9.2009 3.10.0-1160.el7.x86_64
nginx version: nginx-1.14.2 (Community nginx)
upstream server(tomcat) version: apache tomcat 8.0.53
JDK version: Oracle jdk1.8.0_144
nginx enable keepalive (with client and upstream server)；
HTTP Procotol: HTTP/1.1
2、nginx access log format and tomcat access log patten:
(1) nginx access log format
'$remote_addr - $remote_user [$time_iso8601] '
' "$request" $status $body_bytes_sent $bytes_sent '
' $request_trace_id '
' ["$upstream_addr" "$upstream_status" "$upstream_bytes_received" "$upstream_response_length" "$upstream_cache_status" '
' "$upstream_response_time" "$upstream_connect_time" "$upstream_header_time"] '
' "$request_time" "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$ssl_protocol"';
self-defined variable: $request_trace_id:
#trace.setting
set $request_trace_id $http_x_request_id;
if ( $request_trace_id = '' ) {
set $request_trace_id $pid-$connection-$bytes_sent-$msec;
}
(2) tomcat access log pattern:
"[%{Y-M-d H:m:s.S+z}t] real_ip:%{X-Real-IP}i remote:%h requestid:%{X-Request-ID}i first_line:"%r" status:%s bytes:%b cost:%Dms commit_time:%Fms Agent:"%{User-Agent}i" %{Connection}i %{Connection}o %{Keep-Alive}i %{Keep-Alive}o"
3、Problematic logging contents
(1) nginx logs
192.168.26.73 - cgpadmin [2021-09-09T09:58:23+08:00] "POST /cgp2-oauth/oauth/check_token HTTP/1.1" 200 12983 13364 6462-1025729-0-1631152697.976 ["127.0.0.1:8801" "200" "13353" "12991" "-" "5.026" "0.000" "5.026"] "5.026" "-" "Java/1.8.0_144" "-" "-"
1631152697.976 timestamp:
2021-09-09 09:58:17.976
(2) tomcat logs
[2021-9-9 9:58:17.993+CST] real_ip:192.168.26.73 remote:127.0.0.1 requestid:6462-1025729-0-1631152697.976 first_line:"POST /cgp2-oauth/oauth/check_token HTTP/1.1" status:200 bytes:12991 cost:17ms commit_time:16ms Agent:"Java/1.8.0_144" - - - -
4、My judgment and analysis
Several times of nginx:
- variables
$request_time 5.026 seconds
$upstream_response_time 5.026 seconds
$upstream_header_time 0.000 seconds
$upstream_connect_time 5.026 seconds
- logs timestamp
nginx deal with proxy_pass : 2021-09-09 09:58:17.976
The moment when nginx processing is completed: 2021-09-09T09:58:23
Several times of tomcat:
- attributes
%D 17 millisecond
%F 16 millisecond
- logs timestamp
The moment when tomcat processing is completed: 2021-9-9 9:58:17.993
Analyze the problem stage:
The total processing time of nginx ($request_time) is because the processing time of upstream ($upstream_response_time);
In upstream processing, nginx first prepares the contents of the request message,
and then establishes a connection with Tomcat.
The connection time is very short (the log I excerpted here is 0. It may be because the connection is maintained.
I have seen other connections, which are also very short (there are cases where it is not 0),
so $upstream_header_Time can be basically confirmed
After nginx connects with tomcat, "start sending request message to Tomcat", "Tomcat receiving" (I understand that there may be a queue,
Tomcat receiving), "Tomcat processing" and "Tomcat returning" "Response message", "nginx receives the first byte of the response message header",
which takes a long time. Start to analyze the problem of which section
The recording time of nginx completion time is 9:58:23 (I don't know milliseconds here, because I use the default variable),
while the total time consumption of upstream is 5.026 seconds.
The time when tomcat returns the message is 9:58:17.993,Subtracting 5.026 seconds from 9:58:23 is about 9:58:18, which is similar to the time
when Tomcat returns the message, indicating that there is a difference between nginx and the client
The time-consuming is not long. We can basically confirm that the time when nginx establishes a connection to Tomcat is about 9:58:18.
We happen to have a $request_trace_ID, This is when nginx starts processing the reverse proxy, it is 2021-09-09 09:58:17.976,
which can basically confirm our conjecture.
The time of Tomcat reverse message is 9:58:17.993,It subtracts the internal processing time of Tomcat (% d 17 ms),
and obtains that the time when Tomcat starts processing requests is 9:58:17.976, which is the same as our $request_trace_id 09:58:17.976
It is basically the same, indicating that nginx starts sending request messages to tomcat, and the processing time of sorting threads
from the pool is basically No.
Therefore, the processing time 5 of upstream is after Tomcat prepares the response message,
Send the response message to nginx.
The time when nginx receives the first byte of the response message header (127.0.0.1) basically has no network problems,
and the monitoring does not see it.
Is there a queue in nginx? Isn't upstream an event callback based on epoll
The system here sometimes has pre problems and sometimes post problems (in this case, post problems).
Please give me some ideas to promote this problem. Thank you!
5、reference
https://www.nginx.com/blog/using-nginx-logging-for-application-performance-monitoring/
Tomcat log: what's the difference between %D and %F
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#variables
https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html
https://juejin.cn/post/6844903887757901832
https://cloud.tencent.com/developer/article/1778734
I have spring boot application with embedded jetty and its configurations are:
jetty's minThread: 50
jetty's maxThread: 500
jetty's maxQueueSize: 25000 (I changed default queue to LinkedBlockingQueue)
I didn't change acceptors and selectors (since I dont believe on hard coding the value)
With above configuration, I am getting below jmeter test results:
Concurrent Users: 60
summary = 183571 in 00:01:54 = 1611.9/s Avg: 36 Min: 3 Max:
1062 Err: 0 (0.00%)
Concurrent Users: 75
summary = 496619 in 00:05:00 = 1654.6/s Avg: 45 Min: 3 Max:
1169 Err: 0 (0.00%)
If I increase concurrent users, I dont see any improvement. I want to increase concurrency. How to achieve this?
===========================================================================
Updating on 29-March-2019
I was spending more effort on improving business logic. Still no much improvement. Then I decided to develop one hello world spring-boot project.
i.e.,
spring-boot (1.5.9)
jetty 9.4.15
rest controller which has get endpoint
code below:
#GetMapping
public String index() {
return "Greetings from Spring Boot!";
}
Then I tried to benchmark using apachebench
75 concurrent users:
ab -t 120 -n 1000000 -c 75 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 75
Time taken for tests: 37.184 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 26893.28 [#/sec] (mean)
Time per request: 2.789 [ms] (mean)
Time per request: 0.037 [ms] (mean, across all concurrent requests)
Transfer rate: 3755.61 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 23.5 0 3006
Processing: 0 2 7.8 1 404
Waiting: 0 2 7.8 1 404
Total: 0 3 24.9 2 3007
100 concurrent users:
ab -t 120 -n 1000000 -c 100 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 100
Time taken for tests: 36.708 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 27241.77 [#/sec] (mean)
Time per request: 3.671 [ms] (mean)
Time per request: 0.037 [ms] (mean, across all concurrent requests)
Transfer rate: 3804.27 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 35.7 1 3007
Processing: 0 2 9.4 1 405
Waiting: 0 2 9.4 1 405
Total: 0 4 37.0 2 3009
500 concurrent users:
ab -t 120 -n 1000000 -c 500 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 500
Time taken for tests: 36.222 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 27607.83 [#/sec] (mean)
Time per request: 18.111 [ms] (mean)
Time per request: 0.036 [ms] (mean, across all concurrent requests)
Transfer rate: 3855.39 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 14 126.2 1 7015
Processing: 0 4 22.3 1 811
Waiting: 0 3 22.3 1 810
Total: 0 18 129.2 2 7018
1000 concurrent users:
ab -t 120 -n 1000000 -c 1000 http://10.93.243.87:9000/home/
Server Software:
Server Hostname: 10.93.243.87
Server Port: 9000
Document Path: /home/
Document Length: 27 bytes
Concurrency Level: 1000
Time taken for tests: 36.534 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Total transferred: 143000000 bytes
HTML transferred: 27000000 bytes
Requests per second: 27372.09 [#/sec] (mean)
Time per request: 36.534 [ms] (mean)
Time per request: 0.037 [ms] (mean, across all concurrent requests)
Transfer rate: 3822.47 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 30 190.8 1 7015
Processing: 0 6 31.4 2 1613
Waiting: 0 5 31.4 1 1613
Total: 0 36 195.5 2 7018
From above test run, I achieved ~27K per second with 75 users itself but it looks increasing the users also increasing the latency. Also, we can clearly note connect time is increasing.
I have requirement for my application to support 40k concurrent users (assume all are using own separate browsers) and request should be finished within 250 milliseconds.
Please help me on this
You can try increasing or decreasing the number of Jetty threads but the application performance will depend on the application logic. If your current bottleneck is the database query you will see hardly any improvements by tuning HTTP layer, especially when testing over local network.
Find the bottleneck in your application, attempt to improve it, and then measure again to confirm it's better. Repeat this three steps until achieving desired performance. Do not tune performance blindly, it's a waste of time.
I have an akka (Java) application with camel-jetty consumer. Under some minimum load (about 10 TPS), our client starts seeing HTTP 503 error. I tried to reproduce the problem in our lab, and it seems jetty can't handle overlapping HTTP requests. Below is the output from apache bench (ab):
ab sends 10 requests using one single thread (i.e. one request at a time)
ab -n 10 -c 1 -p bad.txt http://192.168.20.103:8899/pim
Benchmarking 192.168.20.103 (be patient).....done
Server Software: Jetty(8.1.16.v20140903)
Server Hostname: 192.168.20.103
Server Port: 8899
Document Path: /pim
Document Length: 33 bytes
Concurrency Level: 1
Time taken for tests: 0.61265 seconds
Complete requests: 10
Failed requests: 0
Requests per second: 163.23 [#/sec] (mean)
Time per request: 6.126 [ms] (mean)
Time per request: 6.126 [ms] (mean, across all concurrent requests)
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 1 2
Processing: 3 4 1.8 5 7
Waiting: 2 4 1.8 5 7
Total: 3 5 1.9 6 8
Percentage of the requests served within a certain time (ms)
50% 6
66% 6
75% 6
80% 8
90% 8
95% 8
98% 8
99% 8
100% 8 (longest request)
ab sends 10 requests using two threads (up to 2 requests at the same time):
ab -n 10 -c 2 -p bad.txt http://192.168.20.103:8899/pim
Benchmarking 192.168.20.103 (be patient).....done
Server Software: Jetty(8.1.16.v20140903)
Server Hostname: 192.168.20.103
Server Port: 8899
Document Path: /pim
Document Length: 33 bytes
Concurrency Level: 2
Time taken for tests: 30.24549 seconds
Complete requests: 10
Failed requests: 1
(Connect: 0, Length: 1, Exceptions: 0)
// obmited for clarity
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.9 1 2
Processing: 3 3005 9492.9 4 30023
Waiting: 2 3005 9492.7 3 30022
Total: 3 3006 9493.0 5 30024
Percentage of the requests served within a certain time (ms)
50% 5
66% 5
75% 7
80% 7
90% 30024
95% 30024
98% 30024
99% 30024
100% 30024 (longest request)
I don't believe jetty is this bad. Hopefully, it's just a configuration issue. This is the setting for my camel consumer URI:
"jetty:http://0.0.0.0:8899/pim?replyTimeout=70000&autoAck=false"
I am using akka 2.3.12 and camel-jetty 2.15.2
Jetty is certain not that bad and should be able to handle 10s of thousands of connections with many thousands of TPS.
Hard to diagnose from what you have said, other than Jetty does not send 503's when it is under load.... unless perhaps if the Denial of Service protection filter is deployed? (and ab would look like a DOS attack.... which it basically is and is not a great load generator for benchmarking).
So you need to track down who/what is sending that 503 and why.
It was my bad code: the sender (client) info was overwritten with overlapping requests. The 503 error message was sent due to Jetty continuation timeout.
Environment is JBoss 5.1.0,
RestEasy web service deployed and load test with about 10 transaction per seconds.
Need to improve the response time of Resteasy Web service,
Then change the working thread number to 100.
jboss-5.1.0.GA/server/default/deploy/jbossweb.sar/server.xml
<!-- connectionTimeout and keepAliveTimeout should be greater than the
Heartbeat period between SIPServer and XS.
(Here 60s for an Heartbeat less than 60s) -->
<Connector protocol="${http.connector.class}"
port="${xs.http.port}" address="${jboss.bind.address}"
allowTrace="false" enableLookups="false"
SSLEnabled="false"
connectionTimeout="60000" keepAliveTimeout="60000"
maxKeepAliveRequests="-1" maxThreads="100" compression="off" />
while admin-console shows the result is not good, the max processing time is still great than 1000ms.
Max threads: 100
Current thread count: 4
Current thread busy: 4
Max processing time: 1061 ms
Processing time: 55.7 s
Request count: 657
Error count: 1
Bytes received: 0.08 MB
Bytes sent: 0.86 MB
Question is how to improve the response time of JBoss Web Service.