Develop Reference

Issue with LoggingFeature not logging the complete response body

I'm currently working building an endpoint that will make a couple of requests to an external API. In this endpoint I'm logging all of the request bodies to the terminal. But I'm having an issue where jerseys LoggingFeature doesn't log the complete response body for a request that is done. The log gets cut in the middle of the body. I've tried messing around with the logger level, verbosity but that did nothing. I tried changing maxEntitySize to be a large value but that doesn't change anything either, I'm not really sure what the issue is and why it won't log the complete response body. I've also tried logging to a file instead of the terminal but unfortunately I got the same result.
Here's what the log looks like (as it's proprietary software I had to change some information)
INFO [2022-10-06 13:54:04,261] se.bla.bla.bla.clients.BlaClientImpl: 1 * Sending client request on thread dw-18 - GET /api/bla/log/98014384
1 > GET https://url.com/2574339575
1 > Accept: application/json
1 > Authorization: Bearer tokenstuff
INFO [2022-10-06 13:54:04,475] se.bla.bla.bla.clients.BlaClientImpl: 1 * Client response received on thread dw-18 - GET /api/bla/log/98014384
1 < 200
1 < Cache-Control: no-cache
1 < Content-Type: application/json; charset=utf-8
1 < Date: Thu, 06 Oct 2022 13:54:04 GMT
1 < Etag: Yee
1 < Server-Response-Time: 169.44395400000002
1 < Transfer-Encoding: chunked
1 < Vary: Origin
1 < X-Request-Id: Woo
1 < X-Runtime: 0.169337
{example1: "2134123123",example2: "2134123123",example3: "2134123123",example4: "2134123123",example5: "2134123123",example6: "2134123123",example7: "2134123123",example8: "2134123123",example9: "2134123123",example10: "2134123123",example11: "2134123123",example12: "2134123123",example13: "2134123123",example14: "2134123123",example15: "2134123123",example16: "2134123123",example17: "2134123123",example18: "21341
And I've registered the logger like so.
val restlog = getLogger(BlaClientImpl::class.java.name)
restlog.addHandler(FileHandler("rest.log", 83886080, 1))
val requestBuilder =
client
.register(
LoggingFeature(
restlog,
Level.INFO,
LoggingFeature.Verbosity.PAYLOAD_ANY,
83886080
)
)
I'm quite new to the Kotlin/Java world so maybe I'm missing something fundamental.

The issue was a settings with Dropwizard/Jersey. Apparently I had to set gzipEnabled = true

RestTemplate with Proxy with kerberos authentication in Java/Kotlin

I'm trying to access rest api through my companies proxy server this requires me authenticate with the proxy.
I can access the outside world using curl with the parameters:
curl -v -s -user $kerbrosId -x $ProxyServer:$proxyPort --proxy-negotiate https://google.com
I receive the following output:
Enter proxy password for user '$user':
* Trying $proxyIP:$proxyPort...
* Connected to $proxyURL ($proxyIP) port $proxyPort (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to google.com:443
* Proxy auth using Negotiate with user 'bradsw'
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> Proxy-Authorization: Negotiate ${shortBase64token}
> User-Agent: curl/7.83.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NEGOTIATE ${longerBase64token}
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: Keep-Alive
< Connection: Keep-Alive
< Content-Length: 5621
<
* Ignore 5621 bytes of response-body
* Establish HTTP proxy tunnel to google.com:443
* Proxy auth using Negotiate with user '$user'
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> Proxy-Authorization: Negotiate ${evenLongerBase64token}
> User-Agent: curl/7.83.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
> GET / HTTP/1.1
> Host: google.com
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Location: https://www.google.com/
< Content-Type: text/html; charset=UTF-8
< Date: Thu, 29 Sep 2022 22:09:00 GMT
< Expires: Sat, 29 Oct 2022 22:09:00 GMT
< Cache-Control: public, max-age=2592000
< Server: gws
< Content-Length: 220
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
I have tried a million variations of
RestTemplate(HttpComponentsClientHttpRequestFactory(
HttpClientBuilder.create()
.setProxy(HttpHost(proxyUrl, proxyPort))
//other calls here to set things up
.build()
)
)
Nothing seems to authenticate correctly. I get back a 407 PROXY_AUTHENTICATION_REQUIRED with the companies default call the help desk if your browser doesn't work page in my stack trace.
Our systems are configured with an appropriate kbr5.config file so that should not be the issue & in other places in the code we are getting an SSO token a based on a ticket stored by kinit using the equivalent of RestTemplate().getForEntity("${sso-endpoint}", String::class.java).
I would ideally like to be able to delegate credentials to the OS via logged on user (either windows or REL based Linux systems) though I would consider simply authenticating at all a win at this point. I'm working in Kotlin but can easily translate a Java solution to Kotlin.
Any ideas appreciated?

Apache HTTP Client 5.0 HTTP/2 POST Example needed

I can use CURL to send a HTTP/2 POST request with a payload to the https://httpbin.org/post as in the following example:
"c:\Program Files\Git\mingw64\bin\curl" -v --http2 https://httpbin.org/post -d "arg1=param1&arg2=param2"
* Trying 54.166.163.67:443...
* Connected to httpbin.org (54.166.163.67) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: c:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=httpbin.org
* start date: Dec 21 00:00:00 2020 GMT
* expire date: Jan 19 23:59:59 2022 GMT
* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x160f6b61d20)
> POST /post HTTP/2
> Host: httpbin.org
> user-agent: curl/7.75.0
> accept: */*
> content-length: 23
> content-type: application/x-www-form-urlencoded
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* We are completely uploaded and fine
< HTTP/2 200
< date: Fri, 05 Mar 2021 16:07:51 GMT
< content-type: application/json
< content-length: 453
< server: gunicorn/19.9.0
< access-control-allow-origin: *
< access-control-allow-credentials: true
<
{
"args": {},
"data": "",
"files": {},
"form": {
"arg1": "param1",
"arg2": "param2"
},
"headers": {
"Accept": "*/*",
"Content-Length": "23",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "curl/7.75.0",
"X-Amzn-Trace-Id": "Root=1-60425757-45ab281925b9b40e70398269"
},
"json": null,
"origin": "51.37.211.242",
"url": "https://httpbin.org/post"
}
* Connection #0 to host httpbin.org left intact
You can see it sends the arg1/arg2 parameters and they are in the echoed back response.
I would like to replicate that code in Java. I have had no luck myself doing it and cannot find an example of a HTTP/2 POST with a payload. There are some examples of GET/POST calls using HTTP/2 without a payload, and indeed I can write the code to make a request without a payload that works, but I cannot get the payload to send. If I use the same code but using HTTP/1 the payload is sent. Using BasicRequestProducer class you can give it producers (to send the request data I hope) and consumers (to consume the response) but the producer never seems to send the data.
See my other question on HTTP/2 Connection Closed for the sort of code I am using.
I have added some DEBUG to my examples and I can see the content/body being added to the request (or at least it looks to me like it is being added) so is there an issue with the correct formatting of the message?
188 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - Frame 1 >>[type=0, flags=0, streamId=1, payoad=java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]]
188 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - Frame info : Frame 1 >> [type=0, flags=0, streamId=1, payoad=java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]]
189 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - stream 1 frame: DATA (0x0); flags: (0x0); length: 23
189 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - Frame Payload : Frame 1 >> java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]
189 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - Code=99&Message= 43 6f 64 65 3d 39 39 26 4d 65 73 73 61 67 65 3d
Goodbye 47 6f 6f 64 62 79 65
189 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - OutputFlow 0 >>0 -23 65512
190 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - OutputFlow 1 >>1 -23 65512
Can anyone give Java code to demonstrate a successful HTTP/2 POST with a payload????

This is the most simple form of a POST request that works the same way with HTTP/1.1 or HTTP/2 transports
final IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
.setSoTimeout(Timeout.ofSeconds(5))
.build();
final CloseableHttpAsyncClient client = HttpAsyncClients.custom()
.setIOReactorConfig(ioReactorConfig)
.build();
client.start();
final HttpHost target = new HttpHost("nghttp2.org");
final String requestUri = "/httpbin/post";
final AsyncRequestProducer requestProducer = new BasicRequestProducer(Method.POST, target, requestUri,
AsyncEntityProducers.create("stuff", ContentType.TEXT_PLAIN));
System.out.println("Executing POST request to " + requestUri);
final Future<Message<HttpResponse, String>> future = client.execute(
requestProducer,
new BasicResponseConsumer<String>(new StringAsyncEntityConsumer()),
new FutureCallback<Message<HttpResponse, String>>() {
#Override
public void completed(final Message<HttpResponse, String> message) {
System.out.println(requestUri + "->" + message.getHead().getCode());
System.out.println(message.getBody());
}
#Override
public void failed(final Exception ex) {
System.out.println(requestUri + "->" + ex);
}
#Override
public void cancelled() {
System.out.println(requestUri + " cancelled");
}
});
future.get();
System.out.println("Shutting down");
client.close(CloseMode.GRACEFUL);
Here's application output including the header / context log of the session
Executing POST request to /httpbin/post
2021-03-07 17:10:10,379 DEBUG ex-0000000001 preparing request execution
2021-03-07 17:10:10,390 DEBUG ex-0000000001 Cookie spec selected: strict
2021-03-07 17:10:10,396 DEBUG ex-0000000001 Auth cache not set in the context
2021-03-07 17:10:10,396 DEBUG ex-0000000001 target auth state: UNCHALLENGED
2021-03-07 17:10:10,397 DEBUG ex-0000000001 proxy auth state: UNCHALLENGED
2021-03-07 17:10:10,399 DEBUG ex-0000000001 acquiring connection with route {s}->https://nghttp2.org:443
2021-03-07 17:10:10,399 DEBUG ex-0000000001 acquiring endpoint (3 MINUTES)
2021-03-07 17:10:10,401 DEBUG ex-0000000001 endpoint lease request (3 MINUTES) [route: {s}->https://nghttp2.org:443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 25]
2021-03-07 17:10:10,406 DEBUG ex-0000000001 endpoint leased [route: {s}->https://nghttp2.org:443][total available: 0; route allocated: 1 of 5; total allocated: 1 of 25]
2021-03-07 17:10:10,406 DEBUG ex-0000000001 acquired ep-0000000000
2021-03-07 17:10:10,407 DEBUG ex-0000000001 acquired endpoint ep-0000000000
2021-03-07 17:10:10,407 DEBUG ep-0000000000 connecting endpoint (3 MINUTES)
2021-03-07 17:10:10,408 DEBUG ep-0000000000 connecting endpoint to https://nghttp2.org:443 (3 MINUTES)
2021-03-07 17:10:10,408 DEBUG https://nghttp2.org:443 resolving remote address
2021-03-07 17:10:10,410 DEBUG https://nghttp2.org:443 resolved to [nghttp2.org/139.162.123.134, nghttp2.org/2400:8902:0:0:f03c:91ff:fe69:a454]
2021-03-07 17:10:10,410 DEBUG https://nghttp2.org:443 connecting null to nghttp2.org/139.162.123.134:443 (3 MINUTES)
2021-03-07 17:10:10,722 DEBUG c-0000000000[ACTIVE][rc:c] protocol upgrade class org.apache.hc.core5.http2.impl.nio.ClientHttpProtocolNegotiator
2021-03-07 17:10:10,723 DEBUG https://nghttp2.org:443 connected c-0000000000 /192.168.8.105:55230->nghttp2.org/139.162.123.134:443
2021-03-07 17:10:10,725 DEBUG c-0000000000 start TLS
2021-03-07 17:10:10,769 DEBUG ep-0000000000 connected c-0000000000
2021-03-07 17:10:10,769 DEBUG ep-0000000000 endpoint connected
2021-03-07 17:10:10,769 DEBUG ex-0000000001 connected to target
2021-03-07 17:10:10,770 DEBUG ex-0000000001 route fully established
2021-03-07 17:10:10,770 DEBUG ex-0000000001 executing POST /httpbin/post HTTP/1.1
2021-03-07 17:10:10,771 DEBUG ep-0000000000 start execution ex-0000000001
2021-03-07 17:10:10,771 DEBUG ep-0000000000 executing exchange ex-0000000001 over c-0000000000
2021-03-07 17:10:10,772 DEBUG c-0000000000 RequestExecutionCommand with NORMAL priority
2021-03-07 17:10:10,773 DEBUG c-0000000000[ACTIVE][rwc:c][ACTIVE][rw][NOT_HANDSHAKING][0][0][0] Enqueued RequestExecutionCommand with priority IMMEDIATE
2021-03-07 17:10:10,773 DEBUG c-0000000000[ACTIVE][rw:c][ACTIVE][rw][NOT_HANDSHAKING][0][0][0] Event cleared [c]
2021-03-07 17:10:10,775 DEBUG Enabled protocols: [TLSv1.2]
2021-03-07 17:10:10,775 DEBUG Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2021-03-07 17:10:11,563 DEBUG Secure session established
2021-03-07 17:10:11,563 DEBUG negotiated protocol: TLSv1.2
2021-03-07 17:10:11,564 DEBUG negotiated cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
2021-03-07 17:10:11,565 DEBUG peer principal: CN=nghttp2.org
2021-03-07 17:10:11,565 DEBUG peer alternative names: [nghttp2.org, www.nghttp2.org]
2021-03-07 17:10:11,566 DEBUG issuer principal: CN=R3, O=Let's Encrypt, C=US
2021-03-07 17:10:11,573 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][0][0] Event set [w]
2021-03-07 17:10:11,575 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][0][53] 24 bytes written
2021-03-07 17:10:11,575 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][r][NOT_HANDSHAKING][62][0][53] Event cleared [w]
2021-03-07 17:10:11,606 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][r][NOT_HANDSHAKING][62][0][53] protocol upgrade class org.apache.hc.core5.http2.impl.nio.ClientH2IOEventHandler
2021-03-07 17:10:11,608 DEBUG c-0000000000 >> stream 0 frame: SETTINGS (0x4); flags: (0x0); length: 36
2021-03-07 17:10:11,609 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][r][NOT_HANDSHAKING][62][0][53] 0 bytes written
2021-03-07 17:10:11,609 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][0][53] Event set [w]
2021-03-07 17:10:11,609 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][0][53] Event set [w]
2021-03-07 17:10:11,609 DEBUG c-0000000000 << stream 0 flow control 2147418112 -> 2147483647
2021-03-07 17:10:11,609 DEBUG c-0000000000 << stream 0 flow control 2147483647 -> 2147483647
2021-03-07 17:10:11,609 DEBUG c-0000000000 >> stream 0 flow control 65535 -> 65535
2021-03-07 17:10:11,610 DEBUG c-0000000000 << stream 0 frame: SETTINGS (0x4); flags: (0x0); length: 24
2021-03-07 17:10:11,611 DEBUG c-0000000000 >> stream 0 flow control 983041 -> 1048576
2021-03-07 17:10:11,611 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][33][53] Event set [w]
2021-03-07 17:10:11,611 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][33][53] 0 bytes read
2021-03-07 17:10:11,611 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][0][0][53] 0 bytes written
2021-03-07 17:10:11,614 DEBUG c-0000000000 << stream 1 flow control 65535 -> 65535
2021-03-07 17:10:11,614 DEBUG c-0000000000 >> stream 1 flow control 1048576 -> 1048576
2021-03-07 17:10:11,615 DEBUG ex-0000000001 send request POST /httpbin/post HTTP/1.1, entity len 5
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> :method: POST
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> :scheme: https
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> :authority: nghttp2.org
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> :path: /httpbin/post
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> user-agent: Apache-HttpAsyncClient/5.1-beta2-SNAPSHOT (Java/1.8.0_282)
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> content-type: text/plain; charset=ISO-8859-1
2021-03-07 17:10:11,620 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][0][0][53] Event set [w]
2021-03-07 17:10:11,620 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,620 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,621 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][74] 45 bytes written
2021-03-07 17:10:11,621 DEBUG c-0000000000 >> stream 0 frame: WINDOW_UPDATE (0x8); flags: (0x0); length: 4
2021-03-07 17:10:11,621 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][74] 0 bytes written
2021-03-07 17:10:11,621 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,621 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,622 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][42] 13 bytes written
2021-03-07 17:10:11,622 DEBUG c-0000000000 >> stream 0 frame: SETTINGS (0x4); flags: ACK (0x1); length: 0
2021-03-07 17:10:11,622 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][42] 0 bytes written
2021-03-07 17:10:11,622 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,622 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,622 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][38] 9 bytes written
2021-03-07 17:10:11,623 DEBUG c-0000000000 >> stream 1 frame: HEADERS (0x1); flags: END_HEADERS (0x4); length: 93
2021-03-07 17:10:11,623 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][38] 0 bytes written
2021-03-07 17:10:11,623 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,623 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,623 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][131] 102 bytes written
2021-03-07 17:10:11,623 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,623 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,623 DEBUG c-0000000000 >> stream 1 frame: DATA (0x0); flags: (0x0); length: 5
2021-03-07 17:10:11,624 DEBUG c-0000000000 >> stream 0 flow control -5 -> 1048571
2021-03-07 17:10:11,624 DEBUG c-0000000000 >> stream 1 flow control -5 -> 1048571
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][131] 0 bytes written
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][131] Event set [w]
2021-03-07 17:10:11,624 DEBUG ex-0000000001 end of request data
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][131] Event set [w]
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][43] 14 bytes written
2021-03-07 17:10:11,624 DEBUG c-0000000000 >> stream 1 frame: DATA (0x0); flags: END_STREAM (0x1); length: 0
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][43] 0 bytes written
2021-03-07 17:10:11,625 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][38] 9 bytes written
2021-03-07 17:10:11,625 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][r][NOT_HANDSHAKING][0][0][38] Event cleared [w]
2021-03-07 17:10:11,968 DEBUG c-0000000000 << stream 0 frame: SETTINGS (0x4); flags: ACK (0x1); length: 0
2021-03-07 17:10:11,969 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][38][9][0] Event set [w]
2021-03-07 17:10:11,969 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][38][9][0] 0 bytes read
2021-03-07 17:10:11,970 DEBUG c-0000000000 << stream 1 frame: HEADERS (0x1); flags: END_HEADERS (0x4); length: 231
2021-03-07 17:10:11,971 DEBUG c-0000000000 << :status: 200
2021-03-07 17:10:11,971 DEBUG c-0000000000 << date: Sun, 07 Mar 2021 16:10:11 GMT
2021-03-07 17:10:11,971 DEBUG c-0000000000 << content-type: application/json
2021-03-07 17:10:11,971 DEBUG c-0000000000 << content-length: 382
2021-03-07 17:10:11,971 DEBUG c-0000000000 << access-control-allow-origin: *
2021-03-07 17:10:11,971 DEBUG c-0000000000 << access-control-allow-credentials: true
2021-03-07 17:10:11,971 DEBUG c-0000000000 << x-backend-header-rtt: 0.003465
2021-03-07 17:10:11,971 DEBUG c-0000000000 << strict-transport-security: max-age=31536000
2021-03-07 17:10:11,971 DEBUG c-0000000000 << server: nghttpx
2021-03-07 17:10:11,971 DEBUG c-0000000000 << via: 1.1 nghttpx
2021-03-07 17:10:11,971 DEBUG c-0000000000 << alt-svc: h3-29=":443"; ma=3600
2021-03-07 17:10:11,972 DEBUG c-0000000000 << x-frame-options: SAMEORIGIN
2021-03-07 17:10:11,972 DEBUG c-0000000000 << x-xss-protection: 1; mode=block
2021-03-07 17:10:11,972 DEBUG c-0000000000 << x-content-type-options: nosniff
2021-03-07 17:10:11,975 DEBUG ex-0000000001 consume response HTTP/2.0 200 OK, entity len -1
2021-03-07 17:10:11,978 DEBUG c-0000000000 << stream 1 frame: DATA (0x0); flags: END_STREAM (0x1); length: 382
2021-03-07 17:10:11,979 DEBUG c-0000000000 << stream 1 flow control -382 -> 65153
2021-03-07 17:10:11,979 DEBUG c-0000000000 << stream 0 flow control -382 -> 2147483265
2021-03-07 17:10:11,979 DEBUG ex-0000000001 consume response data, len 382 bytes
2021-03-07 17:10:11,979 DEBUG ex-0000000001 end of response data
Shutting down
/httpbin/post->200
{
"args": {},
"data": "stuff",
"files": {},
"form": {},
"headers": {
"Content-Type": "text/plain; charset=ISO-8859-1",
"Host": "nghttp2.org",
"Transfer-Encoding": "chunked",
"User-Agent": "Apache-HttpAsyncClient/5.1-beta2-SNAPSHOT (Java/1.8.0_282)"
},
"json": null,
"origin": "213.55.225.170",
"url": "https://nghttp2.org/httpbin/post"
}
2021-03-07 17:10:11,980 DEBUG ex-0000000001 message exchange successfully completed
2021-03-07 17:10:11,980 DEBUG ep-0000000000 releasing valid endpoint
2021-03-07 17:10:11,980 DEBUG Shutdown GRACEFUL
2021-03-07 17:10:11,981 DEBUG ep-0000000000 releasing endpoint
2021-03-07 17:10:11,981 DEBUG ep-0000000000 connection c-0000000000 can be kept alive for 3 MINUTES
2021-03-07 17:10:11,982 DEBUG ep-0000000000 connection released [route: {s}->https://nghttp2.org:443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 25]
2021-03-07 17:10:11,982 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][698][631][0] Event set [w]
2021-03-07 17:10:11,982 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][698][631][0] 0 bytes read
2021-03-07 17:10:11,982 DEBUG c-0000000000[ACTIVE][r:r][ACTIVE][r][NOT_HANDSHAKING][0][0][0] Event cleared [w]
2021-03-07 17:10:11,983 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][0][0][0] Enqueued ShutdownCommand with priority IMMEDIATE
2021-03-07 17:10:11,983 DEBUG c-0000000000[ACTIVE][r:w][ACTIVE][r][NOT_HANDSHAKING][0][0][0] Event cleared [w]
2021-03-07 17:10:11,984 DEBUG c-0000000000 >> stream 0 frame: GOAWAY (0x7); flags: (0x0); length: 25
2021-03-07 17:10:11,985 DEBUG c-0000000000[ACTIVE][r:w][ACTIVE][r][NOT_HANDSHAKING][0][0][63] 34 bytes written
2021-03-07 17:10:11,985 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][63] Event set [w]
2021-03-07 17:10:11,985 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][63] Close GRACEFUL
2021-03-07 17:10:12,275 DEBUG Shutdown connection pool GRACEFUL
2021-03-07 17:10:12,275 DEBUG c-0000000000 Shutdown connection GRACEFUL
2021-03-07 17:10:12,275 DEBUG Connection pool shut down

Stream using libstreaming to VLC 2.2.4 exception

I setup libstreaming to stream video from Android camera to VLC 2.2.4 on my macOS.
SharedPreferences.Editor editor = PreferenceManager.getDefaultSharedPreferences(context).edit();
editor.putString(RtspServer.KEY_PORT, String.valueOf(1777));
editor.apply();
SessionBuilder.getInstance()
.setContext(context)
.setCallback(this)
.setAudioEncoder(SessionBuilder.AUDIO_NONE)
.setVideoEncoder(SessionBuilder.VIDEO_H264)
.setVideoQuality(new VideoQuality(640, 480, 15, 500000));
context.startService(new Intent(context, RtspServer.class));
Then I builded url bellow and opened it in VLC 2.2.4:
rtsp://172.xx.xx.67:1777?h264=500000-15-640-480
Then I received some logs on Android side:
OPTIONS
RTSP/1.0 200 OK
Server: MajorKernelPanic RTSP Server
Cseq: 2
Content-Length: 0
DESCRIBE
SPS: Z0KAHtoCgPaAbQoTUA==
PPS: aM4G4g==
RTSP/1.0 200 OK
Server: MajorKernelPanic RTSP Server
Cseq: 3
Content-Length: 272
Content-Base: 172.xx.xx.53:1777/
Content-Type: application/sdp
v=0
o=- 0 0 IN IP4 172.xx.xx.53
s=Unnamed
i=N/A
c=IN IP4 172.xx.xx.26
t=0 0
a=recvonly
m=video 5006 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=42801e;sprop-parameter-sets=Z0KAHtoCgPaAbQoTUA==,aM4G4g==;
a=control:trackID=1
SETUP
PPS: aM4G4g==
SPS: Z0KAHtoCgPaAbQoTUA==
RTSP/1.0 200
Server: MajorKernelPanic RTSP
Cseq: 4
Content-Length: 0
Transport: RTP/AVP/UDP;unicast;destination=172.xx.xx.26;client_port=5006-5007;server_port=42104-42497;ssrc=68db94da;mode=play
Session: 1185d20035702ca
Cache-Control: no-cache
PLAY
RTSP/1.0 200 OK
Server: MajorKernelPanic RTSP Server
Cseq: 5
Content-Length: 0
RTP-Info: url=rtsp://172.xx.xx.53:1777/trackID=1;seq=0
Session: 1185d20035702ca
TEARDOWN
RTSP/1.0 200 OK
Server: MajorKernelPanic RTSP Server
Cseq: 6
Content-Length: 0
java.lang.IllegalStateException: No successful match so far
at java.util.regex.Matcher.ensureMatch(Matcher.java:471)
at java.util.regex.Matcher.group(Matcher.java:579)
at net.majorkernelpanic.streaming.rtsp.RtspServer$Request.parseRequest(RtspServer.java:643)
at net.majorkernelpanic.streaming.rtsp.RtspServer$WorkerThread.run(RtspServer.java:393)
E/RtspServer:Error parsing CSeq: Attempt to read from field 'java.util.HashMap net.majorkernelpanic.streaming.rtsp.RtspServer$Request.headers' on a null object reference
RTSP/1.0 400 Bad Request
Server: MajorKernelPanic RTSP Server
Content-Length: 0
Or shortly:
OPTIONS 200 -> DESCRIBE 200 -> SETUP 200 -> PLAY 200 -> TEARDOWN 200
-> Error parsing CSeq (IllegalStateException: No successful match so far)
- no method in the request
After successful connect (all steps above) I get exception in RtspServer.
As result I can't play the stream.
Here is the log for official example spydroid-ipcamera (connecting from VLC 2.2.4):
!!! It works on VLC 1.1.5:

Why Apache HttpClient 4.5.2 is does not timeout on non responsive http proxy but block on stream read insteed?

In short: I want to test some http proxies. I have run into a single proxy that seems to be unresponsive, however HttpClient is does not timeout like it should. Stacktrace shows that app is blocked on read method. It should timeout IMHO but it is not. In general code is tested and works fine in plenty of other cases (including timeouts, exceptions etc.)
How I configure my request (yes, I use those on request execution):
requestConfig = RequestConfig.custom()
.setSocketTimeout(2000)
.setConnectionRequestTimeout(1000)
.setConnectTimeout(5000);
RequestConfig config = requestConfig.setProxy(proxyHost).build();
context.setRequestConfig(config);
What HttpClient is doing - request is sent but no response:
09:55:06.719 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default
09:55:06.720 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
09:55:06.721 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {tls}->http://84.28.86.73:80->https://api.ipify.org:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
09:55:06.722 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 19][route: {tls}->http://84.28.86.73:80->https://api.ipify.org:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
09:55:06.723 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Opening connection {tls}->http://84.28.86.73:80->https://api.ipify.org:443
09:55:06.723 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to /84.28.86.73:80
09:55:06.786 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connection established 192.168.2.144:31914<->84.28.86.73:80
09:55:06.787 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.headers - http-outgoing-19 >> CONNECT api.ipify.org:443 HTTP/1.1
09:55:06.787 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.headers - http-outgoing-19 >> Host: api.ipify.org
09:55:06.788 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.headers - http-outgoing-19 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_66)
09:55:06.788 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.wire - http-outgoing-19 >> "CONNECT api.ipify.org:443 HTTP/1.1[\r][\n]"
09:55:06.788 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.wire - http-outgoing-19 >> "Host: api.ipify.org[\r][\n]"
09:55:06.789 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.wire - http-outgoing-19 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_66)[\r][\n]"
09:55:06.789 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.wire - http-outgoing-19 >> "[\r][\n]"
Here is the stackrace where connection is blocked
Thread [ProxyPool-ScheduledWorkers-pool-3-thread-1] (Suspended)
SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) line: not available [native method]
SocketInputStream.socketRead(FileDescriptor, byte[], int, int, int) line: not available
SocketInputStream.read(byte[], int, int, int) line: not available
SocketInputStream.read(byte[], int, int) line: not available
LoggingInputStream.read(byte[], int, int) line: 87
SessionInputBufferImpl.streamRead(byte[], int, int) line: 139
SessionInputBufferImpl.fillBuffer() line: 155
SessionInputBufferImpl.readLine(CharArrayBuffer) line: 284
DefaultHttpResponseParser.parseHead(SessionInputBuffer) line: 140
DefaultHttpResponseParser.parseHead(SessionInputBuffer) line: 57
DefaultHttpResponseParser(AbstractMessageParser<T>).parse() line: 261
LoggingManagedHttpClientConnection(DefaultBHttpClientConnection).receiveResponseHeader() line: 165
CPoolProxy.receiveResponseHeader() line: 167
HttpRequestExecutor.doReceiveResponse(HttpRequest, HttpClientConnection, HttpContext) line: 272
HttpRequestExecutor.execute(HttpRequest, HttpClientConnection, HttpContext) line: 124
MainClientExec.createTunnelToTarget(AuthState, HttpClientConnection, HttpRoute, HttpRequest, HttpClientContext) line: 472
MainClientExec.establishRoute(AuthState, HttpClientConnection, HttpRoute, HttpRequest, HttpClientContext) line: 397
MainClientExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, HttpExecutionAware) line: 236
ProtocolExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, HttpExecutionAware) line: 184
RedirectExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, HttpExecutionAware) line: 110
InternalHttpClient.doExecute(HttpHost, HttpRequest, HttpContext) line: 184
InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest, HttpContext) line: 82
InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest, HttpContext) line: 55
Ipify.getExternalIp(HttpClient, RequestConfig, HttpContext) line: 48
Ipify.getExternalIp(HttpClient, HttpContext) line: 33
<obsolete method in<unknown declaring type>>
ProxyTester.isHttpsProxyValidQuiet(HttpsProxyHost) line: 105
FetchProxiesFromSourceTask.run() line: 72
Executors$RunnableAdapter<T>.call() line: not available
ScheduledThreadPoolExecutor$ScheduledFutureTask<V>(FutureTask<V>).runAndReset() line: not available
ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.access$301(ScheduledThreadPoolExecutor$ScheduledFutureTask) line: not available
ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.run() line: not available
ScheduledThreadPoolExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker) line: not available
ThreadPoolExecutor$Worker.run() line: not available
Thread.run() line: not available
EDIT: Honorable mention:
setting socket timeout in both RequestConfig and SocketConfig works differently for HTTP and HTTPS
Looks like a design flaw to me as there is no way to set soTimeout prior SSL handshake on request basis.
And reported bug here: https://issues.apache.org/jira/browse/HTTPCLIENT-1478