Apache HTTP Client 5.0 HTTP/2 POST Example needed - java
I can use CURL to send a HTTP/2 POST request with a payload to the https://httpbin.org/post as in the following example:
"c:\Program Files\Git\mingw64\bin\curl" -v --http2 https://httpbin.org/post -d "arg1=param1&arg2=param2"
* Trying 54.166.163.67:443...
* Connected to httpbin.org (54.166.163.67) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: c:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=httpbin.org
* start date: Dec 21 00:00:00 2020 GMT
* expire date: Jan 19 23:59:59 2022 GMT
* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x160f6b61d20)
> POST /post HTTP/2
> Host: httpbin.org
> user-agent: curl/7.75.0
> accept: */*
> content-length: 23
> content-type: application/x-www-form-urlencoded
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* We are completely uploaded and fine
< HTTP/2 200
< date: Fri, 05 Mar 2021 16:07:51 GMT
< content-type: application/json
< content-length: 453
< server: gunicorn/19.9.0
< access-control-allow-origin: *
< access-control-allow-credentials: true
<
{
"args": {},
"data": "",
"files": {},
"form": {
"arg1": "param1",
"arg2": "param2"
},
"headers": {
"Accept": "*/*",
"Content-Length": "23",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "curl/7.75.0",
"X-Amzn-Trace-Id": "Root=1-60425757-45ab281925b9b40e70398269"
},
"json": null,
"origin": "51.37.211.242",
"url": "https://httpbin.org/post"
}
* Connection #0 to host httpbin.org left intact
You can see it sends the arg1/arg2 parameters and they are in the echoed back response.
I would like to replicate that code in Java. I have had no luck myself doing it and cannot find an example of a HTTP/2 POST with a payload. There are some examples of GET/POST calls using HTTP/2 without a payload, and indeed I can write the code to make a request without a payload that works, but I cannot get the payload to send. If I use the same code but using HTTP/1 the payload is sent. Using BasicRequestProducer class you can give it producers (to send the request data I hope) and consumers (to consume the response) but the producer never seems to send the data.
See my other question on HTTP/2 Connection Closed for the sort of code I am using.
I have added some DEBUG to my examples and I can see the content/body being added to the request (or at least it looks to me like it is being added) so is there an issue with the correct formatting of the message?
188 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - Frame 1 >>[type=0, flags=0, streamId=1, payoad=java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]]
188 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - Frame info : Frame 1 >> [type=0, flags=0, streamId=1, payoad=java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]]
189 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - stream 1 frame: DATA (0x0); flags: (0x0); length: 23
189 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - Frame Payload : Frame 1 >> java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]
189 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - Code=99&Message= 43 6f 64 65 3d 39 39 26 4d 65 73 73 61 67 65 3d
Goodbye 47 6f 6f 64 62 79 65
189 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - OutputFlow 0 >>0 -23 65512
190 [requester-dispatch-1] DEBUG com.test.HTTP2Tester - OutputFlow 1 >>1 -23 65512
Can anyone give Java code to demonstrate a successful HTTP/2 POST with a payload????
This is the most simple form of a POST request that works the same way with HTTP/1.1 or HTTP/2 transports
final IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
.setSoTimeout(Timeout.ofSeconds(5))
.build();
final CloseableHttpAsyncClient client = HttpAsyncClients.custom()
.setIOReactorConfig(ioReactorConfig)
.build();
client.start();
final HttpHost target = new HttpHost("nghttp2.org");
final String requestUri = "/httpbin/post";
final AsyncRequestProducer requestProducer = new BasicRequestProducer(Method.POST, target, requestUri,
AsyncEntityProducers.create("stuff", ContentType.TEXT_PLAIN));
System.out.println("Executing POST request to " + requestUri);
final Future<Message<HttpResponse, String>> future = client.execute(
requestProducer,
new BasicResponseConsumer<String>(new StringAsyncEntityConsumer()),
new FutureCallback<Message<HttpResponse, String>>() {
#Override
public void completed(final Message<HttpResponse, String> message) {
System.out.println(requestUri + "->" + message.getHead().getCode());
System.out.println(message.getBody());
}
#Override
public void failed(final Exception ex) {
System.out.println(requestUri + "->" + ex);
}
#Override
public void cancelled() {
System.out.println(requestUri + " cancelled");
}
});
future.get();
System.out.println("Shutting down");
client.close(CloseMode.GRACEFUL);
Here's application output including the header / context log of the session
Executing POST request to /httpbin/post
2021-03-07 17:10:10,379 DEBUG ex-0000000001 preparing request execution
2021-03-07 17:10:10,390 DEBUG ex-0000000001 Cookie spec selected: strict
2021-03-07 17:10:10,396 DEBUG ex-0000000001 Auth cache not set in the context
2021-03-07 17:10:10,396 DEBUG ex-0000000001 target auth state: UNCHALLENGED
2021-03-07 17:10:10,397 DEBUG ex-0000000001 proxy auth state: UNCHALLENGED
2021-03-07 17:10:10,399 DEBUG ex-0000000001 acquiring connection with route {s}->https://nghttp2.org:443
2021-03-07 17:10:10,399 DEBUG ex-0000000001 acquiring endpoint (3 MINUTES)
2021-03-07 17:10:10,401 DEBUG ex-0000000001 endpoint lease request (3 MINUTES) [route: {s}->https://nghttp2.org:443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 25]
2021-03-07 17:10:10,406 DEBUG ex-0000000001 endpoint leased [route: {s}->https://nghttp2.org:443][total available: 0; route allocated: 1 of 5; total allocated: 1 of 25]
2021-03-07 17:10:10,406 DEBUG ex-0000000001 acquired ep-0000000000
2021-03-07 17:10:10,407 DEBUG ex-0000000001 acquired endpoint ep-0000000000
2021-03-07 17:10:10,407 DEBUG ep-0000000000 connecting endpoint (3 MINUTES)
2021-03-07 17:10:10,408 DEBUG ep-0000000000 connecting endpoint to https://nghttp2.org:443 (3 MINUTES)
2021-03-07 17:10:10,408 DEBUG https://nghttp2.org:443 resolving remote address
2021-03-07 17:10:10,410 DEBUG https://nghttp2.org:443 resolved to [nghttp2.org/139.162.123.134, nghttp2.org/2400:8902:0:0:f03c:91ff:fe69:a454]
2021-03-07 17:10:10,410 DEBUG https://nghttp2.org:443 connecting null to nghttp2.org/139.162.123.134:443 (3 MINUTES)
2021-03-07 17:10:10,722 DEBUG c-0000000000[ACTIVE][rc:c] protocol upgrade class org.apache.hc.core5.http2.impl.nio.ClientHttpProtocolNegotiator
2021-03-07 17:10:10,723 DEBUG https://nghttp2.org:443 connected c-0000000000 /192.168.8.105:55230->nghttp2.org/139.162.123.134:443
2021-03-07 17:10:10,725 DEBUG c-0000000000 start TLS
2021-03-07 17:10:10,769 DEBUG ep-0000000000 connected c-0000000000
2021-03-07 17:10:10,769 DEBUG ep-0000000000 endpoint connected
2021-03-07 17:10:10,769 DEBUG ex-0000000001 connected to target
2021-03-07 17:10:10,770 DEBUG ex-0000000001 route fully established
2021-03-07 17:10:10,770 DEBUG ex-0000000001 executing POST /httpbin/post HTTP/1.1
2021-03-07 17:10:10,771 DEBUG ep-0000000000 start execution ex-0000000001
2021-03-07 17:10:10,771 DEBUG ep-0000000000 executing exchange ex-0000000001 over c-0000000000
2021-03-07 17:10:10,772 DEBUG c-0000000000 RequestExecutionCommand with NORMAL priority
2021-03-07 17:10:10,773 DEBUG c-0000000000[ACTIVE][rwc:c][ACTIVE][rw][NOT_HANDSHAKING][0][0][0] Enqueued RequestExecutionCommand with priority IMMEDIATE
2021-03-07 17:10:10,773 DEBUG c-0000000000[ACTIVE][rw:c][ACTIVE][rw][NOT_HANDSHAKING][0][0][0] Event cleared [c]
2021-03-07 17:10:10,775 DEBUG Enabled protocols: [TLSv1.2]
2021-03-07 17:10:10,775 DEBUG Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2021-03-07 17:10:11,563 DEBUG Secure session established
2021-03-07 17:10:11,563 DEBUG negotiated protocol: TLSv1.2
2021-03-07 17:10:11,564 DEBUG negotiated cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
2021-03-07 17:10:11,565 DEBUG peer principal: CN=nghttp2.org
2021-03-07 17:10:11,565 DEBUG peer alternative names: [nghttp2.org, www.nghttp2.org]
2021-03-07 17:10:11,566 DEBUG issuer principal: CN=R3, O=Let's Encrypt, C=US
2021-03-07 17:10:11,573 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][0][0] Event set [w]
2021-03-07 17:10:11,575 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][0][53] 24 bytes written
2021-03-07 17:10:11,575 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][r][NOT_HANDSHAKING][62][0][53] Event cleared [w]
2021-03-07 17:10:11,606 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][r][NOT_HANDSHAKING][62][0][53] protocol upgrade class org.apache.hc.core5.http2.impl.nio.ClientH2IOEventHandler
2021-03-07 17:10:11,608 DEBUG c-0000000000 >> stream 0 frame: SETTINGS (0x4); flags: (0x0); length: 36
2021-03-07 17:10:11,609 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][r][NOT_HANDSHAKING][62][0][53] 0 bytes written
2021-03-07 17:10:11,609 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][0][53] Event set [w]
2021-03-07 17:10:11,609 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][0][53] Event set [w]
2021-03-07 17:10:11,609 DEBUG c-0000000000 << stream 0 flow control 2147418112 -> 2147483647
2021-03-07 17:10:11,609 DEBUG c-0000000000 << stream 0 flow control 2147483647 -> 2147483647
2021-03-07 17:10:11,609 DEBUG c-0000000000 >> stream 0 flow control 65535 -> 65535
2021-03-07 17:10:11,610 DEBUG c-0000000000 << stream 0 frame: SETTINGS (0x4); flags: (0x0); length: 24
2021-03-07 17:10:11,611 DEBUG c-0000000000 >> stream 0 flow control 983041 -> 1048576
2021-03-07 17:10:11,611 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][33][53] Event set [w]
2021-03-07 17:10:11,611 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][62][33][53] 0 bytes read
2021-03-07 17:10:11,611 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][0][0][53] 0 bytes written
2021-03-07 17:10:11,614 DEBUG c-0000000000 << stream 1 flow control 65535 -> 65535
2021-03-07 17:10:11,614 DEBUG c-0000000000 >> stream 1 flow control 1048576 -> 1048576
2021-03-07 17:10:11,615 DEBUG ex-0000000001 send request POST /httpbin/post HTTP/1.1, entity len 5
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> :method: POST
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> :scheme: https
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> :authority: nghttp2.org
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> :path: /httpbin/post
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> user-agent: Apache-HttpAsyncClient/5.1-beta2-SNAPSHOT (Java/1.8.0_282)
2021-03-07 17:10:11,617 DEBUG c-0000000000 >> content-type: text/plain; charset=ISO-8859-1
2021-03-07 17:10:11,620 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][0][0][53] Event set [w]
2021-03-07 17:10:11,620 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,620 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,621 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][74] 45 bytes written
2021-03-07 17:10:11,621 DEBUG c-0000000000 >> stream 0 frame: WINDOW_UPDATE (0x8); flags: (0x0); length: 4
2021-03-07 17:10:11,621 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][74] 0 bytes written
2021-03-07 17:10:11,621 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,621 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,622 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][42] 13 bytes written
2021-03-07 17:10:11,622 DEBUG c-0000000000 >> stream 0 frame: SETTINGS (0x4); flags: ACK (0x1); length: 0
2021-03-07 17:10:11,622 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][42] 0 bytes written
2021-03-07 17:10:11,622 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,622 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,622 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][38] 9 bytes written
2021-03-07 17:10:11,623 DEBUG c-0000000000 >> stream 1 frame: HEADERS (0x1); flags: END_HEADERS (0x4); length: 93
2021-03-07 17:10:11,623 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][38] 0 bytes written
2021-03-07 17:10:11,623 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,623 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,623 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][131] 102 bytes written
2021-03-07 17:10:11,623 DEBUG ex-0000000001 produce request data
2021-03-07 17:10:11,623 DEBUG ex-0000000001 produce request data, len 5 bytes
2021-03-07 17:10:11,623 DEBUG c-0000000000 >> stream 1 frame: DATA (0x0); flags: (0x0); length: 5
2021-03-07 17:10:11,624 DEBUG c-0000000000 >> stream 0 flow control -5 -> 1048571
2021-03-07 17:10:11,624 DEBUG c-0000000000 >> stream 1 flow control -5 -> 1048571
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][131] 0 bytes written
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][131] Event set [w]
2021-03-07 17:10:11,624 DEBUG ex-0000000001 end of request data
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][131] Event set [w]
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][43] 14 bytes written
2021-03-07 17:10:11,624 DEBUG c-0000000000 >> stream 1 frame: DATA (0x0); flags: END_STREAM (0x1); length: 0
2021-03-07 17:10:11,624 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][43] 0 bytes written
2021-03-07 17:10:11,625 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][38] 9 bytes written
2021-03-07 17:10:11,625 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][r][NOT_HANDSHAKING][0][0][38] Event cleared [w]
2021-03-07 17:10:11,968 DEBUG c-0000000000 << stream 0 frame: SETTINGS (0x4); flags: ACK (0x1); length: 0
2021-03-07 17:10:11,969 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][38][9][0] Event set [w]
2021-03-07 17:10:11,969 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][38][9][0] 0 bytes read
2021-03-07 17:10:11,970 DEBUG c-0000000000 << stream 1 frame: HEADERS (0x1); flags: END_HEADERS (0x4); length: 231
2021-03-07 17:10:11,971 DEBUG c-0000000000 << :status: 200
2021-03-07 17:10:11,971 DEBUG c-0000000000 << date: Sun, 07 Mar 2021 16:10:11 GMT
2021-03-07 17:10:11,971 DEBUG c-0000000000 << content-type: application/json
2021-03-07 17:10:11,971 DEBUG c-0000000000 << content-length: 382
2021-03-07 17:10:11,971 DEBUG c-0000000000 << access-control-allow-origin: *
2021-03-07 17:10:11,971 DEBUG c-0000000000 << access-control-allow-credentials: true
2021-03-07 17:10:11,971 DEBUG c-0000000000 << x-backend-header-rtt: 0.003465
2021-03-07 17:10:11,971 DEBUG c-0000000000 << strict-transport-security: max-age=31536000
2021-03-07 17:10:11,971 DEBUG c-0000000000 << server: nghttpx
2021-03-07 17:10:11,971 DEBUG c-0000000000 << via: 1.1 nghttpx
2021-03-07 17:10:11,971 DEBUG c-0000000000 << alt-svc: h3-29=":443"; ma=3600
2021-03-07 17:10:11,972 DEBUG c-0000000000 << x-frame-options: SAMEORIGIN
2021-03-07 17:10:11,972 DEBUG c-0000000000 << x-xss-protection: 1; mode=block
2021-03-07 17:10:11,972 DEBUG c-0000000000 << x-content-type-options: nosniff
2021-03-07 17:10:11,975 DEBUG ex-0000000001 consume response HTTP/2.0 200 OK, entity len -1
2021-03-07 17:10:11,978 DEBUG c-0000000000 << stream 1 frame: DATA (0x0); flags: END_STREAM (0x1); length: 382
2021-03-07 17:10:11,979 DEBUG c-0000000000 << stream 1 flow control -382 -> 65153
2021-03-07 17:10:11,979 DEBUG c-0000000000 << stream 0 flow control -382 -> 2147483265
2021-03-07 17:10:11,979 DEBUG ex-0000000001 consume response data, len 382 bytes
2021-03-07 17:10:11,979 DEBUG ex-0000000001 end of response data
Shutting down
/httpbin/post->200
{
"args": {},
"data": "stuff",
"files": {},
"form": {},
"headers": {
"Content-Type": "text/plain; charset=ISO-8859-1",
"Host": "nghttp2.org",
"Transfer-Encoding": "chunked",
"User-Agent": "Apache-HttpAsyncClient/5.1-beta2-SNAPSHOT (Java/1.8.0_282)"
},
"json": null,
"origin": "213.55.225.170",
"url": "https://nghttp2.org/httpbin/post"
}
2021-03-07 17:10:11,980 DEBUG ex-0000000001 message exchange successfully completed
2021-03-07 17:10:11,980 DEBUG ep-0000000000 releasing valid endpoint
2021-03-07 17:10:11,980 DEBUG Shutdown GRACEFUL
2021-03-07 17:10:11,981 DEBUG ep-0000000000 releasing endpoint
2021-03-07 17:10:11,981 DEBUG ep-0000000000 connection c-0000000000 can be kept alive for 3 MINUTES
2021-03-07 17:10:11,982 DEBUG ep-0000000000 connection released [route: {s}->https://nghttp2.org:443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 25]
2021-03-07 17:10:11,982 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][698][631][0] Event set [w]
2021-03-07 17:10:11,982 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][698][631][0] 0 bytes read
2021-03-07 17:10:11,982 DEBUG c-0000000000[ACTIVE][r:r][ACTIVE][r][NOT_HANDSHAKING][0][0][0] Event cleared [w]
2021-03-07 17:10:11,983 DEBUG c-0000000000[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][0][0][0] Enqueued ShutdownCommand with priority IMMEDIATE
2021-03-07 17:10:11,983 DEBUG c-0000000000[ACTIVE][r:w][ACTIVE][r][NOT_HANDSHAKING][0][0][0] Event cleared [w]
2021-03-07 17:10:11,984 DEBUG c-0000000000 >> stream 0 frame: GOAWAY (0x7); flags: (0x0); length: 25
2021-03-07 17:10:11,985 DEBUG c-0000000000[ACTIVE][r:w][ACTIVE][r][NOT_HANDSHAKING][0][0][63] 34 bytes written
2021-03-07 17:10:11,985 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][63] Event set [w]
2021-03-07 17:10:11,985 DEBUG c-0000000000[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][63] Close GRACEFUL
2021-03-07 17:10:12,275 DEBUG Shutdown connection pool GRACEFUL
2021-03-07 17:10:12,275 DEBUG c-0000000000 Shutdown connection GRACEFUL
2021-03-07 17:10:12,275 DEBUG Connection pool shut down
Related
Reading HTTP headers as they become available
I currently have the need to read HTTP headers as they are sent. I have tried the new Java 11 HTTP client (java.net.http.HttpClient), but I have no luck intercepting the headers, as they are not available until the content is starting to be sent (so all headers are returned at once).
I need this functionality to handle progress/status updates as the server sends multiple headers with the progress of the operation, and then the response is sent normally once ready. This is not my design (the response is sent from ClickHouse), so I just need to be able to find an HTTP client that can read it.
The curl command line works just fine:
curl -vsS "http://localhost:8123?send_progress_in_http_headers=1&wait_end_of_query=1" -d "select count() from numbers(100000000000) format JSON;"
* Trying 127.0.0.1:8123...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8123 (#0)
> POST /?send_progress_in_http_headers=1&wait_end_of_query=1 HTTP/1.1
> Host: localhost:8123
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 54
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 54 out of 54 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sat, 20 Feb 2021 11:35:14 GMT
< Connection: Keep-Alive
< Content-Type: application/json; charset=UTF-8
< X-ClickHouse-Server-Display-Name: 78b40ec2d4a2
< Transfer-Encoding: chunked
< X-ClickHouse-Query-Id: 19bd4955-09ad-47f2-a64f-93d926383f55
< X-ClickHouse-Format: JSON
< X-ClickHouse-Timezone: UTC
< Keep-Alive: timeout=3
< X-ClickHouse-Progress: {"read_rows":"100794368","read_bytes":"806354944","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"249954304","read_bytes":"1999634432","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"414515200","read_bytes":"3316121600","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"558759936","read_bytes":"4470079488","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"722206720","read_bytes":"5777653760","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"901185536","read_bytes":"7209484288","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1110573056","read_bytes":"8884584448","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1323302912","read_bytes":"10586423296","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1534722048","read_bytes":"12277776384","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1729953792","read_bytes":"13839630336","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1943535616","read_bytes":"15548284928","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"2165768192","read_bytes":"17326145536","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"2382692352","read_bytes":"19061538816","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"2597519360","read_bytes":"20780154880","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"2807758848","read_bytes":"22462070784","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"3021733888","read_bytes":"24173871104","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"3234922496","read_bytes":"25879379968","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"3457875968","read_bytes":"27663007744","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"3663790080","read_bytes":"29310320640","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
I can see from the debug logs for the client that the data is read and parsed incrementally, but there is no way to access the information that I'm aware of:
...
2021-02-21 10:31:03,070 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [747ms] SelectorAttachment Registering jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent#7e37cce4 for 0 (false)
2021-02-21 10:31:03,072 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [749ms] SocketTube(1) read bytes: 490
2021-02-21 10:31:03,072 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [750ms] Http1AsyncReceiver(SocketTube(1)) Putting 490 bytes into the queue
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [750ms] SocketTube(1) resuming read event
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-Worker-0] [jdk.internal.httpclient.debug:286] - [HttpClient-1-Worker-0] [750ms] Http1AsyncReceiver(SocketTube(1)) Got 490 bytes for delegate jdk.internal.net.http.Http1Response$HeadersReader#2bcbc994
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [750ms] SelectorAttachment Registering jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent#7e37cce4 for 1 (false)
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-Worker-0] [jdk.internal.httpclient.debug:286] - [HttpClient-1-Worker-0] [750ms] Http1AsyncReceiver(SocketTube(1)) downstream subscription demand is 1
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [750ms] SocketTube(1) leaving read() loop after onNext: Reading: [ops=1, demand=0, stopped=false], Writing: [ops=0, demand=1]
....
Any ideas on how the information can be extracted as it is becoming available? There seem to be multiple async HTTP clients, but they are not having a callback for the headers.
Edit: I shared my project here: https://github.com/ethlo/clackshack
The issue in question would have to be handled here: https://github.com/ethlo/clackshack/blob/main/src/main/java/com/ethlo/clackshack/Java11Client.java
The project contains a test with a simple HTTP server, responding with HTTP headers with 500 ms delay, to avoid the dependency on ClickHouse for reproducing.
pact-jvm-server Complete returns 400
I've got a pact-jvm-server instance in Docker and when ever I call the /create method it creates a stub server and returns the correct responses. But now when I try to delete the stub server on the selected port with /complete call and port in body of the POST request it always returns 400 even though it deletes the stub service on provided port.
Any help would be appreciated.
Docker logs:
b51ce9 starting unfiltered app at 0.0.0.0 on port 8080
08:26:44.847 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
08:26:44.853 [main] DEBUG i.n.c.MultithreadEventLoopGroup - -
Dio.netty.eventLoopThreads: 408:26:44.860 [main] DEBUG i.n.util.internal.PlatformDependent0 -
java.nio.Buffer.address: available
08:26:44.860 [main] DEBUG i.n.util.internal.PlatformDependent0 sun.misc.Unsafe.theUnsafe: available
08:26:44.861 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
08:26:44.861 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true
08:26:44.913 [main] DEBUG i.n.util.internal.PlatformDependent - UID: 0
08:26:44.914 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
08:26:44.914 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
08:26:44.914 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
08:26:44.914 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
08:26:44.916 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
08:26:44.916 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
08:26:44.916 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
08:26:44.917 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
08:26:44.917 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
08:26:44.938 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
08:26:44.938 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
08:26:44.984 [main] DEBUG i.n.util.internal.ThreadLocalRandom - -Dio.netty.initialSeedUniquifier: 0x1f1143a58dddfa74 (took 20 ms)
08:26:45.006 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: unpooled
08:26:45.006 [main] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 65536
08:26:45.009 [main] DEBUG io.netty.util.NetUtil - Loopback interface: lo (lo, 127.0.0.1)
08:26:45.009 [main] DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 128
08:26:46.383 [nioEventLoopGroup-3-1] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetectionLevel: simple
08:26:46.405 [nioEventLoopGroup-3-1] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacity.default: 262144
08:26:46.409 [nioEventLoopGroup-3-1] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
08:27:08.830 [nioEventLoopGroup-3-2] DEBUG au.com.dius.pact.server.Create$ - path=/create
08:27:08.830 [nioEventLoopGroup-3-2] DEBUG au.com.dius.pact.server.Create$ - query={state=[HarryPotter]}
08:27:08.871 [nioEventLoopGroup-3-2] DEBUG au.com.dius.pact.server.Create$ - au.com.dius.pact.model.OptionalBody(PRESENT, {
"provider": {
"name": "microservices-simple-catalogue"
},
"consumer": {
"name": "microservices-simple-orders"
},
"interactions": [
{
"description": "catalogue test interaction",
"request": {
"method": "GET",
"path": "/books/1"
},
"response": {
"status": 200,
"headers": {
"Content-Type": "application/json"
},
"body": {
"bookOrders": [
],
"author": "J.K. Rowling",
"description": "Books about a wizard",
"id": 1,
"title": "Harry Potter"
},
"matchingRules": {
"body": {
"$.bookOrders": {
"matchers": [
{
"match": "type",
"min": 0
}
],
"combine": "AND"
}
}
}
},
"providerStates": [
{
"name": "test state for returning Harry Potter book"
}
]
}
],
"metadata": {
"pact-specification": {
"version": "3.0.0"
},
"pact-jvm": {
"version": "3.5.1"
}
}
})
If I call it over the curl:
$ curl -v -d '{"port":20004}' localhost:8080/complete
* timeout on name lookup is not supported
* Trying ::1...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (::1) port 8080 (#0)
> POST /complete HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.51.0
> Accept: */*
> Content-Length: 14
> Content-Type: application/x-www-form-urlencoded
>
} [14 bytes data]
* upload completely sent off: 14 out of 14 bytes
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Connection: keep-alive
< Content-Length: 613
<
{ [613 bytes data]
* Curl_http_done: called premature == 0
100 627 100 613 100 14 40866 933 --:--:-- --:--:-- --:--:-- 598k{"error": "Pact verification failed for the following reasons:
The following requests were not received:
Interaction: catalogue test interaction
in state None
request:
method: GET
path: /books/1
query: null
headers: null
matchers: null
body: au.com.dius.pact.model.OptionalBody(MISSING, null)
response:
status: 200
headers: [Content-Type:application/json]
matchers: [body:[$.bookOrders:[combine:AND, matchers:[[match:type, min:0]]]]]
body: au.com.dius.pact.model.OptionalBody(PRESENT, {"author":"J.K. Rowling","bookOrders":[],"description":"Books about a wizard","id":1,"title":"Harry Potter"})"}
* Connection #0 to host localhost left intact
Why Apache HttpClient 4.5.2 is does not timeout on non responsive http proxy but block on stream read insteed?
In short: I want to test some http proxies. I have run into a single proxy that seems to be unresponsive, however HttpClient is does not timeout like it should. Stacktrace shows that app is blocked on read method. It should timeout IMHO but it is not. In general code is tested and works fine in plenty of other cases (including timeouts, exceptions etc.)
How I configure my request (yes, I use those on request execution):
requestConfig = RequestConfig.custom()
.setSocketTimeout(2000)
.setConnectionRequestTimeout(1000)
.setConnectTimeout(5000);
RequestConfig config = requestConfig.setProxy(proxyHost).build();
context.setRequestConfig(config);
What HttpClient is doing - request is sent but no response:
09:55:06.719 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default
09:55:06.720 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
09:55:06.721 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {tls}->http://84.28.86.73:80->https://api.ipify.org:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
09:55:06.722 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 19][route: {tls}->http://84.28.86.73:80->https://api.ipify.org:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
09:55:06.723 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Opening connection {tls}->http://84.28.86.73:80->https://api.ipify.org:443
09:55:06.723 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to /84.28.86.73:80
09:55:06.786 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connection established 192.168.2.144:31914<->84.28.86.73:80
09:55:06.787 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.headers - http-outgoing-19 >> CONNECT api.ipify.org:443 HTTP/1.1
09:55:06.787 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.headers - http-outgoing-19 >> Host: api.ipify.org
09:55:06.788 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.headers - http-outgoing-19 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_66)
09:55:06.788 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.wire - http-outgoing-19 >> "CONNECT api.ipify.org:443 HTTP/1.1[\r][\n]"
09:55:06.788 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.wire - http-outgoing-19 >> "Host: api.ipify.org[\r][\n]"
09:55:06.789 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.wire - http-outgoing-19 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_66)[\r][\n]"
09:55:06.789 [ProxyPool-ScheduledWorkers-pool-3-thread-1] DEBUG org.apache.http.wire - http-outgoing-19 >> "[\r][\n]"
Here is the stackrace where connection is blocked
Thread [ProxyPool-ScheduledWorkers-pool-3-thread-1] (Suspended)
SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) line: not available [native method]
SocketInputStream.socketRead(FileDescriptor, byte[], int, int, int) line: not available
SocketInputStream.read(byte[], int, int, int) line: not available
SocketInputStream.read(byte[], int, int) line: not available
LoggingInputStream.read(byte[], int, int) line: 87
SessionInputBufferImpl.streamRead(byte[], int, int) line: 139
SessionInputBufferImpl.fillBuffer() line: 155
SessionInputBufferImpl.readLine(CharArrayBuffer) line: 284
DefaultHttpResponseParser.parseHead(SessionInputBuffer) line: 140
DefaultHttpResponseParser.parseHead(SessionInputBuffer) line: 57
DefaultHttpResponseParser(AbstractMessageParser<T>).parse() line: 261
LoggingManagedHttpClientConnection(DefaultBHttpClientConnection).receiveResponseHeader() line: 165
CPoolProxy.receiveResponseHeader() line: 167
HttpRequestExecutor.doReceiveResponse(HttpRequest, HttpClientConnection, HttpContext) line: 272
HttpRequestExecutor.execute(HttpRequest, HttpClientConnection, HttpContext) line: 124
MainClientExec.createTunnelToTarget(AuthState, HttpClientConnection, HttpRoute, HttpRequest, HttpClientContext) line: 472
MainClientExec.establishRoute(AuthState, HttpClientConnection, HttpRoute, HttpRequest, HttpClientContext) line: 397
MainClientExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, HttpExecutionAware) line: 236
ProtocolExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, HttpExecutionAware) line: 184
RedirectExec.execute(HttpRoute, HttpRequestWrapper, HttpClientContext, HttpExecutionAware) line: 110
InternalHttpClient.doExecute(HttpHost, HttpRequest, HttpContext) line: 184
InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest, HttpContext) line: 82
InternalHttpClient(CloseableHttpClient).execute(HttpUriRequest, HttpContext) line: 55
Ipify.getExternalIp(HttpClient, RequestConfig, HttpContext) line: 48
Ipify.getExternalIp(HttpClient, HttpContext) line: 33
<obsolete method in<unknown declaring type>>
ProxyTester.isHttpsProxyValidQuiet(HttpsProxyHost) line: 105
FetchProxiesFromSourceTask.run() line: 72
Executors$RunnableAdapter<T>.call() line: not available
ScheduledThreadPoolExecutor$ScheduledFutureTask<V>(FutureTask<V>).runAndReset() line: not available
ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.access$301(ScheduledThreadPoolExecutor$ScheduledFutureTask) line: not available
ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.run() line: not available
ScheduledThreadPoolExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker) line: not available
ThreadPoolExecutor$Worker.run() line: not available
Thread.run() line: not available
EDIT: Honorable mention:
setting socket timeout in both RequestConfig and SocketConfig works differently for HTTP and HTTPS
Looks like a design flaw to me as there is no way to set soTimeout prior SSL handshake on request basis.
And reported bug here: https://issues.apache.org/jira/browse/HTTPCLIENT-1478
OpenNMS v18 AMQP Message Sending Issue
I cannot get OpenNMS to send messages to an AMQP end point.
I have never had this working and this is the first time i have used OpenNMS and AMQP so it could be down my inexperience.
I have configured RabbitMQ 3.5.7 and tested it as per this question .
It works fine when using an external QPID 0.32 client and also works fine when using either python or perl.
Definition of works fine is that communications are established and the message payload is transmitted into an exchange and then delivered into a backend queue.
Message can then be viewed within the RabbitMQ Admin GUI.
In OpenNMS i have been following the instructs here
Started with the EventForwarder and then tried the AlarmNorthbounder both yield a NullPointerException.
I set up Karaf as follows setting the properties using these statements:-
opennms> config:edit org.opennms.features.amqp.alarmnorthbounder
opennms> propset connectionUrl amqp://simon:simon#/test?brokerlist=\'localhost:5672\'
opennms> propset destination "amqp:onms3/Simon;{'create':'always','node':{'type':'topic'} }"
opennms> propset processorName default-alarm-northbounder-processor
opennms> config:update
opennms> config:list '(service.pid=org.opennms.features.amqp.alarmnorthbounder)'
----------------------------------------------------------------
Pid: org.opennms.features.amqp.alarmnorthbounder
BundleLocation: mvn:org.opennms.features.amqp/org.opennms.features.amqp.alarm-northbounder/18.0.0
Properties:
connectionUrl = amqp://simon:simon#/test?brokerlist='localhost:5672'
destination = amqp:onms3/Simon;{'create':'always','node':{'type':'topic'} }
felix.fileinstall.filename = file:/usr/share/opennms/etc/org.opennms.features.amqp.alarmnorthbounder.cfg
processorName = default-alarm-northbounder-processor
service.pid = org.opennms.features.amqp.alarmnorthbounder
I get the following error message in the logs
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[forwardAlarm ] [forwardAlarm ] [seda://forwardAlarm ] [ 8]
[forwardAlarm ] [convertBodyTo3 ] [convertBodyTo[org.opennms.netmgt.alarmd.api.NorthboundAlarm] ] [ 0]
[forwardAlarm ] [log3 ] [log ] [ 1]
[forwardAlarm ] [bean3 ] [bean[ref:dynamicallyTrackedProcessor] ] [ 0]
[forwardAlarm ] [to3 ] [amqp:onms3/Simon;{'create':'always','node':{'type':'topic'} } ] [ 7]
Exchange
---------------------------------------------------------------------------------------------------------------------------------------
Exchange[
Id ID-ubuntu-1604-35241-1465752556843-2-60
ExchangePattern InOnly
Headers {breadcrumbId=ID-ubuntu-1604-35241-1465752556843-2-58, CamelRedelivered=false, CamelRedeliveryCounter=0}
BodyType String
Body NorthboundAlarm[id=3, uei='uei.opennms.org/generic/traps/EnterpriseDefault', nodeId=1]
]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
java.lang.NullPointerException
at org.apache.qpid.client.BasicMessageProducer_0_8.declareDestination(BasicMessageProducer_0_8.java:63)[212:org.apache.servicemix.bundles.qpid:0.28.0.1]
at org.apache.qpid.client.BasicMessageProducer.<init>(BasicMessageProducer.java:136)[212:org.apache.servicemix.bundles.qpid:0.28.0.1]
at org.apache.qpid.client.BasicMessageProducer_0_8.<init>(BasicMessageProducer_0_8.java:55)[212:org.apache.servicemix.bundles.qpid:0.28.0.1]
at org.apache.qpid.client.AMQSession_0_8.createMessageProducer(AMQSession_0_8.java:559)[212:org.apache.servicemix.bundles.qpid:0.28.0.1]
at org.apache.qpid.client.AMQSession_0_8.createMessageProducer(AMQSession_0_8.java:62)[212:org.apache.servicemix.bundles.qpid:0.28.0.1]
My expectation is that it should be able to deliver the message to the queue using the same libraries as i am using externally.
Putting DEBUG on for org.apache.qpid i get:-
2016-06-12 19:00:38,725 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.state.AMQStateManager: Notififying State change to 1 : [org.apache.qpid.client.state.StateWaiter#5d7e5d44]
2016-06-12 19:00:38,725 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.framing.FieldTable: FieldTable::writeToBuffer: Writing encoded length of 254...
2016-06-12 19:00:38,725 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.framing.FieldTable: {instance=[LONG_STRING: ubuntu-16041465757105204], product=[LONG_STRING: qpid], version=[LONG_STRING: 0.28], platform=[LONG_STRING: Java(TM) SE Runtime Environment, 1.8.0_45-b14, Oracle Corporation, amd64, Linux, 4.4.0-22-generic, unknown], qpid.client_process=[LONG_STRING: Qpid Java Client], qpid.client_pid=[INT: 1318]}
2016-06-12 19:00:38,725 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.protocol.AMQProtocolHandler: (1404676419)Method frame received: [ConnectionTuneBodyImpl: channelMax=0, frameMax=131072, heartbeat=60]
2016-06-12 19:00:38,726 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.handler.ConnectionTuneMethodHandler: ConnectionTune frame received
2016-06-12 19:00:38,726 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.state.AMQStateManager: State changing to AMQState: id = 3 name: CONNECTION_NOT_OPENED from old state AMQState: id = 2 name: CONNECTION_NOT_TUNED
2016-06-12 19:00:38,726 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.state.AMQStateManager: Notififying State change to 1 : [org.apache.qpid.client.state.StateWaiter#5d7e5d44]
2016-06-12 19:00:38,726 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.protocol.AMQProtocolHandler: (1404676419)Method frame received: [ConnectionOpenOkBodyImpl: knownHosts=null]
2016-06-12 19:00:38,726 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.state.AMQStateManager: State changing to AMQState: id = 4 name: CONNECTION_OPEN from old state AMQState: id = 3 name: CONNECTION_NOT_OPENED
2016-06-12 19:00:38,726 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.state.AMQStateManager: Notififying State change to 1 : [org.apache.qpid.client.state.StateWaiter#5d7e5d44]
2016-06-12 19:00:38,726 INFO org.apache.servicemix.bundles.qpid:0.28.0.1(212) [Camel (amqpAlarmNorthbounderCamelContext) thread #4 - seda://forwardAlarm] org.apache.qpid.client.AMQConnection: Connection 44 now connected from /127.0.0.1:45388 to localhost/127.0.0.1:5672
2016-06-12 19:00:38,727 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [Camel (amqpAlarmNorthbounderCamelContext) thread #4 - seda://forwardAlarm] org.apache.qpid.client.AMQConnection: Are we connected:true
2016-06-12 19:00:38,727 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [Camel (amqpAlarmNorthbounderCamelContext) thread #4 - seda://forwardAlarm] org.apache.qpid.client.AMQConnection: Connected with ProtocolHandler Version:0-91
2016-06-12 19:00:38,727 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [Camel (amqpAlarmNorthbounderCamelContext) thread #4 - seda://forwardAlarm] org.apache.qpid.client.AMQConnectionDelegate_8_0: Write channel open frame for channel id 1
2016-06-12 19:00:38,727 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [Camel (amqpAlarmNorthbounderCamelContext) thread #4 - seda://forwardAlarm] org.apache.qpid.client.AMQSession: Created session:org.apache.qpid.client.AMQSession_0_8#179483f0
2016-06-12 19:00:38,728 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.protocol.AMQProtocolHandler: (1404676419)Method frame received: [ChannelOpenOkBodyImpl: channelId=null]
2016-06-12 19:00:38,728 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [IoReceiver - localhost/127.0.0.1:5672] org.apache.qpid.client.protocol.AMQProtocolHandler: (1404676419)Method frame received: [BasicQosOkBodyImpl: ]
2016-06-12 19:00:38,731 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [Camel (amqpAlarmNorthbounderCamelContext) thread #4 - seda://forwardAlarm] org.apache.qpid.client.AMQDestination: Based on onms3/Simon;{'create':'always','node':{'type':'topic'} } the selected destination syntax is ADDR
2016-06-12 19:00:38,731 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [Camel (amqpAlarmNorthbounderCamelContext) thread #4 - seda://forwardAlarm] org.apache.qpid.client.AMQSession: Closing session: org.apache.qpid.client.AMQSession_0_8#179483f0
2016-06-12 19:00:38,731 DEBUG org.apache.servicemix.bundles.qpid:0.28.0.1(212) [Camel (amqpAlarmNorthbounderCamelContext) thread #4 - seda://forwardAlarm] org.apache.qpid.client.protocol.AMQProtocolSession: closeSession called on protocol session for session 1
It closes the session before writing any information.
When i do essentially the same thing from external qpid client - executed as follows:-
#!/bin/bash
rm log.out
java -Dqpid.amqp.version=0-91 -Dlog4j.debug -Dlog4j.configuration=file:./log4j.properties -cp "client/example/target/classes/:client/example/target/dependency/*:slf4j-1.7.21/slf4j-log4j12-1.7.
21.jar:apache-log4j-1.2.17/log4j-1.2.17.jar" \
org.apache.qpid.example.ListSender
I get this:-
142 [main] DEBUG org.apache.qpid.client.AMQConnection - Are we connected:true
142 [main] DEBUG org.apache.qpid.client.AMQConnection - Connected with ProtocolHandler Version:0-91
146 [main] DEBUG org.apache.qpid.client.AMQConnectionDelegate_8_0 - Write channel open frame for channel id 1
162 [main] DEBUG org.apache.qpid.client.AMQSession - Created session:org.apache.qpid.client.AMQSession_0_8#6bdf28bb
164 [IoReceiver - localhost/127.0.0.1:5672] DEBUG org.apache.qpid.client.protocol.AMQProtocolHandler - (333274164)Method frame received: [ChannelOpenOkBody]
165 [IoReceiver - localhost/127.0.0.1:5672] DEBUG org.apache.qpid.client.protocol.AMQProtocolHandler - (333274164)Method frame received: [BasicQosOkBodyImpl: ]
173 [main] DEBUG org.apache.qpid.client.AMQDestination - Based on onms3/Simon;{create: always, node:{type: topic } } the selected destination syntax is ADDR
177 [main] DEBUG org.apache.qpid.framing.FieldTable - FieldTable::writeToBuffer: Writing encoded length of 0...
178 [IoReceiver - localhost/127.0.0.1:5672] DEBUG org.apache.qpid.client.protocol.AMQProtocolHandler - (333274164)Method frame received: [ExchangeDeclareOkBodyImpl: ]
179 [IoReceiver - localhost/127.0.0.1:5672] DEBUG org.apache.qpid.client.protocol.AMQProtocolHandler - (333274164)Method frame received: [ExchangeDeclareOkBodyImpl: ]
179 [IoReceiver - localhost/127.0.0.1:5672] DEBUG org.apache.qpid.client.protocol.AMQProtocolHandler - (333274164)Method frame received: [ExchangeDeclareOkBodyImpl: ]
180 [main] DEBUG org.apache.qpid.client.BasicMessageProducer_0_8 - MessageProducer org.apache.qpid.client.BasicMessageProducer_0_8#1936f0f5 using publish mode : ASYNC_PUBLISH_ALL
190 [main] DEBUG org.apache.qpid.client.BasicMessageProducer_0_8 - Sending content body frames to 'onms3'/'Simon'; {
'create': 'always',
'node': {
'type': 'topic'
}
}
190 [main] DEBUG org.apache.qpid.client.BasicMessageProducer_0_8 - Sending content header frame to 'onms3'/'Simon'; {
'create': 'always',
'node': {
'type': 'topic'
}
}
190 [main] DEBUG org.apache.qpid.framing.FieldTable - FieldTable::writeToBuffer: Writing encoded length of 90...
191 [main] DEBUG org.apache.qpid.framing.FieldTable - {Id=[INT: 987654321], name=[LONG_STRING: WidgetSimon], price=[DOUBLE: 0.99], qpid.subject=[LONG_STRING: Simon], JMS_QPID_DESTTYPE=[INT: 2]}
192 [main] DEBUG org.apache.qpid.client.AMQSession - Closing session: org.apache.qpid.client.AMQSession_0_8#6bdf28bb
192 [main] DEBUG org.apache.qpid.client.protocol.AMQProtocolSession - closeSession called on protocol session for session 1
194 [IoReceiver - localhost/127.0.0.1:5672] DEBUG org.apache.qpid.client.protocol.AMQProtocolHandler - (333274164)Method frame received: [ChannelCloseOkBody]
194 [IoReceiver - localhost/127.0.0.1:5672] INFO org.apache.qpid.client.handler.ChannelCloseOkMethodHandler - Received channel-close-ok for channel-id 1
195 [IoReceiver - localhost/127.0.0.1:5672] DEBUG org.apache.qpid.client.protocol.AMQProtocolHandler - (333274164)Method frame received: [ConnectionCloseOkBody]
196 [main] DEBUG org.apache.qpid.client.protocol.AMQProtocolHandler - Session closed called by client
ListSender.java as follows:-
package org.apache.qpid.example;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import javax.jms.Connection;
import javax.jms.Destination;
import javax.jms.Message;
import javax.jms.MessageProducer;
import javax.jms.Session;
import org.apache.qpid.client.AMQAnyDestination;
import org.apache.qpid.client.AMQConnection;
import org.apache.qpid.framing.AMQShortString;
import org.apache.qpid.jms.ListMessage;
public class ListSender {
public static void main(String[] args) throws Exception
{
Connection connection =
new AMQConnection("amqp://simon:simon#localhost/test?brokerlist='tcp://localhost:5672'");
AMQShortString a1 = new AMQShortString("");
AMQShortString a2 = new AMQShortString("");
AMQShortString[] bindvars = new AMQShortString[]{a1,a2};
boolean is_durable = true;
/*
Destination queue = new AMQAnyDestination( new AMQShortString("onms2"),
new AMQShortString("direct"),
new AMQShortString("Simon"),
true,
true,
new AMQShortString(""),
false,
bindvars);
*/
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
//Destination queue = new AMQAnyDestination("onms3/Simon");
Destination queue = new AMQAnyDestination("onms3/Simon;{create: always, node:{type: topic } }");
//Destination queue = new AMQAnyDestination("onms3/Simon;%7Bcreate%3A%20always%2C%20node%3A%7Btype%3A%20topic%20%7D%20%7D");
//Destination queue = new AMQAnyDestination("amqp:OpenNMSExchange/Taylor; {create: always}");
//Destination queue = new AMQAnyDestination("OpenNMSExchange; {create: always}");
MessageProducer producer = session.createProducer(queue);
ListMessage m = ((org.apache.qpid.jms.Session)session).createListMessage();
m.setIntProperty("Id", 987654321);
m.setStringProperty("name", "WidgetSimon");
m.setDoubleProperty("price", 0.99);
List<String> colors = new ArrayList<String>();
colors.add("red");
colors.add("green");
colors.add("white");
m.add(colors);
Map<String,Double> dimensions = new HashMap<String,Double>();
dimensions.put("length",10.2);
dimensions.put("width",5.1);
dimensions.put("depth",2.0);
m.add(dimensions);
List<List<Integer>> parts = new ArrayList<List<Integer>>();
parts.add(Arrays.asList(new Integer[] {1,2,5}));
parts.add(Arrays.asList(new Integer[] {8,2,5}));
m.add(parts);
Map<String,Object> specs = new HashMap<String,Object>();
specs.put("colours", colors);
specs.put("dimensions", dimensions);
specs.put("parts", parts);
m.add(specs);
producer.send((Message)m);
System.out.println("Sent: " + m);
connection.close();
}
}
My assumption was that this was a connectivity issue to the AMQP server of some description. Having faced a number of issues whilst troubleshooting this if i faced the issue from the external jar it was clear from the logs what the issue was.
Is this an issue with OpenNMS?
Does anyone have this working successfully?
Any ideas?
Cheers
Simon
Following guidance from the OpenNMS list I decided to install the QPID Broker v6.0.3 in place of RabbitMQ and messages are flowing no problems now.
AWS PutObject Connection reset
My AWS Java Client is throwing
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541) ~[na:1.8.0_60]
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553) ~[na:1.8.0_60]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) ~[na:1.8.0_60]
at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159) ~[httpcore-4.3.3.jar:4.3.3]
My code is
public void save(String name, byte[] file) {
ObjectMetadata metaData = new ObjectMetadata();
String streamMD5 = new String(Base64.encodeBase64(file));
metaData.setContentMD5(streamMD5);
metaData.setContentLength(file.length);
InputStream stream = new ByteArrayInputStream(file);
try {
PutObjectRequest put = new PutObjectRequest(
configuration.getBucketName(), name, stream, metaData);
s3client.putObject(put);
} finally {
IOUtils.closeQuietly(stream);
}
}
The s3client is a spring bean and is not garbage collected before the stream has finished uploading. I've tried without specifying MD5 and/or content length but still has the same exception thrown.
Logging through AWS library shows:
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: best-match
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.c.p.RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
10:09:15.540 [http-nio-8080-exec-1] DEBUG c.a.http.impl.client.SdkHttpClient - Attempt 1 to execute request
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.i.conn.DefaultClientConnection - Sending request: PUT /documeent.pdf HTTP/1.1
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "PUT /document.pdf HTTP/1.1[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "Host: bucket.s3.amazonaws.com[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "Authorization: AWS 123445667788=[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "User-Agent: aws-sdk-java/1.10.21 Linux/3.13.0-65-generic Java_HotSpot(TM)_Server_VM/25.60-b23/1.8.0_60[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "Date: Tue, 06 Oct 2015 09:09:15 GMT[\r][\n]"
10:09:15.663 [http-nio-8080-exec-1] DEBUG com.amazonaws.internal.SdkSSLSocket - closing bucket.s3.amazonaws.com/12.34.56.78:443
10:09:15.665 [http-nio-8080-exec-1] DEBUG o.a.h.i.conn.DefaultClientConnection - I/O error closing connection
I've checked that the file size (3.2M) does not exceed the max file siez for this bucket.
Get/List requests work fine and I can copy files into the S3 bucket using the s3 client tools.
Does anyone know of anything else I should check?
Thanks.