What is the recommended way to use Conscrypt with Apache HttpClient 5?
I've tried adding the conscrypt-openjdk-uber-2.2.1.jar jar to my classpath and configuring my sslcontext as SSLContexts.custom().setProvider(Conscrypt.newProvider()), but when I test HttpClient with that sslcontext it throws:
[main] INFO org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec -
Recoverable I/O exception (java.net.NoRouteToHostException) caught when processing request to
{s}->https://www.wikipedia.org:443
If I remove the custom setProvider line then it works perfectly (via regular JSSE).
I notice that Conscrypt is listed as a dependency here: https://hc.apache.org/httpcomponents-client-5.0.x/httpclient5/dependencies.html, so perhaps there is built-in support for Conscrypt that I need to enable somewhere?
You do not really need to do anything. HttpClient automatically detects and configures Conscrypt as a provider for its async TLS layer on Java 1.7 and 1.8.
For all newer JREs one can explicitly configure the connection manager to use Conscrypt based TLS strategy:
PoolingAsyncClientConnectionManager cm = PoolingAsyncClientConnectionManagerBuilder.create()
.setTlsStrategy(ConscryptClientTlsStrategy.getSystemDefault())
.build();
CloseableHttpAsyncClient client = HttpAsyncClients.custom()
.setVersionPolicy(HttpVersionPolicy.NEGOTIATE)
.setConnectionManager(cm)
.build();
UPDATED
The following code snippet works for me with HttpClient 5.0-beta7
final SSLContext sslcontext = SSLContexts.custom()
.setProvider(Conscrypt.newProvider())
.build();
final SSLConnectionSocketFactory sslSocketFactory = SSLConnectionSocketFactoryBuilder.create()
.setSslContext(sslcontext)
.build();
final HttpClientConnectionManager cm = PoolingHttpClientConnectionManagerBuilder.create()
.setSSLSocketFactory(sslSocketFactory)
.build();
try (CloseableHttpClient httpclient = HttpClients.custom()
.setConnectionManager(cm)
.build()) {
final HttpGet httpget = new HttpGet("https://www.wikipedia.org/");
System.out.println("Executing request " + httpget.getMethod() + " " + httpget.getUri());
final HttpClientContext clientContext = HttpClientContext.create();
try (CloseableHttpResponse response = httpclient.execute(httpget, clientContext)) {
System.out.println("----------------------------------------");
System.out.println(response.getCode() + " " + response.getReasonPhrase());
final SSLSession sslSession = clientContext.getSSLSession();
if (sslSession != null) {
System.out.println("SSL protocol " + sslSession.getProtocol());
System.out.println("SSL cipher suite " + sslSession.getCipherSuite());
}
}
}
Console output:
Executing request GET https://www.wikipedia.org/
2020-02-06 10:33:22,619 DEBUG ex-00000001: preparing request execution
2020-02-06 10:33:22,625 DEBUG Cookie spec selected: strict
2020-02-06 10:33:22,629 DEBUG Auth cache not set in the context
2020-02-06 10:33:22,629 DEBUG ex-00000001: target auth state: UNCHALLENGED
2020-02-06 10:33:22,630 DEBUG ex-00000001: proxy auth state: UNCHALLENGED
2020-02-06 10:33:22,630 DEBUG ex-00000001: acquiring connection with route {s}->https://www.wikipedia.org:443
2020-02-06 10:33:22,630 DEBUG ex-00000001: acquiring endpoint (3 MINUTES)
2020-02-06 10:33:22,632 DEBUG ex-00000001: endpoint lease request (3 MINUTES) [route: {s}->https://www.wikipedia.org:443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 25]
2020-02-06 10:33:22,636 DEBUG ex-00000001: endpoint leased [route: {s}->https://www.wikipedia.org:443][total available: 0; route allocated: 1 of 5; total allocated: 1 of 25]
2020-02-06 10:33:22,649 DEBUG ex-00000001: acquired ep-00000000
2020-02-06 10:33:22,649 DEBUG ex-00000001: acquired endpoint ep-00000000
2020-02-06 10:33:22,649 DEBUG ex-00000001: opening connection {s}->https://www.wikipedia.org:443
2020-02-06 10:33:22,650 DEBUG ep-00000000: connecting endpoint (3 MINUTES)
2020-02-06 10:33:22,650 DEBUG ep-00000000: connecting endpoint to https://www.wikipedia.org:443 (3 MINUTES)
2020-02-06 10:33:22,654 DEBUG http-outgoing-0: connecting to www.wikipedia.org/91.198.174.192:443
2020-02-06 10:33:22,654 DEBUG Connecting socket to www.wikipedia.org/91.198.174.192:443 with timeout 3 MINUTES
2020-02-06 10:33:22,759 DEBUG Enabled protocols: [TLSv1.2, TLSv1.3]
2020-02-06 10:33:22,759 DEBUG Enabled cipher suites:[TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2020-02-06 10:33:22,759 DEBUG Starting handshake
2020-02-06 10:33:23,192 DEBUG Secure session established
2020-02-06 10:33:23,192 DEBUG negotiated protocol: TLSv1.2
2020-02-06 10:33:23,192 DEBUG negotiated cipher suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
2020-02-06 10:33:23,192 DEBUG peer principal: CN=*.wikipedia.org, O="Wikimedia Foundation, Inc.", L=San Francisco, ST=California, C=US
2020-02-06 10:33:23,193 DEBUG peer alternative names: [*.wikipedia.org, *.wikimedia.org, *.wmfusercontent.org, *.wikimediafoundation.org, *.wiktionary.org, *.wikivoyage.org, *.wikiversity.org, *.wikisource.org, *.wikiquote.org, *.wikinews.org, *.wikidata.org, *.wikibooks.org, wikimedia.org, *.mediawiki.org, wikipedia.org, wikiquote.org, mediawiki.org, wmfusercontent.org, w.wiki, wikimediafoundation.org, wikibooks.org, wiktionary.org, wikivoyage.org, wikidata.org, wikiversity.org, wikisource.org, wikinews.org, *.m.wikipedia.org, *.m.wiktionary.org, *.m.wikivoyage.org, *.m.wikiquote.org, *.m.wikiversity.org, *.m.wikisource.org, *.m.wikimedia.org, *.m.wikinews.org, *.m.wikidata.org, *.m.wikibooks.org, *.planet.wikimedia.org, *.m.mediawiki.org]
2020-02-06 10:33:23,193 DEBUG issuer principal: CN=DigiCert SHA2 High Assurance Server CA, OU=www.digicert.com, O=DigiCert Inc, C=US
2020-02-06 10:33:23,196 DEBUG http-outgoing-0: connection established 192.168.43.143:55022<->91.198.174.192:443
2020-02-06 10:33:23,196 DEBUG ep-00000000: connected http-outgoing-0
2020-02-06 10:33:23,196 DEBUG ep-00000000: endpoint connected
2020-02-06 10:33:23,197 DEBUG ex-00000001: executing GET / HTTP/1.1
2020-02-06 10:33:23,197 DEBUG ep-00000000: start execution ex-00000001
2020-02-06 10:33:23,197 DEBUG ep-00000000: executing exchange ex-00000001 over http-outgoing-0
2020-02-06 10:33:23,198 DEBUG http-outgoing-0 >> GET / HTTP/1.1
2020-02-06 10:33:23,198 DEBUG http-outgoing-0 >> Accept-Encoding: gzip, x-gzip, deflate
2020-02-06 10:33:23,198 DEBUG http-outgoing-0 >> Host: www.wikipedia.org
2020-02-06 10:33:23,198 DEBUG http-outgoing-0 >> Connection: keep-alive
2020-02-06 10:33:23,198 DEBUG http-outgoing-0 >> User-Agent: Apache-HttpClient/5.0-beta8-SNAPSHOT (Java/1.8.0_181)
2020-02-06 10:33:23,402 DEBUG http-outgoing-0 << HTTP/1.1 200 OK
2020-02-06 10:33:23,403 DEBUG http-outgoing-0 << Date: Wed, 05 Feb 2020 20:39:26 GMT
2020-02-06 10:33:23,403 DEBUG http-outgoing-0 << Cache-Control: s-maxage=86400, must-revalidate, max-age=3600
2020-02-06 10:33:23,403 DEBUG http-outgoing-0 << Server: ATS/8.0.5
2020-02-06 10:33:23,404 DEBUG http-outgoing-0 << X-ATS-Timestamp: 1580935166
2020-02-06 10:33:23,404 DEBUG http-outgoing-0 << ETag: W/"12be8-59c0633ed3519"
2020-02-06 10:33:23,404 DEBUG http-outgoing-0 << Content-Type: text/html
2020-02-06 10:33:23,404 DEBUG http-outgoing-0 << Last-Modified: Mon, 13 Jan 2020 14:22:18 GMT
2020-02-06 10:33:23,405 DEBUG http-outgoing-0 << Backend-Timing: D=320 t=1579084179579408
2020-02-06 10:33:23,405 DEBUG http-outgoing-0 << Content-Encoding: gzip
2020-02-06 10:33:23,405 DEBUG http-outgoing-0 << Vary: Accept-Encoding
2020-02-06 10:33:23,405 DEBUG http-outgoing-0 << X-Varnish: 118503554 495852195
2020-02-06 10:33:23,406 DEBUG http-outgoing-0 << Age: 46437
2020-02-06 10:33:23,406 DEBUG http-outgoing-0 << X-Cache: cp3062 miss, cp3052 hit/600912
2020-02-06 10:33:23,406 DEBUG http-outgoing-0 << X-Cache-Status: hit-front
2020-02-06 10:33:23,407 DEBUG http-outgoing-0 << Server-Timing: cache;desc="hit-front"
2020-02-06 10:33:23,407 DEBUG http-outgoing-0 << Strict-Transport-Security: max-age=106384710; includeSubDomains; preload
2020-02-06 10:33:23,407 DEBUG http-outgoing-0 << Set-Cookie: WMF-Last-Access=06-Feb-2020;Path=/;HttpOnly;secure;Expires=Mon, 09 Mar 2020 00:00:00 GMT
2020-02-06 10:33:23,407 DEBUG http-outgoing-0 << Set-Cookie: WMF-Last-Access-Global=06-Feb-2020;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Mon, 09 Mar 2020 00:00:00 GMT
2020-02-06 10:33:23,408 DEBUG http-outgoing-0 << X-Client-IP: 213.55.225.99
2020-02-06 10:33:23,418 DEBUG http-outgoing-0 << Set-Cookie: GeoIP=CH:ZH:Zurich:47.37:8.55:v4; Path=/; secure; Domain=.wikipedia.org
2020-02-06 10:33:23,418 DEBUG http-outgoing-0 << Accept-Ranges: bytes
2020-02-06 10:33:23,418 DEBUG http-outgoing-0 << Content-Length: 18800
2020-02-06 10:33:23,419 DEBUG http-outgoing-0 << Connection: keep-alive
2020-02-06 10:33:23,429 DEBUG ex-00000001: connection can be kept alive for -1 MILLISECONDS
2020-02-06 10:33:23,437 DEBUG Cookie accepted [WMF-Last-Access="06-Feb-2020", domain:www.wikipedia.org, path:/, expiry:Mon Mar 09 01:00:00 CET 2020]
2020-02-06 10:33:23,438 DEBUG Cookie accepted [WMF-Last-Access-Global="06-Feb-2020", domain:wikipedia.org, path:/, expiry:Mon Mar 09 01:00:00 CET 2020]
2020-02-06 10:33:23,438 DEBUG Cookie accepted [GeoIP="CH:ZH:Zurich:47.37:8.55:v4", domain:wikipedia.org, path:/, expiry:null]
----------------------------------------
200 OK
SSL protocol TLSv1.2
SSL cipher suite TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
2020-02-06 10:33:23,467 DEBUG http-outgoing-0: close connection GRACEFUL
2020-02-06 10:33:23,468 DEBUG ep-00000000: endpoint closed
2020-02-06 10:33:23,468 DEBUG ep-00000000: endpoint closed
2020-02-06 10:33:23,468 DEBUG ep-00000000: discarding endpoint
2020-02-06 10:33:23,468 DEBUG ep-00000000: releasing endpoint
2020-02-06 10:33:23,469 DEBUG ep-00000000: connection released [route: {s}->https://www.wikipedia.org:443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 25]
2020-02-06 10:33:23,469 DEBUG Shutdown connection pool GRACEFUL
2020-02-06 10:33:23,469 DEBUG Connection pool shut down
Related
I have a PUT requests that is done by a dependency scanner we are using
I'm behind a corporate proxy
Request is done with Apache HTTP Client 4
I get a "read time out" after the configured timeout
I recreated the request with curl and this works fine (with the same proxy) and finishes within seconds
It works fine with Apache HTTP Client without the proxy.
So I'm trying to find out what config is missing in the HTTP client config or what else could be the error. Unfortunately neither our networking team nor the product vendor could help me out :-(
This is the curl request:
curl --connect-timeout 60 -m 60 -v -X PUT --data-binary #test.json
https://someblackduckserver/api/developer-scans/12345
-H "Content-type: application/vnd.blackducksoftware.developer-scan-1-ld-2+json"
-H "X-BD-DOCUMENT-COUNT: 1"
-H "Accept: application/vnd.blackducksoftware.scan-4+json"
-H "X-BD-MODE: append"
-H "X-BD-RAPID-SCAN-MODE: ALL" -H "X-BD-VERSION-NAME: scan-test"
-H "User-Agent: synopsys_detect/8.1.0-SNAPSHOT BlackDuckCommon/65.0.0 ..."
-H "X-BD-PROJECT-NAME: scan-test"
-H "Authorization: Bearer ..... left out ....."
Where test.json is a json file with about 1.3MB.
Minified code when using Apache HTTP Client:
RequestBuilder requestBuilder = RequestBuilder.create("PUT");
URIBuilder uriBuilder = new URIBuilder(new URI("https://someblackduckserver/api/developer-scans/12345"));
requestBuilder.setUri(uriBuilder.build());
Charset bodyEncoding = StandardCharsets.UTF_8;
requestBuilder.setCharset(bodyEncoding);
requestBuilder.addHeader("Content-type", "application/vnd.blackducksoftware.developer-scan-1-ld-2+json");
requestBuilder.addHeader("X-BD-DOCUMENT-COUNT", "1");
requestBuilder.addHeader("Accept", "application/vnd.blackducksoftware.scan-4+json");
requestBuilder.addHeader("X-BD-MODE", "append");
requestBuilder.addHeader("X-BD-RAPID-SCAN-MODE", "ALL");
requestBuilder.addHeader("X-BD-VERSION-NAME", "scan-test");
requestBuilder.addHeader("User-Agent", "synopsys_detect/8.1.0-SNAPSHOT BlackDuckCommon/65.0.0 ...");
requestBuilder.addHeader("X-BD-PROJECT-NAME", "scan-test");
requestBuilder.addHeader("Authorization", "Bearer ..... left out .....");
HttpEntity httpEntity = new FileEntity(new File("test.json"));
requestBuilder.setEntity(httpEntity);
HttpUriRequest request = requestBuilder.build();
HttpContext httpContext = new BasicHttpContext();
HttpClientBuilder clientBuilder = HttpClientBuilder.create();
RequestConfig.Builder defaultRequestConfigBuilder = RequestConfig.custom().setCookieSpec(CookieSpecs.STANDARD);
// set our proxy
defaultRequestConfigBuilder.setProxy(new HttpHost("..... (left out)", 8080));
// set timeout
int timeoutInSeconds = 60;
defaultRequestConfigBuilder.setConnectTimeout(timeoutInSeconds * 1000);
defaultRequestConfigBuilder.setSocketTimeout(timeoutInSeconds * 1000);
defaultRequestConfigBuilder.setConnectionRequestTimeout(timeoutInSeconds * 1000);
clientBuilder.setDefaultRequestConfig(defaultRequestConfigBuilder.build());
// for testing: ignore ssl
HostnameVerifier hostnameVerifier;
SSLContext sslContext = SSLContextBuilder.create().loadTrustMaterial(new TrustAllStrategy()).build();
hostnameVerifier = new NoopHostnameVerifier();
SSLConnectionSocketFactory connectionFactory = new SSLConnectionSocketFactory(sslContext, hostnameVerifier);
clientBuilder.setSSLSocketFactory(connectionFactory);
try (CloseableHttpClient client = clientBuilder.build()) {
CloseableHttpResponse closeableHttpResponse = client.execute(request, httpContext);
System.out.println(closeableHttpResponse);
}
I removed some sensitive details from the request.
Any idea what I'm missing here or what could go wrong?
Logs from a sample the call:
2022-07-25 10:38:38.201 [main] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default
2022-07-25 10:38:38.206 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
2022-07-25 10:38:38.207 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {tls}->http://our.proxy.ip.address:8080->https://someblackduckserver.com:443][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20]
2022-07-25 10:38:38.259 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {tls}->http://our.proxy.ip.address:8080->https://someblackduckserver.com:443][total available: 0; route allocated: 1 of 2; total allocated: 1 of 20]
2022-07-25 10:38:38.260 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Opening connection {tls}->http://our.proxy.ip.address:8080->https://someblackduckserver.com:443
2022-07-25 10:38:38.261 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to /our.proxy.ip.address:8080
2022-07-25 10:38:38.301 [main] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connection established 10.17.10.44:34272<->our.proxy.ip.address:8080
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> CONNECT someblackduckserver.com:443 HTTP/1.1
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: someblackduckserver.com
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.13 (Java/11.0.13)
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "CONNECT someblackduckserver.com:443 HTTP/1.1[\r][\n]"
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: someblackduckserver.com[\r][\n]"
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.13 (Java/11.0.13)[\r][\n]"
2022-07-25 10:38:38.317 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
2022-07-25 10:38:38.346 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 200 Connection established[\r][\n]"
2022-07-25 10:38:38.346 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Proxy-Agent: Proxy-Vendor-Proxy/1.0[\r][\n]"
2022-07-25 10:38:38.346 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
2022-07-25 10:38:38.347 [main] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 Connection established
2022-07-25 10:38:38.347 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Proxy-Agent: Proxy-Vendor-Proxy/1.0
2022-07-25 10:38:38.347 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Tunnel to target created.
2022-07-25 10:38:38.432 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled protocols: [TLSv1.3, TLSv1.2]
2022-07-25 10:38:38.440 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Enabled cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2022-07-25 10:38:38.440 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Starting handshake
2022-07-25 10:38:39.252 [main] DEBUG jdk.event.security - X509Certificate: ... our ssl inspect certificate ...
2022-07-25 10:38:39.252 [main] DEBUG jdk.event.security - X509Certificate: ... our internal certificate ...
2022-07-25 10:38:39.275 [main] DEBUG jdk.event.security - TLSHandshake: someblackduckserver.com:8080, TLSv1.2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, 1942523025
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - Secure session established
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - negotiated protocol: TLSv1.2
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - peer principal: CN=*.someblackduckserver.com, O="Synopsys, Inc.", L=Mountain View, ST=California, C=US
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - peer alternative names: [*.someblackduckserver.com]
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.c.s.SSLConnectionSocketFactory - issuer principal: ... our ssl inspect certificate ...
2022-07-25 10:38:39.276 [main] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 60000
2022-07-25 10:38:39.277 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Executing request PUT /api/developer-scans/12345 HTTP/1.1
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> PUT /api/developer-scans/12345 HTTP/1.1
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-type: application/vnd.blackducksoftware.developer-scan-1-ld-2+json
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-DOCUMENT-COUNT: 1
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept: application/vnd.blackducksoftware.scan-4+json
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-MODE: append
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-RAPID-SCAN-MODE: ALL
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-VERSION-NAME: rapid-scan-test-igl-2
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: synopsys_detect/8.1.0-SNAPSHOT BlackDuckCommon/65.0.0 (Eclipse Foundation 11.0.12 amd64 Windows 10 10.0)
2022-07-25 10:38:39.277 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> X-BD-PROJECT-NAME: project_name
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Authorization: Bearer ...
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 287381
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: someblackduckserver.com
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "PUT /api/developer-scans/12345 HTTP/1.1[\r][\n]"
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-type: application/vnd.blackducksoftware.developer-scan-1-ld-2+json[\r][\n]"
2022-07-25 10:38:39.278 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-DOCUMENT-COUNT: 1[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept: application/vnd.blackducksoftware.scan-4+json[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-MODE: append[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-RAPID-SCAN-MODE: ALL[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-VERSION-NAME: rapid-scan-test-igl-2[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: synopsys_detect/8.1.0-SNAPSHOT BlackDuckCommon/65.0.0 (Eclipse Foundation 11.0.12 amd64 Windows 10 10.0)[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "X-BD-PROJECT-NAME: project_name[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Authorization: Bearer ...[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Length: 287381[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: someblackduckserver.com[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2022-07-25 10:38:39.279 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2022-07-25 10:38:39.423 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> .... all the logs for file content ....
2022-07-25 10:39:39.483 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[read] I/O error: Read timed out"
2022-07-25 10:39:39.483 [main] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: Close connection
2022-07-25 10:39:39.483 [main] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: Shutdown connection
2022-07-25 10:39:39.483 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Connection discarded
2022-07-25 10:39:39.483 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {tls}->http://our.proxy.ip.address:8080->https://someblackduckserver.com:443][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20]
2022-07-25 10:39:39.484 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection manager is shutting down
2022-07-25 10:39:39.484 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection manager shut down
Exception in thread "main" java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1318)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:959)
at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.example.BdTest.main(BdTest.java:93)
May be not a solution but a workaround if it works. I wrote my own Http client that is much simpler than Apache Http client. The reason I wrote it is that in many cases you just need a simplistic functionality so using a thin library lessens complexity and allows you to debug it if need be and it would allow you easier way to get to the bottom of the problem. So, may be you can give it a shot. At may even work or if not than it would be much easier to debug than Apache Http client. My Http client is part of MgntUtils Open-Source library written and maintained by me. Here is HttpClient Javadoc. The library can be obtained as Maven artifact from Maven Central and from Github (including source code and Javadoc)
I'm trying to make a request through a NTLM proxy like this:
try (CloseableHttpClient httpClient = WinHttpClients.createDefault()) {
HttpHost target = HttpHost.create("http://example.com");
HttpHost proxy = new HttpHost("http", "localhost", 80);
RequestConfig config = RequestConfig.custom().setProxy(proxy).build();
HttpGet request = new HttpGet("/");
request.setConfig(config);
try (CloseableHttpResponse response = httpClient.execute(target, request)) {
StatusLine statusLine = new StatusLine(response);
if (statusLine.getStatusCode() == HttpStatus.SC_OK) {
System.out.println("Test OK");
} else if (statusLine.getStatusCode() == HttpStatus.SC_PROXY_AUTHENTICATION_REQUIRED) {
throw new CredentialException(statusLine.getReasonPhrase());
}
}
} catch (Exception e) {
e.printStackTrace();
}
but it fails, here is the log:
[main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - ex-00000001: preparing request execution
[main] DEBUG org.apache.hc.client5.http.protocol.RequestAddCookies - Cookie spec selected: strict
[main] DEBUG org.apache.hc.client5.http.protocol.RequestAuthCache - Auth cache not set in the context
[main] DEBUG org.apache.hc.client5.http.impl.classic.ProtocolExec - ex-00000001: target auth state: UNCHALLENGED
[main] DEBUG org.apache.hc.client5.http.impl.classic.ProtocolExec - ex-00000001: proxy auth state: UNCHALLENGED
[main] DEBUG org.apache.hc.client5.http.impl.classic.ConnectExec - ex-00000001: acquiring connection with route {}->http://localhost:80->http://example.com:80
[main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - ex-00000001: acquiring endpoint (3 MINUTES)
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ex-00000001: endpoint lease request (3 MINUTES) [route: {}->http://localhost:80->http://example.com:80][total available: 0; route allocated: 0 of 5; total allocated: 0 of 25]
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ex-00000001: endpoint leased [route: {}->http://localhost:80->http://example.com:80][total available: 0; route allocated: 1 of 5; total allocated: 1 of 25]
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ex-00000001: acquired ep-00000000
[main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - ex-00000001: acquired endpoint ep-00000000
[main] DEBUG org.apache.hc.client5.http.impl.classic.ConnectExec - ex-00000001: opening connection {}->http://localhost:80->http://example.com:80
[main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-00000000: connecting endpoint (3 MINUTES)
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-00000000: connecting endpoint to http://localhost:80 (3 MINUTES)
[main] DEBUG org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator - http-outgoing-0: connecting to localhost/127.0.0.1:80
[main] DEBUG org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator - http-outgoing-0: connection established 127.0.0.1:57194<->127.0.0.1:80
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-00000000: connected http-outgoing-0
[main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-00000000: endpoint connected
[main] DEBUG org.apache.hc.client5.http.impl.classic.MainClientExec - ex-00000001: executing GET http://example.com/ HTTP/1.1
[main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-00000000: start execution ex-00000001
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-00000000: executing exchange ex-00000001 over http-outgoing-0
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 >> GET http://example.com/ HTTP/1.1
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 >> Accept-Encoding: gzip, x-gzip, deflate
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 >> Host: example.com
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/5.0 (Java/1.8.0_222)
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 >> "GET http://example.com/ HTTP/1.1[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip, x-gzip, deflate[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 >> "Host: example.com[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpClient/5.0 (Java/1.8.0_222)[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 >> "[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "HTTP/1.1 407 Proxy authorization required[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "Proxy-Authenticate: NTLM[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "Proxy-Authenticate: Basic realm="/", charset="UTF-8"[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "Proxy-Support: session-based-authentication[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "Cache-control: no-store, no-cache[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "Content-Type: text/html; charset=utf-8[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "Content-Length: 855[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "Date: Sun, 05 Apr 2020 06:15:20 GMT[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "Server: WinGate Engine[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "Connection: Keep-Alive[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/strict.dtd"><html><head><title>Proxy authorization required</title>[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "<style>[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "body { font-family:helvetica; font-size:11pt; color:#0071BB; margin:30px; padding:0px; }[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "h1 { color:#005395; margin:0px 0px 0px 0px; }[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "h2 { color:#005395; }[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "p { margin:6px 0px 6px 0px;}[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "p.footer { font-size:8pt; margin:3px 0px 0px 0px;}[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "b { font-weight:550; color:#7F7F7F; }[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "</style>[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "</head>[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "<body><h1>Proxy authorization required</h1>[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "<div style="margin:12px 0px 12px 0px; padding:12px 0px 12px 0px; min-height:300px; border-top:1px solid #0071BB; border-bottom:1px solid #0071BB;">You must authenticate to gain access to this resource</div>[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "<p class="footer">WinGate™ © Qbik New Zealand Limited 1995-2020[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.wire - http-outgoing-0 << "</p></body></html>[\r][\n]"
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << HTTP/1.1 407 Proxy authorization required
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << Proxy-Authenticate: NTLM
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << Proxy-Authenticate: Basic realm="/", charset="UTF-8"
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << Proxy-Support: session-based-authentication
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << Cache-control: no-store, no-cache
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << Content-Type: text/html; charset=utf-8
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << Content-Length: 855
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << Date: Sun, 05 Apr 2020 06:15:20 GMT
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << Server: WinGate Engine
[main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-0 << Connection: Keep-Alive
[main] DEBUG org.apache.hc.client5.http.impl.classic.MainClientExec - ex-00000001: connection can be kept alive for 3 MINUTES
[main] DEBUG org.apache.hc.client5.http.impl.classic.ProtocolExec - Authentication required
[main] DEBUG org.apache.hc.client5.http.impl.classic.ProtocolExec - localhost:80 requested authentication
[main] DEBUG org.apache.hc.client5.http.impl.DefaultAuthenticationStrategy - Authentication schemes in the order of preference: [Negotiate, Kerberos, NTLM, Digest, Basic]
[main] DEBUG org.apache.hc.client5.http.impl.DefaultAuthenticationStrategy - Challenge for Negotiate authentication scheme not available
[main] DEBUG org.apache.hc.client5.http.impl.DefaultAuthenticationStrategy - Challenge for Kerberos authentication scheme not available
[main] DEBUG org.apache.hc.client5.http.impl.win.WindowsNegotiateScheme - Created WindowsNegotiateScheme using NTLM
[main] DEBUG org.apache.hc.client5.http.impl.DefaultAuthenticationStrategy - Challenge for Digest authentication scheme not available
[main] WARN org.apache.hc.client5.http.impl.classic.ProtocolExec - Missing auth challenge
[main] DEBUG org.apache.hc.client5.http.impl.io.DefaultManagedHttpClientConnection - http-outgoing-0: close connection GRACEFUL
[main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-00000000: endpoint closed
[main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-00000000: endpoint closed
[main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-00000000: discarding endpoint
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-00000000: releasing endpoint
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-00000000: connection released [route: {}->http://localhost:80->http://example.com:80][total available: 0; route allocated: 0 of 5; total allocated: 0 of 25]
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - Shutdown connection pool GRACEFUL
[main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - Connection pool shut down
javax.security.auth.login.CredentialException: Proxy authorization required
at org.kpax.http_client_5_testing.App.main(App.java:47)
Environment
OS: Windows 10
JRE: AdoptOpenJdk v11
Proxy type: NTLM (Wingate)
Question Is there something wrong with the above code?
Obs The same code works fine with Apache Http Client v4.5.12
It is a minor bug, solved now, for v5.0.1.
I am trying to fetch the list of objects from a bucket of amazon s3. Below is the code that i am using.
AWSCredentials credentials = new BasicAWSCredentials(serverSettings.getAccessKey(), serverSettings.getSecretesKey());
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setSignerOverride("AWSS3V4SignerType");
AmazonS3 s3Client = AmazonS3ClientBuilder
.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(serverSettings.getStorageUri(), Regions.AP_SOUTH_1.name()))
.withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfiguration)
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.build();
return s3Client;
public List<String> getBucketFiles(String bucket) {
ListObjectsRequest listObjectsRequest = new ListObjectsRequest()
.withBucketName(bucket)
.withDelimiter("/");
ObjectListing listing = s3Client.listObjects(listObjectsRequest);
log.debug("Bucket : {} contain files : {}", bucket, filesInBucket);
return filesInBucket;
}
While fetching s3Client.listObjects(listObjectsRequest) i get the below error.
2020-02-25 15:45:03.955 DEBUG com.amazonaws.AmazonWebServiceClient - Internal logging successfully configured to commons logger: true
2020-02-25 15:45:03.955 DEBUG com.amazonaws.metrics.AwsSdkMetrics - Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics
2020-02-25 15:45:04.002 WARN com.amazonaws.http.AmazonHttpClient - SSL Certificate checking for endpoints has been explicitly disabled.
2020-02-25 15:45:04.086 DEBUG com.amazonaws.request - Sending Request: GET https://s3.console.aws.amazon.com/s3/home?region=ap-southeast-1# /r9a2/ Parameters: ({"delimiter":["/"],"encoding-type":["url"]}Headers: (User-Agent: aws-sdk-java/1.11.232 Windows_10/10.0 Java_HotSpot(TM)_64-Bit_Server_VM/25.241-b07 java/1.8.0_241, amz-sdk-invocation-id: 24587b25-9691-3123-343a-329a6e57d106, Content-Type: application/octet-stream, )
2020-02-25 15:45:04.171 DEBUG com.amazonaws.auth.AWS4Signer - AWS4 Canonical Request: '"GET
/s3/home/r9a2/
delimiter=%2F&encoding-type=url
amz-sdk-invocation-id:24587b25-9691-3123-343a-329a6e57d106
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:s3.console.aws.amazon.com
user-agent:aws-sdk-java/1.11.232 Windows_10/10.0 Java_HotSpot(TM)_64-Bit_Server_VM/25.241-b07 java/1.8.0_241
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20200225T101504Z
amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD"
2020-02-25 15:45:04.171 DEBUG com.amazonaws.auth.AWS4Signer - AWS4 String to Sign: '"AWS4-HMAC-SHA256
20200225T101504Z
20200225/AP_SOUTH_1/s3/aws4_request
2104ae348572863769961aabcbe488d3aae778eb807905f0d8e67edf95b6f659"
2020-02-25 15:45:04.171 DEBUG com.amazonaws.auth.AWS4Signer - Generating a new signing key as the signing key not available in the cache for the date 1582588800000
2020-02-25 15:45:04.187 DEBUG org.apache.http.client.protocol.RequestAddCookies - CookieSpec selected: default
2020-02-25 15:45:04.187 DEBUG org.apache.http.client.protocol.RequestAuthCache - Auth cache not set in the context
2020-02-25 15:45:04.187 DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {s}->https://s3.console.aws.amazon.com:443][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
2020-02-25 15:45:04.202 DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {s}->https://s3.console.aws.amazon.com:443][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
2020-02-25 15:45:04.202 DEBUG org.apache.http.impl.execchain.MainClientExec - Opening connection {s}->https://s3.console.aws.amazon.com:443
2020-02-25 15:45:05.961 DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator - Connecting to s3.console.aws.amazon.com/54.240.226.19:443
2020-02-25 15:45:05.975 DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator - Connection established 172.27.147.254:63178<->54.240.226.19:443
2020-02-25 15:45:05.975 DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 50000
2020-02-25 15:45:05.975 DEBUG org.apache.http.impl.execchain.MainClientExec - Executing request GET /s3/home?region=ap-southeast-1 HTTP/1.1
2020-02-25 15:45:05.975 DEBUG org.apache.http.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> GET /s3/home?region=ap-southeast-1 HTTP/1.1
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> Host: s3.console.aws.amazon.com
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIA3N5SPIMUI7VMSV6J/20200225/AP_SOUTH_1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2f08656e1ec077e9b5c69ffd4d5ffff623a8576e66298fb77634ce04a0f7b27b
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> X-Amz-Date: 20200225T101504Z
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: aws-sdk-java/1.11.232 Windows_10/10.0 Java_HotSpot(TM)_64-Bit_Server_VM/25.241-b07 java/1.8.0_241
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> amz-sdk-invocation-id: 24587b25-9691-3123-343a-329a6e57d106
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> amz-sdk-retry: 0/0/500
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Type: application/octet-stream
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 0
2020-02-25 15:45:05.975 DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 OK
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Server: Server
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Date: Tue, 25 Feb 2020 10:15:06 GMT
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Content-Type: text/html;charset=ISO-8859-1
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Content-Length: 989
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Strict-Transport-Security: max-age=47304000; includeSubDomains
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Cache-Control: proxy-revalidate
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Connection: Keep-Alive
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Set-Cookie: awsc-authTimer=%7B%22start%22%3A%221582625706301%22%7D; Secure; HttpOnly
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Set-Cookie: aws-creds=""; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/s3; Secure
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Set-Cookie: aws-creds=""; Domain=s3.console.aws.amazon.com; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/s3; Secure
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Set-Cookie: aws-creds=""; Domain=console.aws.amazon.com; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/s3; Secure
2020-02-25 15:45:06.346 DEBUG org.apache.http.headers - http-outgoing-0 << Set-Cookie: JSESSIONID=C16202E2D237233CBB71A1BC2EC29B0A; Path=/; HttpOnly
2020-02-25 15:45:06.346 DEBUG org.apache.http.impl.execchain.MainClientExec - Connection can be kept alive for 60000 MILLISECONDS
2020-02-25 15:45:06.346 DEBUG org.apache.http.client.protocol.ResponseProcessCookies - Cookie accepted [awsc-authTimer="%7B%22start%22%3A%221582625706301%22%7D", version:0, domain:s3.console.aws.amazon.com, path:/s3, expiry:null]
2020-02-25 15:45:06.346 DEBUG org.apache.http.client.protocol.ResponseProcessCookies - Cookie accepted [aws-creds="""", version:0, domain:s3.console.aws.amazon.com, path:/s3, expiry:Thu Jan 01 05:30:10 IST 1970]
2020-02-25 15:45:06.346 DEBUG org.apache.http.client.protocol.ResponseProcessCookies - Cookie accepted [aws-creds="""", version:0, domain:s3.console.aws.amazon.com, path:/s3, expiry:Thu Jan 01 05:30:10 IST 1970]
2020-02-25 15:45:06.346 DEBUG org.apache.http.client.protocol.ResponseProcessCookies - Cookie accepted [aws-creds="""", version:0, domain:console.aws.amazon.com, path:/s3, expiry:Thu Jan 01 05:30:10 IST 1970]
2020-02-25 15:45:06.346 DEBUG org.apache.http.client.protocol.ResponseProcessCookies - Cookie accepted [JSESSIONID="C16202E2D237233CBB71A1BC2EC29B0A", version:0, domain:s3.console.aws.amazon.com, path:/, expiry:null]
2020-02-25 15:45:06.362 DEBUG com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser - Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
2020-02-25 15:45:06.362 DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {s}->https://s3.console.aws.amazon.com:443] can be kept alive for 60.0 seconds
2020-02-25 15:45:06.362 DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {s}->https://s3.console.aws.amazon.com:443][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
2020-02-25 15:45:06.362 DEBUG com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser - Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:161) ~[aws-java-sdk-s3-1.11.232.jar:?]
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:317) ~[aws-java-sdk-s3-1.11.232.jar:?]
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70) ~[aws-java-sdk-s3-1.11.232.jar:?]
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59) ~[aws-java-sdk-s3-1.11.232.jar:?]
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62) ~[aws-java-sdk-s3-1.11.232.jar:?]
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31) ~[aws-java-sdk-s3-1.11.232.jar:?]
at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.232.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4319) ~[aws-java-sdk-s3-1.11.232.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4266) ~[aws-java-sdk-s3-1.11.232.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4260) ~[aws-java-sdk-s3-1.11.232.jar:?]
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:831) ~[aws-java-sdk-s3-1.11.232.jar:?]
at org.apache.sshd.common.file.root.RootedFileSystemProvider$1.iterator(RootedFileSystemProvider.java:204) [sshd-core-2.1.0.jar:2.1.0]
at org.apache.sshd.server.subsystem.sftp.DirectoryHandle.<init>(DirectoryHandle.java:52) [sshd-sftp-2.1.0.jar:2.1.0]
at org.apache.sshd.server.subsystem.sftp.SftpSubsystem.doOpenDir(SftpSubsystem.java:746) [sshd-sftp-2.1.0.jar:2.1.0]
at org.apache.sshd.server.subsystem.sftp.AbstractSftpSubsystemHelper.doOpenDir(AbstractSftpSubsystemHelper.java:696) [sshd-sftp-2.1.0.jar:2.1.0]
at org.apache.sshd.server.subsystem.sftp.AbstractSftpSubsystemHelper.doProcess(AbstractSftpSubsystemHelper.java:369) [sshd-sftp-2.1.0.jar:2.1.0]
at org.apache.sshd.server.subsystem.sftp.SftpSubsystem.doProcess(SftpSubsystem.java:335) [sshd-sftp-2.1.0.jar:2.1.0]
at org.apache.sshd.server.subsystem.sftp.AbstractSftpSubsystemHelper.process(AbstractSftpSubsystemHelper.java:336) [sshd-sftp-2.1.0.jar:2.1.0]
at org.apache.sshd.server.subsystem.sftp.SftpSubsystem.run(SftpSubsystem.java:308) [sshd-sftp-2.1.0.jar:2.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
Caused by: org.xml.sax.SAXParseException: The reference to entity "state" must end with the ';' delimiter.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEntityReference(XMLDocumentFragmentScannerImpl.java:1850) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3061) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[?:1.8.0_241]
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213) ~[?:1.8.0_241]
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:147) ~[aws-java-sdk-s3-1.11.232.jar:?]
... 36 more
The things which i have tried to solve this are:
- Added encoding url as .withEncodingType("url"); but doesnt work.
- Updated the access of the bucket as public access.
Version of amazon sdk used is 1.11.232
Any idea how to resolve this issue?
I'm using KerberosTemplate to make a POST request to a kerberos authenticated rest API.
I can successfully run an insecure request on the command line but when I try to do a secure call with HTTP client It returns a 401 unauthorized error.
I tried to turn of certificate checking as we dont use it and I also tries the username without the domain but that has no affect either.
I also tried to kerb4j rest template too but I get the same issue. From the code you can also see that Im checking the keytab file exists as well.
Appreciate some help, thanks.
import ErrorHandlers.KerberosRestTemplateResponseErrorHandler;
import POJOs.PostBody;
import POJOs.PostResponse;
import com.fasterxml.jackson.databind.SerializationFeature;
//import com.kerb4j.client.spring.KerberosRestTemplate;
import org.springframework.security.kerberos.client.KerberosRestTemplate;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.http.converter.json.MappingJackson2HttpMessageConverter;
import javax.net.ssl.SSLContext;
import java.io.File;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.X509Certificate;
public class KerbTemplate {
public static void main(String[] args) throws KeyStoreException, NoSuchAlgorithmException, KeyManagementException {
PostBody postRequestBody = new PostBody();
postRequestBody.setAggregation("data");
postRequestBody.setEndDate("2119-03-30");
postRequestBody.setStartDate("1900-03-30");
postRequestBody.setMetadata(new String[] {"PARTIAL"});
postRequestBody.setSymbols(new String[] {"requestdata"});
postRequestBody.setModifiedDate("2019-07-20");
postRequestBody.setFirstWeekday("0");
TrustStrategy acceptingTrustStrategy = ((X509Certificate[] chain, String authType) -> true);
SSLContext sslContext = org.apache.http.ssl.SSLContexts.custom().loadTrustMaterial(null, acceptingTrustStrategy).build();
SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(sslContext);
CloseableHttpClient httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setHttpClient(httpClient);
String base = System.getProperty("user.dir");
String keytab = base+ File.separator+"src"+File.separator+"main"+File.separator+"resources"+File.separator+"username.keytab";
System.out.println(keytab);
File f = new File(keytab);
if(f.exists() && !f.isDirectory()) {
System.out.println("File exists");
}
KerberosRestTemplate kerberosRestTemplate = new KerberosRestTemplate( keytab,"'username#domain",requestFactory.getHttpClient());
kerberosRestTemplate.setErrorHandler(new KerberosRestTemplateResponseErrorHandler());
MappingJackson2HttpMessageConverter jsonHttpMessageConverter = new MappingJackson2HttpMessageConverter();
jsonHttpMessageConverter.getObjectMapper().configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false);
kerberosRestTemplate.getMessageConverters().add(jsonHttpMessageConverter);
// Add CSRF header if required:
HttpHeaders headers = new HttpHeaders();
headers.set("X-Requested-By", "'username#domain");
headers.setContentType(MediaType.APPLICATION_JSON);
headers.set("Client-Type","Web");
headers.set("Client-UI-Component","Jim");
HttpEntity<PostBody> postRequest = new HttpEntity<PostBody>(postRequestBody, headers);
PostResponse pr = kerberosRestTemplate.postForObject("https://url.com/1/data/loadList", postRequest, PostResponse.class);
}
}
package ErrorHandlers;
import org.springframework.http.HttpStatus;
import org.springframework.http.client.ClientHttpResponse;
import org.springframework.web.client.ResponseErrorHandler;
import java.io.IOException;
public class KerberosRestTemplateResponseErrorHandler implements ResponseErrorHandler {
#Override
public boolean hasError(ClientHttpResponse httpResponse) throws IOException {
return (
httpResponse.getStatusCode().series() == HttpStatus.Series.CLIENT_ERROR
|| httpResponse.getStatusCode().series() == HttpStatus.Series.SERVER_ERROR);
}
#Override
public void handleError(ClientHttpResponse httpResponse) throws IOException {
System.out.println("Error Response code " + httpResponse.getRawStatusCode());
System.out.println(httpResponse.getStatusText());
System.out.println(httpResponse.getBody().toString());
}
}
Updated with Partial logs
1:56:58.233 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager is shutting down
21:56:58.245 [Finalizer] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection manager shut down
21:58:30.261 [main] DEBUG org.springframework.security.kerberos.client.KerberosRestTemplate - HTTP POST myendpoint.com
21:58:30.460 [main] DEBUG org.springframework.security.kerberos.client.KerberosRestTemplate - Accept=[application/json, application/*+json]
21:58:30.500 [main] DEBUG org.springframework.security.kerberos.client.KerberosRestTemplate - Writing [ClassPojo [firstWeekday = 0, metadata = [Ljava.lang.String;#2f465398, endDate = 2119-03-30, modifiedDate = 2019-07-20, aggregation = DAILY, symbols = [Ljava.lang.String;#548e6d58, startDate = 1900-03-30]] as "application/json"
21:58:30.540 [main] DEBUG org.apache.http.client.protocol.RequestAddCookies - CookieSpec selected: default
21:58:30.561 [main] DEBUG org.apache.http.client.protocol.RequestAuthCache - Auth cache not set in the context
21:58:30.565 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {s}->myendpoint.com:443][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
21:58:30.587 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {s}->myendpoint.com:443][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
21:58:30.591 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Opening connection {s}->myendpoint.com:443
21:58:30.617 [main] DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator - Connecting to myendpoint.com:443
21:58:30.617 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - Connecting socket to myendpoint.com:443 with timeout 0
21:58:30.747 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
21:58:30.747 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
21:58:30.747 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - Starting handshake
21:58:30.923 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - Secure session established
21:58:30.923 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - negotiated protocol: TLSv1.2
21:58:30.924 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
21:58:30.933 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - peer principal: CN=myendpoint.com, O=EXAMPLE p.l.c.
21:58:30.934 [main] DEBUG org.apache.http.conn.ssl.SSLConnectionSocketFactory - peer alternative names: [myendpoint.com]
21:58:30.962 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Executing request POST /1/data/loadList HTTP/1.1
21:58:30.963 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED
21:58:30.968 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED
21:58:30.976 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> POST /1/data/loadList HTTP/1.1
21:58:30.979 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept: application/json, application/+json
21:58:30.979 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Client-Type: Fred
21:58:30.980 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Type: application/json
21:58:30.981 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Client-UI-Component: Jim
21:58:30.981 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 220
21:58:30.982 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: myendpoint.com
21:58:30.983 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
21:58:30.983 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.9 (Java/1.8.0_212-3-redhat)
21:58:30.983 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
21:58:30.984 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "POST /1/data/loadList HTTP/1.1[\r][\n]"
21:58:30.984 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept: application/json, application/+json[\r][\n]"
21:58:30.984 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Client-Type: Fred[\r][\n]"
21:58:30.985 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Type: application/json[\r][\n]"
21:58:30.986 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Client-UI-Component: Jim[\r][\n]"
21:58:30.986 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Content-Length: 220[\r][\n]"
21:58:30.986 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: myendpoint.com[\r][\n]"
21:58:30.988 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
21:58:30.990 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.9 (Java/1.8.0_212-3-redhat)[\r][\n]"
21:58:30.990 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
21:58:30.991 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
21:58:30.992 [main] DEBUG org.apache.http.wire - http-outgoing-0 >> "{"firstWeekday":"0","metadata":["All"],"endDate":"2119-03-30","modifiedDate":"2019-07-20","aggregation":"DAILY","symbols":["symbol1","symbol2"],"startDate":"1900-03-30"}"
21:58:31.043 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 401 [\r][\n]"
21:58:31.043 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Credentials: true[\r][\n]"
21:58:31.043 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE[\r][\n]"
21:58:31.043 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Max-Age: 3600[\r][\n]"
21:58:31.043 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Headers: Client-Type, Content-Type, Accept, X-Requested-With, Authorization, rejectunauthorized[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "WWW-Authenticate: Negotiate[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "X-Content-Type-Options: nosniff[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "X-XSS-Protection: 1; mode=block[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Cache-Control: no-cache, no-store, max-age=0, must-revalidate[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Pragma: no-cache[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Expires: 0[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Strict-Transport-Security: max-age=31536000 ; includeSubDomains[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "X-Frame-Options: DENY[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "Date: Wed, 11 Sep 2019 20:58:31 GMT[\r][\n]"
21:58:31.044 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
21:58:31.050 [main] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 401
21:58:31.050 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Credentials: true
21:58:31.050 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Max-Age: 3600
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Headers: Client-Type, Content-Type, Accept, X-Requested-With, Authorization, rejectunauthorized
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << WWW-Authenticate: Negotiate
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << X-Content-Type-Options: nosniff
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << X-XSS-Protection: 1; mode=block
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Cache-Control: no-cache, no-store, max-age=0, must-revalidate
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Pragma: no-cache
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Expires: 0
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Strict-Transport-Security: max-age=31536000 ; includeSubDomains
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << X-Frame-Options: DENY
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Transfer-Encoding: chunked
21:58:31.051 [main] DEBUG org.apache.http.headers - http-outgoing-0 << Date: Wed, 11 Sep 2019 20:58:31 GMT
21:58:31.063 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Connection can be kept alive indefinitely
21:58:31.063 [main] DEBUG org.apache.http.impl.auth.HttpAuthenticator - Authentication required
21:58:31.063 [main] DEBUG org.apache.http.impl.auth.HttpAuthenticator - myendpoint.com:443 requested authentication
21:58:31.063 [main] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Authentication schemes in the order of preference: [Negotiate, Kerberos, NTLM, CredSSP, Digest, Basic]
21:58:31.081 [main] DEBUG org.apache.http.impl.auth.SPNegoScheme - Received challenge '' from the auth server
21:58:31.082 [main] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for Kerberos authentication scheme not available
21:58:31.083 [main] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for NTLM authentication scheme not available
21:58:31.083 [main] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for CredSSP authentication scheme not available
21:58:31.083 [main] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for Digest authentication scheme not available
21:58:31.083 [main] DEBUG org.apache.http.impl.client.TargetAuthenticationStrategy - Challenge for Basic authentication scheme not available
21:58:31.093 [main] DEBUG org.springframework.security.kerberos.client.KerberosRestTemplate - Response 401 UNAUTHORIZED
Error Response code 401
21:58:31.096 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "0[\r][\n]"
21:58:31.097 [main] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]"
21:58:31.097 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {s}->myendpoint.com:443] can be kept alive indefinitely
21:58:31.097 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0
21:58:31.097 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {s}->myendpoint.com:443][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
Process finished with exit code 0
I am writing a clojure app that uses Amazon S3. It is using the standard amazonaws library, under this wrapper. For some reason, it is logging to System.out a ton of DEBUG info, including but not limited to the entire binary file I am sending to S3.
As of writing this, I have tried setting up a log4j.properties file:
# Direct log messages to NULL gdmnt
log4j.appender.devnull=org.apache.log4j.varia.NullAppender
log4j.rootLogger=fatal, devnull
log4j.logger.httpclient.wire.header=ERROR
log4j.logger.httpclient.wire.content=ERROR
org.appache.http=ERROR
org.appache.http.wire=ERROR
org.appache.http.headers=ERROR
log4j.logger.com.amazonaws=WARN
log4j.logger.com.amazonaws.request=WARN
log4j.logger.com.amazonaws.http=WARN
log4j.logger.org.apache.http.wire = WARN
Yet it still sends DEBUG info. Yes, log4j is on my classpath. Here is a printout of my classpath filtered for log4j
/Users/collinbell/.m2/repository/org/apache/logging/log4j/log4j-api/2.7/log4j-api-2.7.jar
/Users/collinbell/.m2/repository/org/apache/logging/log4j/log4j-core/2.7/log4j-core-2.7.jar
Here is an example of the logs produced when I send just a simple string to an S3 bucket.
17:53:52.092 [nREPL-worker-0] DEBUG c.a.internal.config.InternalConfig - Configuration override awssdk_config_override.json not found.
17:53:52.126 [nREPL-worker-0] DEBUG c.a.services.s3.internal.S3Signer - Calculated string to sign:
"PUT
text/plain; charset=UTF-8
Tue, 10 Jan 2017 22:53:52 GMT
/eden-cybernetics-voice/key"
17:53:52.225 [nREPL-worker-0] DEBUG com.amazonaws.request - Sending Request: PUT https://eden-cybernetics-voice.s3.amazonaws.com /key Headers: (Authorization: AWS YA,FUCKTHAT=, User-Agent: aws-sdk-java/1.7.5 Mac_OS_X/10.11.5 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Content-Length: 3, Date: Tue, 10 Jan 2017 22:53:52 GMT, Content-Type: text/plain; charset=UTF-8, )
17:53:52.326 [nREPL-worker-0] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Connection request: [route: {s}->https://eden-cybernetics-voice.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
17:53:52.356 [nREPL-worker-0] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Connection leased: [id: 0][route: {s}->https://eden-cybernetics-voice.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
17:53:52.889 [nREPL-worker-0] DEBUG o.a.h.i.c.DefaultClientConnectionOperator - Connecting to eden-cybernetics-voice.s3.amazonaws.com:443
17:53:53.145 [nREPL-worker-0] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default
17:53:53.147 [nREPL-worker-0] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
17:53:53.148 [nREPL-worker-0] DEBUG o.a.h.c.p.RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
17:53:53.149 [nREPL-worker-0] DEBUG c.a.http.impl.client.SdkHttpClient - Attempt 1 to execute request
17:53:53.150 [nREPL-worker-0] DEBUG o.a.h.i.conn.DefaultClientConnection - Sending request: PUT /key HTTP/1.1
17:53:53.151 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "PUT /key HTTP/1.1[\r][\n]"
17:53:53.153 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "Host: eden-cybernetics-voice.s3.amazonaws.com[\r][\n]"
17:53:53.154 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "Authorization: AWS YA, FUCKTHAT=[\r][\n]"
17:53:53.155 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "User-Agent: aws-sdk-java/1.7.5 Mac_OS_X/10.11.5 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102[\r][\n]"
17:53:53.155 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "Date: Tue, 10 Jan 2017 22:53:52 GMT[\r][\n]"
17:53:53.156 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17:53:53.156 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "Content-Length: 3[\r][\n]"
17:53:53.157 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "Connection: Keep-Alive[\r][\n]"
17:53:53.157 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "Expect: 100-continue[\r][\n]"
17:53:53.157 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "[\r][\n]"
17:53:53.158 [nREPL-worker-0] DEBUG org.apache.http.headers - >> PUT /key HTTP/1.1
17:53:53.158 [nREPL-worker-0] DEBUG org.apache.http.headers - >> Host: eden-cybernetics-voice.s3.amazonaws.com
17:53:53.158 [nREPL-worker-0] DEBUG org.apache.http.headers - >> Authorization: AWS YA, I AM NOT SHOWING YOU THIS=
17:53:53.159 [nREPL-worker-0] DEBUG org.apache.http.headers - >> User-Agent: aws-sdk-java/1.7.5 Mac_OS_X/10.11.5 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102
17:53:53.159 [nREPL-worker-0] DEBUG org.apache.http.headers - >> Date: Tue, 10 Jan 2017 22:53:52 GMT
17:53:53.159 [nREPL-worker-0] DEBUG org.apache.http.headers - >> Content-Type: text/plain; charset=UTF-8
17:53:53.159 [nREPL-worker-0] DEBUG org.apache.http.headers - >> Content-Length: 3
17:53:53.159 [nREPL-worker-0] DEBUG org.apache.http.headers - >> Connection: Keep-Alive
17:53:53.160 [nREPL-worker-0] DEBUG org.apache.http.headers - >> Expect: 100-continue
17:53:53.209 [nREPL-worker-0] DEBUG org.apache.http.wire - << "HTTP/1.1 100 Continue[\r][\n]"
17:53:53.213 [nREPL-worker-0] DEBUG org.apache.http.wire - << "[\r][\n]"
17:53:53.214 [nREPL-worker-0] DEBUG o.a.h.i.conn.DefaultClientConnection - Receiving response: HTTP/1.1 100 Continue
17:53:53.214 [nREPL-worker-0] DEBUG org.apache.http.headers - << HTTP/1.1 100 Continue
17:53:53.215 [nREPL-worker-0] DEBUG org.apache.http.wire - >> "key"
17:53:53.252 [nREPL-worker-0] DEBUG org.apache.http.wire - << "HTTP/1.1 200 OK[\r][\n]"
17:53:53.253 [nREPL-worker-0] DEBUG org.apache.http.wire - << "x-amz-id-2: KxZ8+cp4/BDEm4hdEjaVI/8tUaz0RRh6hcM041BB5pMlHXGXTAjJS3hA3VojH6H4UlleHmD2HSE=[\r][\n]"
17:53:53.253 [nREPL-worker-0] DEBUG org.apache.http.wire - << "x-amz-request-id: F83687CB8CDEB189[\r][\n]"
17:53:53.254 [nREPL-worker-0] DEBUG org.apache.http.wire - << "Date: Tue, 10 Jan 2017 22:53:54 GMT[\r][\n]"
17:53:53.254 [nREPL-worker-0] DEBUG org.apache.http.wire - << "ETag: "3c6e0b8a9c15224a8228b9a98ca1531d"[\r][\n]"
17:53:53.255 [nREPL-worker-0] DEBUG org.apache.http.wire - << "Content-Length: 0[\r][\n]"
17:53:53.256 [nREPL-worker-0] DEBUG org.apache.http.wire - << "Server: AmazonS3[\r][\n]"
17:53:53.257 [nREPL-worker-0] DEBUG org.apache.http.wire - << "[\r][\n]"
17:53:53.258 [nREPL-worker-0] DEBUG o.a.h.i.conn.DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
17:53:53.258 [nREPL-worker-0] DEBUG org.apache.http.headers - << HTTP/1.1 200 OK
17:53:53.258 [nREPL-worker-0] DEBUG org.apache.http.headers - << x-amz-id-2: KxZ8+cp4/BDEm4hdEjaVI/8tUaz0RRh6hcM041BB5pMlHXGXTAjJS3hA3VojH6H4UlleHmD2HSE=
17:53:53.259 [nREPL-worker-0] DEBUG org.apache.http.headers - << x-amz-request-id: F83687CB8CDEB189
17:53:53.259 [nREPL-worker-0] DEBUG org.apache.http.headers - << Date: Tue, 10 Jan 2017 22:53:54 GMT
17:53:53.259 [nREPL-worker-0] DEBUG org.apache.http.headers - << ETag: "3c6e0b8a9c15224a8228b9a98ca1531d"
17:53:53.259 [nREPL-worker-0] DEBUG org.apache.http.headers - << Content-Length: 0
17:53:53.260 [nREPL-worker-0] DEBUG org.apache.http.headers - << Server: AmazonS3
17:53:53.265 [nREPL-worker-0] DEBUG c.a.http.impl.client.SdkHttpClient - Connection can be kept alive indefinitely
17:53:53.276 [nREPL-worker-0] DEBUG com.amazonaws.request - Received successful response: 200, AWS Request ID: F83687CB8CDEB189
17:53:53.277 [nREPL-worker-0] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Connection [id: 0][route: {s}->https://eden-cybernetics-voice.s3.amazonaws.com:443] can be kept alive indefinitely
17:53:53.278 [nREPL-worker-0] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Connection released: [id: 0][route: {s}->https://eden-cybernetics-voice.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
#object[com.amazonaws.services.s3.model.PutObjectResult 0x3822e74d "com.amazonaws.services.s3.model.PutObjectResult#3822e74d"]
17:54:51.712 [java-sdk-http-connection-reaper] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Closing connections idle longer than 60 SECONDS
17:55:51.793 [java-sdk-http-connection-reaper] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Closing connections idle longer than 60 SECONDS
17:55:51.881 [java-sdk-http-connection-reaper] DEBUG o.a.h.i.conn.DefaultClientConnection - Connection 0.0.0.0:61322<->52.216.80.64:443 closed
17:56:51.896 [java-sdk-http-connection-reaper] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Closing connections idle longer than 60 SECONDS
17:57:51.909 [java-sdk-http-connection-reaper] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Closing connections idle longer than 60 SECONDS
17:58:51.918 [java-sdk-http-connection-reaper] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Closing connections idle longer than 60 SECONDS
17:59:51.929 [java-sdk-http-connection-reaper] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Closing connections idle longer than 60 SECOND
My current strategy is to first find out if appache.commons.logging is even using log4j, but I don't know how to do that.
Next, I want to check that log4j is loading in the log4j.properties file.
Is this a good strategy? How do I accomplish it? Do you know another strategy?
S3 under clojure is unusable as is.
Here is my summarized project.clj:
(defproject voice "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:plugins [[lein-cljsbuild "1.1.5"]
[lein-figwheel "0.5.8"]]
:dependencies [[org.clojure/clojure "1.8.0"]
[org.apache.logging.log4j/log4j-core "2.7"]
[org.apache.logging.log4j/log4j-api "2.7"]
;browser repl
[org.clojure/clojurescript "1.9.293"]
[lein-figwheel "0.5.8"]
[figwheel-sidecar "0.5.8"]
;explicitly get the right version
[instaparse "1.4.0"]
;for the trainer
[org.deeplearning4j/deeplearning4j-core "0.7.1"]
[org.deeplearning4j/deeplearning4j-ui_2.10 "0.7.1"]
;[org.nd4j/nd4j-cuda-7.5 "0.7.1"] ;for cuda
[org.nd4j/nd4j-native "0.7.1"]
;for the sampler
[clojure-opennlp "0.3.3"]
[compojure "1.5.1"]
[hiccup "1.0.5"]
[http-kit "2.2.0"]
[garden "1.3.2"]
;;for the sampler client
[jayq "2.5.4"]
[cljs-ajax "0.5.8"]
;for the database
[korma "0.4.3"]
[org.postgresql/postgresql "9.4.1212"]
[clj-aws-s3 "0.3.10" :exclusions [joda-time]]
[org.clojure/data.json "0.2.6"]]
:profiles {:dev {:dependencies [[alembic "0.3.2"]
[org.clojure/tools.nrepl "0.2.10"]
[com.cemerick/piggieback "0.2.1"]]}}
:repl-options {:nrepl-middleware [cemerick.piggieback/wrap-cljs-repl]}
:jvm-opts ["-Dlog4j.debug=true"]
:cljsbuild {
:builds [{
:source-paths ["src/cljs/"]
:id "dev"
:figwheel {:websocket-host "localhost"}
:compiler {
:main "voice.sampler"
:asset-path "js/out"
:output-to "resources/public/js/main.js"
:output-dir "resources/public/js/out"}}]})
Here is my fullish classpath:
/Users/collinbell/voice/test/
/Users/collinbell/voice/src/
/Users/collinbell/voice/dev-resources
/Users/collinbell/voice/resources/
/Users/collinbell/voice/target/classes/
/Users/collinbell/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.4.4/jackson-annotations-2.4.4.jar
/Users/collinbell/.m2/repository/org/nd4j/nd4j-jackson/0.7.1/nd4j-jackson-0.7.1.jar
/Users/collinbell/.m2/repository/net/jodah/typetools/0.4.3/typetools-0.4.3.jar
/Users/collinbell/.m2/repository/org/flatland/classlojure/0.7.0/classlojure-0.7.0.jar
/Users/collinbell/.m2/repository/com/mchange/mchange-commons-java/0.2.11/mchange-commons-java-0.2.11.jar
/Users/collinbell/.m2/repository/org/nd4j/jackson/0.7.1/jackson-0.7.1.jar
/Users/collinbell/.m2/repository/lein-as-resource/lein-as-resource/2.5.0/lein-as-resource-2.5.0.jar
/Users/collinbell/.m2/repository/org/apache/commons/commons-lang3/3.3.1/commons-lang3-3.3.1.jar
/Users/collinbell/.m2/repository/ring-cors/ring-cors/0.1.8/ring-cors-0.1.8.jar
/Users/collinbell/.m2/repository/org/clojure/tools.analyzer/0.6.9/tools.analyzer-0.6.9.jar
/Users/collinbell/.m2/repository/com/amazonaws/aws-java-sdk/1.7.5/aws-java-sdk-1.7.5.jar
/Users/collinbell/.m2/repository/ring/ring-core/1.5.0/ring-core-1.5.0.jar
/Users/collinbell/.m2/repository/org/nd4j/nd4j-common/0.7.1/nd4j-common-0.7.1.jar
/Users/collinbell/.m2/repository/org/bytedeco/javacpp-presets/leptonica/1.73-1.2/leptonica-1.73-1.2-macosx-x86_64.jar
/Users/collinbell/.m2/repository/com/typesafe/play/play-functional_2.10/2.4.6/play-functional_2.10-2.4.6.jar
/Users/collinbell/.m2/repository/com/google/jsinterop/jsinterop-annotations/1.0.0/jsinterop-annotations-1.0.0.jar
/Users/collinbell/.m2/repository/org/eclipse/collections/eclipse-collections-api/7.1.0/eclipse-collections-api-7.1.0.jar
/Users/collinbell/.m2/repository/com/typesafe/config/1.3.0/config-1.3.0.jar
/Users/collinbell/.m2/repository/com/stuartsierra/dependency/0.2.0/dependency-0.2.0.jar
/Users/collinbell/.m2/repository/org/bytedeco/javacpp-presets/leptonica/1.73-1.2/leptonica-1.73-1.2.jar
/Users/collinbell/.m2/repository/com/cemerick/piggieback/0.2.1/piggieback-0.2.1.jar
/Users/collinbell/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar
/Users/collinbell/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.5.3/jackson-dataformat-cbor-2.5.3.jar
/Users/collinbell/.m2/repository/org/eclipse/collections/eclipse-collections/7.1.0/eclipse-collections-7.1.0.jar
/Users/collinbell/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar
/Users/collinbell/.m2/repository/compojure/compojure/1.5.1/compojure-1.5.1.jar
/Users/collinbell/.m2/repository/org/springframework/spring-beans/4.1.6.RELEASE/spring-beans-4.1.6.RELEASE.jar
/Users/collinbell/.m2/repository/com/twelvemonkeys/imageio/imageio-core/3.1.1/imageio-core-3.1.1.jar
/Users/collinbell/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.4.4/jackson-core-2.4.4.jar
/Users/collinbell/.m2/repository/org/slf4j/jul-to-slf4j/1.7.12/jul-to-slf4j-1.7.12.jar
/Users/collinbell/.m2/repository/org/bytedeco/javacpp-presets/opencv/3.1.0-1.2/opencv-3.1.0-1.2-macosx-x86_64.jar
/Users/collinbell/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar
/Users/collinbell/.m2/repository/org/springframework/spring-context/4.1.6.RELEASE/spring-context-4.1.6.RELEASE.jar
/Users/collinbell/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
/Users/collinbell/.m2/repository/net/cgrand/sjacket/0.1.1/sjacket-0.1.1.jar
/Users/collinbell/.m2/repository/org/eclipse/collections/eclipse-collections-forkjoin/7.1.0/eclipse-collections-forkjoin-7.1.0.jar
/Users/collinbell/.m2/repository/org/apache/httpcomponents/httpclient/4.5.1/httpclient-4.5.1.jar
/Users/collinbell/.m2/repository/org/bytedeco/javacpp-presets/ffmpeg/3.0.2-1.2/ffmpeg-3.0.2-1.2.jar
/Users/collinbell/.m2/repository/org/agrona/Agrona/0.5.4/Agrona-0.5.4.jar
/Users/collinbell/.m2/repository/com/typesafe/play/play-exceptions/2.4.6/play-exceptions-2.4.6.jar
/Users/collinbell/.m2/repository/commons-codec/commons-codec/1.6/commons-codec-1.6.jar
/Users/collinbell/.m2/repository/com/stuartsierra/component/0.3.1/component-0.3.1.jar
/Users/collinbell/.m2/repository/org/apache/directory/studio/org.apache.commons.codec/1.8/org.apache.commons.codec-1.8.jar
/Users/collinbell/.m2/repository/cider/cider-nrepl/0.15.0-SNAPSHOT/cider-nrepl-0.15.0-SNAPSHOT.jar
/Users/collinbell/.m2/repository/org/bytedeco/javacpp-presets/ffmpeg/3.0.2-1.2/ffmpeg-3.0.2-1.2-windows-x86_64.jar
/Users/collinbell/.m2/repository/org/apache/commons/commons-compress/1.8/commons-compress-1.8.jar
/Users/collinbell/.m2/repository/clj-aws-s3/clj-aws-s3/0.3.10/clj-aws-s3-0.3.10.jar
/Users/collinbell/.m2/repository/org/postgresql/postgresql/9.4.1212/postgresql-9.4.1212.jar
/Users/collinbell/.m2/repository/org/bytedeco/javacpp-presets/leptonica/1.73-1.2/leptonica-1.73-1.2-linux-x86_64.jar
/Users/collinbell/.m2/repository/org/ow2/asm/asm-all/4.2/asm-all-4.2.jar
/Users/collinbell/.m2/repository/com/twelvemonkeys/common/common-image/3.1.1/common-image-3.1.1.jar
/Users/collinbell/.m2/repository/garden/garden/1.3.2/garden-1.3.2.jar
/Users/collinbell/.m2/repository/com/twelvemonkeys/imageio/imageio-psd/3.1.1/imageio-psd-3.1.1.jar
/Users/collinbell/.m2/repository/org/nd4j/nd4j-buffer/0.7.1/nd4j-buffer-0.7.1.jar
/Users/collinbell/.m2/repository/io/netty/netty/3.10.4.Final/netty-3.10.4.Final.jar
/Users/collinbell/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar
/Users/collinbell/.m2/repository/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar
/Users/collinbell/.m2/repository/org/clojure/data.json/0.2.6/data.json-0.2.6.jar
/Users/collinbell/.m2/repository/org/deeplearning4j/deeplearning4j-core/0.7.1/deeplearning4j-core-0.7.1.jar
/Users/collinbell/.m2/repository/org/clojure/data.priority-map/0.0.7/data.priority-map-0.0.7.jar
/Users/collinbell/.m2/repository/com/twelvemonkeys/imageio/imageio-bmp/3.1.1/imageio-bmp-3.1.1.jar
/Users/collinbell/.m2/repository/cljs-ajax/cljs-ajax/0.5.8/cljs-ajax-0.5.8.jar
/Users/collinbell/.m2/repository/org/clojure/tools.reader/1.0.0-beta3/tools.reader-1.0.0-beta3.jar
/Users/collinbell/.m2/repository/org/tcrawley/dynapath/0.2.5/dynapath-0.2.5.jar
/Users/collinbell/.m2/repository/xerces/xercesImpl/2.11.0/xercesImpl-2.11.0.jar
/Users/collinbell/.m2/repository/org/nd4j/nd4j-api/0.7.1/nd4j-api-0.7.1.jar
/Users/collinbell/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar
/Users/collinbell/.m2/repository/org/projectlombok/lombok/1.16.10/lombok-1.16.10.jar
/Users/collinbell/.m2/repository/com/yahoo/platform/yui/yuicompressor/2.4.8/yuicompressor-2.4.8.jar
/Users/collinbell/.m2/repository/org/yaml/snakeyaml/1.12/snakeyaml-1.12.jar
/Users/collinbell/.m2/repository/crypto-random/crypto-random/1.2.0/crypto-random-1.2.0.jar
/Users/collinbell/.m2/repository/org/mapdb/elsa/3.0.0-M5/elsa-3.0.0-M5.jar
/Users/collinbell/.m2/repository/org/deeplearning4j/deeplearning4j-ui-model/0.7.1/deeplearning4j-ui-model-0.7.1.jar
/Users/collinbell/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
/Users/collinbell/.m2/repository/cheshire/cheshire/5.5.0/cheshire-5.5.0.jar
/Users/collinbell/.m2/repository/org/clojure/tools.nrepl/0.2.12/tools.nrepl-0.2.12.jar
/Users/collinbell/.m2/repository/com/google/code/findbugs/annotations/2.0.1/annotations-2.0.1.jar
/Users/collinbell/.m2/repository/org/deeplearning4j/deeplearning4j-nlp/0.7.1/deeplearning4j-nlp-0.7.1.jar
/Users/collinbell/.m2/repository/com/cognitect/transit-java/0.8.311/transit-java-0.8.311.jar
/Users/collinbell/.m2/repository/org/mapdb/mapdb/3.0.2/mapdb-3.0.2.jar
/Users/collinbell/.m2/repository/org/apache/logging/log4j/log4j-core/2.7/log4j-core-2.7.jar
/Users/collinbell/.m2/repository/org/mozilla/rhino/1.7R5/rhino-1.7R5.jar
/Users/collinbell/.m2/repository/strictly-specking-standalone/strictly-specking-standalone/0.1.1/strictly-specking-standalone-0.1.1.jar
/Users/collinbell/.m2/repository/com/typesafe/play/play-iteratees_2.10/2.4.6/play-iteratees_2.10-2.4.6.jar
/Users/collinbell/.m2/repository/com/twelvemonkeys/common/common-io/3.1.1/common-io-3.1.1.jar
/Users/collinbell/.m2/repository/org/nd4j/nd4j-native/0.7.1/nd4j-native-0.7.1.jar
/Users/collinbell/.m2/repository/ch/qos/logback/logback-core/1.1.3/logback-core-1.1.3.jar
/Users/collinbell/.m2/repository/http-kit/http-kit/2.2.0/http-kit-2.2.0.jar
/Users/collinbell/.m2/repository/org/jboss/logging/jboss-logging/3.2.1.Final/jboss-logging-3.2.1.Final.jar
/Users/collinbell/.m2/repository/org/bytedeco/javacpp-presets/leptonica/1.73-1.2/leptonica-1.73-1.2-windows-x86_64.jar
/Users/collinbell/.m2/repository/org/msgpack/msgpack/0.6.10/msgpack-0.6.10.jar
/Users/collinbell/.m2/repository/clojure-opennlp/clojure-opennlp/0.3.3/clojure-opennlp-0.3.3.jar
/Users/collinbell/.m2/repository/net/cgrand/parsley/0.9.3/parsley-0.9.3.jar
/Users/collinbell/.m2/repository/org/hibernate/hibernate-validator/5.0.3.Final/hibernate-validator-5.0.3.Final.jar
/Users/collinbell/.m2/repository/commons-fileupload/commons-fileupload/1.3.1/commons-fileupload-1.3.1.jar
/Users/collinbell/.m2/repository/net/incongru/watchservice/barbary-watchservice/1.0/barbary-watchservice-1.0.jar
/Users/collinbell/.m2/repository/com/google/guava/guava/19.0/guava-19.0.jar
/Users/collinbell/.m2/repository/com/mchange/c3p0/0.9.5.2/c3p0-0.9.5.2.jar
/Users/collinbell/.m2/repository/org/clojure/core.cache/0.6.5/core.cache-0.6.5.jar
/Users/collinbell/.m2/repository/org/bytedeco/javacpp-presets/ffmpeg/3.0.2-1.2/ffmpeg-3.0.2-1.2-linux-x86_64.jar
/Users/collinbell/.m2/repository/org/bytedeco/javacpp-presets/ffmpeg/3.0.2-1.2/ffmpeg-3.0.2-1.2-macosx-x86_64.jar
/Users/collinbell/.m2/repository/net/sf/jwordnet/jwnl/1.3.3/jwnl-1.3.3.jar
/Users/collinbell/.m2/repository/org/scala-lang/modules/scala-java8-compat_2.10/0.3.0/scala-java8-compat_2.10-0.3.0.jar
/Users/collinbell/.m2/repository/org/apache/httpcomponents/httpasyncclient/4.1.1/httpasyncclient-4.1.1.jar
/Users/collinbell/.m2/repository/com/twelvemonkeys/imageio/imageio-jpeg/3.1.1/imageio-jpeg-3.1.1.jar
/Users/collinbell/.m2/repository/clout/clout/2.1.2/clout-2.1.2.jar
/Users/collinbell/.m2/repository/clj-stacktrace/clj-stacktrace/0.2.8/clj-stacktrace-0.2.8.jar
/Users/collinbell/.m2/repository/com/typesafe/play/twirl-api_2.10/1.1.1/twirl-api_2.10-1.1.1.jar
/Users/collinbell/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar
/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/src.zip
/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/lib/tools.jar
/Users/collinbell/.m2/repository/org/apache/logging/log4j/log4j-api/2.7/log4j-api-2.7.jar
This isnt an answer but I cant comment with my low reputation :(. The spelling for the 'apache' namespace seems to be wrong in the config
org.appache.http=ERROR
org.appache.http.wire=ERROR
org.appache.http.headers=ERROR
I had the same issue,
creating a logback.xml and putting it on classpath with below config fixed it:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<logger name="org.apache" level="ERROR" />
<logger name="httpclient" level="ERROR" />
</configuration>
Hope it helps the others.