AWS PutObject Connection reset - java

My AWS Java Client is throwing
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541) ~[na:1.8.0_60]
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553) ~[na:1.8.0_60]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) ~[na:1.8.0_60]
at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159) ~[httpcore-4.3.3.jar:4.3.3]
My code is
public void save(String name, byte[] file) {
ObjectMetadata metaData = new ObjectMetadata();
String streamMD5 = new String(Base64.encodeBase64(file));
metaData.setContentMD5(streamMD5);
metaData.setContentLength(file.length);
InputStream stream = new ByteArrayInputStream(file);
try {
PutObjectRequest put = new PutObjectRequest(
configuration.getBucketName(), name, stream, metaData);
s3client.putObject(put);
} finally {
IOUtils.closeQuietly(stream);
}
}
The s3client is a spring bean and is not garbage collected before the stream has finished uploading. I've tried without specifying MD5 and/or content length but still has the same exception thrown.
Logging through AWS library shows:
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: best-match
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.c.p.RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
10:09:15.540 [http-nio-8080-exec-1] DEBUG c.a.http.impl.client.SdkHttpClient - Attempt 1 to execute request
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.i.conn.DefaultClientConnection - Sending request: PUT /documeent.pdf HTTP/1.1
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "PUT /document.pdf HTTP/1.1[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "Host: bucket.s3.amazonaws.com[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "Authorization: AWS 123445667788=[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "User-Agent: aws-sdk-java/1.10.21 Linux/3.13.0-65-generic Java_HotSpot(TM)_Server_VM/25.60-b23/1.8.0_60[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "Date: Tue, 06 Oct 2015 09:09:15 GMT[\r][\n]"
10:09:15.663 [http-nio-8080-exec-1] DEBUG com.amazonaws.internal.SdkSSLSocket - closing bucket.s3.amazonaws.com/12.34.56.78:443
10:09:15.665 [http-nio-8080-exec-1] DEBUG o.a.h.i.conn.DefaultClientConnection - I/O error closing connection
I've checked that the file size (3.2M) does not exceed the max file siez for this bucket.
Get/List requests work fine and I can copy files into the S3 bucket using the s3 client tools.
Does anyone know of anything else I should check?
Thanks.

Related

How make feign raw http request/response logging?

I have enabled feign logging andgot some thing like that:
15:27:15.420 [ForkJoinPool-1-worker-229] DEBUG feign.Logger - [UserEndpoint#postUser] Authorization: Bearer test
15:27:15.420 [ForkJoinPool-1-worker-229] DEBUG feign.Logger - [UserEndpoint#postUser] Content-Length: 19
15:27:15.420 [ForkJoinPool-1-worker-229] DEBUG feign.Logger - [UserEndpoint#postUser] Content-Type: application/json
15:27:15.420 [ForkJoinPool-1-worker-229] DEBUG feign.Logger - [UserEndpoint#postUser]
15:27:15.420 [ForkJoinPool-1-worker-229] DEBUG feign.Logger - [UserEndpoint#postUser] {"attributes":null}
15:27:15.420 [ForkJoinPool-1-worker-229] DEBUG feign.Logger - [UserEndpoint#postUser] ---> END HTTP (19-byte body)
15:27:16.585 [ForkJoinPool-1-worker-117] DEBUG feign.Logger - [UserEndpoint#getUserById] <--- HTTP/1.1 500 Internal Server Error (1223ms)
15:27:16.585 [ForkJoinPool-1-worker-117] DEBUG feign.Logger - [UserEndpoint#getUserById] apigw-requestid: RpKPuhaFPHcESkA=
15:27:16.585 [ForkJoinPool-1-worker-117] DEBUG feign.Logger - [UserEndpoint#getUserById] connection: keep-alive
15:27:16.585 [ForkJoinPool-1-worker-117] DEBUG feign.Logger - [UserEndpoint#getUserById] content-length: 35
15:27:16.585 [ForkJoinPool-1-worker-117] DEBUG feign.Logger - [UserEndpoint#getUserById] content-type: application/json
15:27:16.585 [ForkJoinPool-1-worker-117] DEBUG feign.Logger - [UserEndpoint#getUserById] date: Thu, 05 May 2022 08:27:16 GMT
15:27:16.585 [ForkJoinPool-1-worker-117] DEBUG feign.Logger - [UserEndpoint#getUserById]
Ishtere a way to output like that:
GET https://myhost/api/users/eac92ab2-682b-45d9-8aa2-435f933b2661
Authorization: mytoken
E.g. one request - one piece of output in one place (not line by header)
Ideally there request and response should be output tohether, idealy in single line with escaped line breaks
Isit possible with feign?

Spring WebClient downloading PDF gives an HTTP error

While using Spring's WebClient to retrieve a PDF file from a REST API, I get an error.
Here's the code with the WebClient :
return WebClient.create().get()
.uri(builder.build().toUri())
.accept(MediaType.APPLICATION_PDF)
.exchange()
.flatMap(response -> response.bodyToMono(byte[].class))
.block();
And I'm getting this error from the REST API that serves the file :
03-02-2021 14:32:01.180 [http-nio-8080-exec-6] DEBUG o.s.w.s.m.m.a.HttpEntityMethodProcessor.writeWithMessageConverters - Found 'Content-Type:application/pdf' in response
03-02-2021 14:32:01.181 [http-nio-8080-exec-6] DEBUG o.s.w.s.m.m.a.HttpEntityMethodProcessor.traceDebug - Writing [InputStream resource [resource loaded through InputStream]]
03-02-2021 14:32:01.185 [http-nio-8080-exec-6] DEBUG o.s.o.j.s.OpenEntityManagerInViewInterceptor.afterCompletion - Closing JPA EntityManager in OpenEntityManagerInViewInterceptor
03-02-2021 14:32:01.186 [http-nio-8080-exec-6] DEBUG o.s.web.servlet.DispatcherServlet.logResult - Completed 200 OK
03-02-2021 14:32:01.187 [http-nio-8080-exec-6] DEBUG o.a.t.util.net.SocketWrapperBase.log - Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#d1d3000:org.apache.tomcat.util.net.NioChannel#4b5f2cea:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8080 remote=/127.0.0.1:57797]], Read from buffer: [0]
03-02-2021 14:32:01.187 [http-nio-8080-exec-6] DEBUG o.a.coyote.http11.Http11Processor.log - Error parsing HTTP request header
java.io.EOFException: null
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1230)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1140)
at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:780)
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:356)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
03-02-2021 14:32:01.188 [http-nio-8080-exec-6] DEBUG o.a.coyote.http11.Http11Processor.log - Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#d1d3000:org.apache.tomcat.util.net.NioChannel#4b5f2cea:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8080 remote=/127.0.0.1:57797]], Status in: [OPEN_READ], State out: [CLOSED]
03-02-2021 14:32:01.188 [http-nio-8080-exec-6] DEBUG o.a.coyote.http11.Http11NioProtocol.log - Pushed Processor [org.apache.coyote.http11.Http11Processor#1c06182a]
03-02-2021 14:32:01.188 [http-nio-8080-exec-6] DEBUG o.a.tomcat.util.threads.LimitLatch.log - Counting down[http-nio-8080-exec-6] latch=2
03-02-2021 14:32:01.188 [http-nio-8080-exec-6] DEBUG o.apache.tomcat.util.net.NioEndpoint.log - Calling [org.apache.tomcat.util.net.NioEndpoint#6df434e4].closeSocket([org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#d1d3000:org.apache.tomcat.util.net.NioChannel#4b5f2cea:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8080 remote=/127.0.0.1:57797]])
03-02-2021 14:32:08.051 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool.logPoolState - HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
03-02-2021 14:32:08.051 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool.fillPool - HikariPool-1 - Fill pool skipped, pool is at sufficient level.
03-02-2021 14:32:31.180 [Catalina-utility-2] DEBUG o.a.catalina.session.ManagerBase.log - Start expire sessions StandardManager at 1612359151180 sessioncount 0
03-02-2021 14:32:31.180 [Catalina-utility-2] DEBUG o.a.catalina.session.ManagerBase.log - End expire sessions StandardManager processingTime 0 expired sessions: 0
03-02-2021 14:32:34.196 [http-nio-8080-exec-1] DEBUG o.a.coyote.http11.Http11NioProtocol.log - Processing socket [org.apache.tomcat.util.net.NioChannel#6a7679d5:java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:8080 remote=/0:0:0:0:0:0:0:1:57771]] with status [ERROR]
03-02-2021 14:32:34.196 [http-nio-8080-exec-1] DEBUG o.a.coyote.http11.Http11NioProtocol.log - Found processor [null] for socket [org.apache.tomcat.util.net.NioChannel#6a7679d5:java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:8080 remote=/0:0:0:0:0:0:0:1:57771]]
03-02-2021 14:32:34.196 [http-nio-8080-exec-1] DEBUG o.a.tomcat.util.threads.LimitLatch.log - Counting down[http-nio-8080-exec-1] latch=1
03-02-2021 14:32:34.196 [http-nio-8080-exec-1] DEBUG o.apache.tomcat.util.net.NioEndpoint.log - Calling [org.apache.tomcat.util.net.NioEndpoint#6df434e4].closeSocket([org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#1bf3167e:org.apache.tomcat.util.net.NioChannel#6a7679d5:java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:8080 remote=/0:0:0:0:0:0:0:1:57771]])
Any idea where the problem is ?
Thanx

REST-assured JSON output

I am trying to retrieve a JSON file, but it doesn't seem to be a JSON file, but it outputs application/json
12:39:39.403[main] DEBUG org.apache.http.wire - << "[0xb9][0x12][0x3d][0xr4][0xu6][0x43][0xd9][0x12][0xw3][0x43][0xb6][0x1q][0xef][0x13]"
12:39:39.403[main] DEBUG org.apache.http.wire - << "[0xwq][0xq2][0x9d][0x14][0xk6][0xw3][0x9q][0xz2][0xm3][0x4m][0xq6][0xkq][0xlf][0xq3]"
12:39:39.403[main] DEBUG org.apache.http.wire - << "30[\r][\n]"
12:39:39.403[main] DEBUG org.apache.http.wire - << "[0xb9][0x12][0x3d][0xr4][0xu6][0x43][0x9][0x12][0xw3][0x43][0xb6][0x1q][0xef][0x13]"
My rest assured code is
Response response = given().contentType(ContentType.JSON).auth().get("https://sample-url");
I would I like to see an actual JSON output

CloseableHttpAsyncClient not send out http request

I want to use CloseableHttpAsyncClient to send a http request in Async mode. But the request is not send out. When enable comment HttpResponse response = future.get(). It works. But I'd like to know why I need future.get() even I didn't care about the response.
Code is here
public class CloseableHttpAsyncClientTest {
#Test
public void whenUseHttpAsyncClient_thenCorrect() throws Exception {
CloseableHttpAsyncClient client = HttpAsyncClients.createDefault();
client.start();
HttpDelete request = new HttpDelete("http://www.bing.com");
Future<HttpResponse> future = client.execute(request, null);
// The delete request will send out if we remove comment here. We just want to send out delete http
// request but not care about the response
// HttpResponse response = future.get();
client.close();
}
}
The console like this
20:28:48.930 [main] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 1] start execution
20:28:48.941 [main] DEBUG org.apache.http.client.protocol.RequestAddCookies - CookieSpec selected: default
20:28:48.950 [main] DEBUG org.apache.http.client.protocol.RequestAuthCache - Auth cache not set in the context
20:28:48.951 [main] DEBUG org.apache.http.impl.nio.client.InternalHttpAsyncClient - [exchange: 1] Request connection for {}->http://www.bing.com:80
20:28:48.953 [main] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection request: [route: {}->http://www.bing.com:80][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
20:28:48.976 [main] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection manager is shutting down
20:28:49.003 [main] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection manager shut down
After I enable the comments. Like this. It works.
public class CloseableHttpAsyncClientTest {
#Test
public void whenUseHttpAsyncClient_thenCorrect() throws Exception {
CloseableHttpAsyncClient client = HttpAsyncClients.createDefault();
client.start();
HttpDelete request = new HttpDelete("http://www.bing.com");
Future<HttpResponse> future = client.execute(request, null);
// The delete request will send out if we remove comment here. We just want to send out delete http
// request but not care about the response
HttpResponse response = future.get();
client.close();
}
}
From the log, we can see. The http delete has been send out.
20:39:15.998 [main] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 1] start execution
......
......
20:39:16.137 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> DELETE / HTTP/1.1
20:39:16.137 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: www.bing.com
......
......
20:39:16.142 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "DELETE / HTTP/1.1[\r][\n]"
20:39:16.142 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: www.bing.com[\r][\n]"
20:39:16.142 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
20:39:16.142 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_202)[\r][\n]"
20:39:16.143 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
20:39:16.143 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch - http-outgoing-0 [ACTIVE] Request ready
20:39:16.143 [I/O dispatcher 1] DEBUG
......
20:39:16.209 [main] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection manager shut down

Future<HttpResponse> future = client.execute(request, null);
client.execute will send out http request in Async way. It will take another thread to complete the http request.
HttpResponse response = future.get();
future.get() will block the main thread until client.execute complete and fill the response into future. Without this, the client will be close before http request send out. Cos http request send out by one thread, but client close be closed by main thread. So, main thread need be blocked until future.get() get the result by another thread.

Proxy-Authorization header is not send within first request in Apache HttpComponents

I'm using Apache HttpComponents (4.5.2) and I'm trying to request HTTPS page via proxy server (SSH Tunneling).
The problem is that according to logs Client sends first request without Proxy-Authorization header, but after Proxy respond with 407 error (Proxy Authentication Required), it retires authentication with sending Proxy-Authorization header.
I think problem in my code, I need something like enabling primitive auth , but I couldn't find any information about how to do that.
Below is logs for confirming my words.
First request:
03:12:06,643 DEBUG headers:135 - http-outgoing-0 >> CONNECT t.myhost.com:443 HTTP/1.1
03:12:06,643 DEBUG headers:138 - http-outgoing-0 >> Host: t.myhost.com
03:12:06,643 DEBUG headers:138 - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_45)
03:12:06,793 DEBUG headers:124 - http-outgoing-0 << HTTP/1.1 407 Proxy Authentication Required
03:12:06,794 DEBUG headers:127 - http-outgoing-0 << Proxy-Authenticate: Basic realm="ProxyCompany"
03:12:06,794 DEBUG headers:127 - http-outgoing-0 << Proxy-Connection: close
// then it retries request with included Proxy-Authorization header
03:12:06,795 DEBUG HttpAuthenticator:77 - Authentication required
03:12:06,795 DEBUG HttpAuthenticator:107 - 162.243.116.56:71223 requested authentication
03:12:06,795 DEBUG ProxyAuthenticationStrategy:174 - Authentication schemes in the order of preference: [Negotiate, Kerberos, NTLM, Digest, Basic]
03:12:06,795 DEBUG ProxyAuthenticationStrategy:203 - Challenge for Negotiate authentication scheme not available
03:12:06,796 DEBUG ProxyAuthenticationStrategy:203 - Challenge for Kerberos authentication scheme not available
03:12:06,796 DEBUG ProxyAuthenticationStrategy:203 - Challenge for NTLM authentication scheme not available
03:12:06,796 DEBUG ProxyAuthenticationStrategy:203 - Challenge for Digest authentication scheme not available
03:12:06,800 DEBUG HttpAuthenticator:157 - Selected authentication options: [BASIC [complete=true]]
03:12:06,800 DEBUG DefaultManagedHttpClientConnection:81 - http-outgoing-0: Close connection
03:12:06,801 DEBUG DefaultHttpClientConnectionOperator:138 - Connecting to /162.243.116.56:71223
03:12:06,942 DEBUG DefaultHttpClientConnectionOperator:145 - Connection established 192.168.0.100:13391<->162.243.116.56:71223
03:12:06,942 DEBUG HttpAuthenticator:198 - Generating response to an authentication challenge using basic scheme
03:12:06,947 DEBUG headers:135 - http-outgoing-0 >> CONNECT t.myhost.com:443 HTTP/1.1
03:12:06,947 DEBUG headers:138 - http-outgoing-0 >> Host: t.myhost.com
03:12:06,947 DEBUG headers:138 - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_45)
03:12:06,947 DEBUG headers:138 - http-outgoing-0 >> Proxy-Authorization: Basic bHVtXXXXXXXXXXXXxOTE5NTUXXXXXXRmNmRkYmI1Mjk0MA==
03:12:07,304 DEBUG HttpAuthenticator:86 - Authentication succeeded
03:12:07,305 DEBUG ProxyAuthenticationStrategy:227 - Caching 'basic' auth scheme for http://162.243.116.56:71223
And this is my code (it's Scala, but pretty easy to read):
val credProvider = {
val provider = new BasicCredentialsProvider()
provider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials("myUser", "myPass"))
provider
}
val connManager = {
val mngr = new PoolingHttpClientConnectionManager()
mngr.setDefaultMaxPerRoute(Integer.MAX_VALUE)
mngr.setMaxTotal(Integer.MAX_VALUE)
mngr
}
val client = HttpClients.custom()
.setConnectionManager(connManager)
.disableRedirectHandling()
.setDefaultCredentialsProvider(credProvider)
.setProxy(new HttpHost(162.243.116.56, 71223 ))
.build()
val requestConfig = RequestConfig.custom()
.setConnectTimeout(30000)
.setConnectionRequestTimeout(30000)
.build()
val request = new HttpGet(url)
request.setConfig(requestConfig)
val response = client.execute(request)
How I can solve this problem (cause client to always send Proxy-Authorization )?

I'm not sure if it is the same problem, but with version 4.5.2 there was a bug introduced with the SPN (HTTP/something#somerealm) and https : HTTPCLIENT-1712 (the comments are especially interesting since they show a history of what happened).
A switch to version 4.5.1 should solve it (if this is the same problem of course)

Categories

Resources