I am using Apache HTTP Components to send a POST request to a server that is not owned by me (therefore I cannot modify its configuration).
The server, in response back to me, sets Content-Encoding: none header which causes the org.apache.http.HttpException: Unsupported Content-Coding: none exception.
I found somewhere on StackOverflow that I can use HttpClients.custom().disableContentCompression().build() to disable RequestAcceptEncoding and ResponseContentEncoding interceptors, therefore making the exception not to be thrown. However, I do not want to reconfigure my HttpClient in that way globally.
Is there any workaround of this issue without adding changes to HttpClient?
As a note, I do not have to read response body at all, as it's empty. I just need to be able to read response code (ie. 200 OK).
Thank you for your time.
Finally after months I found the solution for this problem!
There is a workaround with using interceptors to fix the bad Content-Encoding header, see this blog entry.
I met similar problem. A simple fix is to disable compression in the http client builder.
HttpClientBuilder clientBuilder = HttpClients.custom();
clientBuilder.disableContentCompression();
HttpClient client = clientBuilder.build();
Related
I write a java program like I saw here
How to read the https page content using java?
but for some sites the code does not work.
I got Error Server returned HTTP response code: 403 for URL: https://research.investors.com/stock-quotes/nyse-sailpoint-tech-holdings-sail.htm
It works for
url = "https://maven.apache.org/guides/mini/guide-repository-ssl.html";
Can someone help me ?
403 HTTP status stands for "Forbidden", most likely investors.com can check your request headers and deny the resource.
Try modifying the request headers using an User-Agent that site might accept.
403 Forbidden
The request contained valid data and was understood by the server, but the server is refusing action. This may be due to the user not having the necessary permissions for a resource or needing an account of some sort, or attempting a prohibited action (e.g. creating a duplicate record where only one is allowed). This code is also typically used if the request provided authentication by answering the WWW-Authenticate header field challenge, but the server did not accept that authentication. The request should not be repeated.
So probably website, which you want to scrape, just restricted requests like yours (i mean requests, that was made not from browser).
But you can try Selenium.
OK , I solved.
I use con.setRequestProperty and set "User-Agent", "Accept", "Content-Type", "Accept-Language".
Thank you.
I've been experimenting with the HttpClient stuff in the Java 9/10 incubator, and have the following trivial code (virtually stolen from the project home page!):
URI uri = URI.create("http://192.168.1.102:8080/");
HttpRequest getRequest = HttpRequest.newBuilder()
.uri(uri)
.GET()
.build();
HttpResponse<String> response = client.send(getRequest,
HttpResponse.BodyHandler.asString());
System.out.println("response to get: " + response.body());
I find it works fine if it's pointed at a URL that is not the localhost, but fails if I ask for the localhost (whether by the name "localhost", by 172.0.0.1, or by the actual IP address of the local host). The error is very strange, and the entire stack trace does not mention any of my code.
WARNING: Using incubator modules: jdk.incubator.httpclient
Exception in thread "main" java.io.EOFException: EOF reached while reading
at jdk.incubator.httpclient/jdk.incubator.http.Http1AsyncReceiver$Http1TubeSubscriber.onComplete(Http1AsyncReceiver.java:507)
at jdk.incubator.httpclient/jdk.incubator.http.SocketTube$InternalReadPublisher$ReadSubscription.signalCompletion(SocketTube.java:551)
at jdk.incubator.httpclient/jdk.incubator.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:728)
at jdk.incubator.httpclient/jdk.incubator.http.SocketTube$SocketFlowTask.run(SocketTube.java:171)
at jdk.incubator.httpclient/jdk.incubator.http.internal.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
at jdk.incubator.httpclient/jdk.incubator.http.internal.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
at jdk.incubator.httpclient/jdk.incubator.http.internal.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
at jdk.incubator.httpclient/jdk.incubator.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:675)
at jdk.incubator.httpclient/jdk.incubator.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:829)
at jdk.incubator.httpclient/jdk.incubator.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:243)
at jdk.incubator.httpclient/jdk.incubator.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:769)
at jdk.incubator.httpclient/jdk.incubator.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:731)
There is a server running locally, and I can connect to it just fine using a simple request from a web browser.
Any thoughts?
[EDIT]I found, I beleive, the mail list for this project. It's "obfuscated" (which fooled me completely!) but shown as: net dash dev at openjdk dot java dot net I'll post there too, and see if they have any input.
[EDIT 2]I'm pretty sure that this has nothing to do with localhost (per original title) but is something in the protocol negotiation with node.js/express (which is the server I'm using because it's easy to experiment with). Node occasionally (e.g. with a last line of text that's not LF terminated) seems to report the wrong content-length, but this isn't the problem, as the failure still occurs with correct length. I think it's possibly a bug in the attempt to upgrade the connection to HTTP/2.0 but don't know yet...
[EDIT 3]After wasting way too much of my life experimenting, I'm fairly sure that there's something in the way node.js 8.11.1 (and express 4.13.4 and body-parser 1.15.1) handle a request to upgrade a to HTTP 2.0 that's causing the problem. But I have no idea what. I'm giving up, and will continue the learning process for httpClient using a different server.
Updated. I finally got curl built with http 2.0 support, and the blame is entirely on node/express. When this server sees an upgrade request (node 8.something) it simply fails to create any output.Consequently, the client correctly fails with an EOF error.
As a side note, node/express also sets the content-length header "off by one" on occasions (not always!?)
try this
HttpRequest request = HttpRequest.newBuilder()
.uri(new URI("http://localhost:3000"))
.POST(BodyPublisher.fromString("hello"))
.version(Version.HTTP_1_1).build();
While searching the solution for this issue, I read somewhere that max size of get request is 8Kb. However when I am trying to execute get request of content length of only 248 bytes and total URL length of only 282 characters through Apache HttpClient execute method, Apache HttpClient is giving me error: org.apache.http.HttpException: HTTP/1.1 413 Request Entity Too Large.
However the same get request (the same URL) gives expected response in browser (and NOT "413 Request Entity Too Large").
Apache HttpClient execute method is working fine for some other get request which is slightly smaller in length and has lesser no. of query params.
I also tried sending the Post request but still got the same error.
Please help me resolve this issue. Any help will be appreciated
The other seemingly similar questions didn't solve my problem.
Request Entity too large error, comes due to server receiving request that are larger than configured to process. It should not be from the client side, you need to modify you server setting to allow larger request body length. This parameter will differ server to server.
Some of them are listed here
Sorry the issue was not on client side. But the internal API that I am using was sending incorrect 413 response code instead of (almost) correct response code 507.
I meet this error when send a request and get back response with code 401:
com.android.volley.NoConnectionError: java.io.IOException: No authentication challenges found
Some people say that:
This error happens beause the server sends a 401 (Unauthorized) but does not give a "WWW-Authenticate" which is a hint for the client what to do next. The "WWW-Authenticate" Header tells the client which kind of authentication is needed (either Basic or Digest). This is usually not very useful in headless http clients, but thats how the standard is defined. The error occurs because the lib tries to parse the "WWW-Authenticate" header but can't.
( android - volley error No authentication challenges found )
But it's quite weird for me because I don't want to use WWW-authenticate things, I just want to get the code 401, but I always get the exception.
How can I bypass this problem? Any suggestion is really appreciated.
I have do some research and come to conclusion that, this is a server issue, that did not follow the convention.
From wiki:
401 Unauthorized (RFC 7235)
Similar to 403 Forbidden, but specifically for use when authentication is required and has failed or has not yet been provided. The response must include a WWW-Authenticate header field containing a challenge applicable to the requested resource. See Basic access authentication and Digest access authentication.
I think the best way to solve this problem is to solve in the server by add the header (something like:
WWW-Authenticate: xBasic realm=""
For me, I cannot change the server, so I have to check the error message to detect that a 401 error:
if (error.getMessage().equalsIgnoreCase("java.io.IOException: No authentication challenges found")){
showLoginError();
}
Not very elegant solution but work for now.
java.io.IOException: Server returned HTTP response code: 500 for URL: https://***/fiwebservice/services/FIUsbWebService
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1459)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:234)
at com.abcde.testClient.TestClientTry.main(TestClientTry.java:109)
I have replaced the url as *** for security purpose, as it is confidential..
Why is there an error when I call a soap webservice in eclipse?
Please help me regarding this.
It seems that there is an error in com.abcde.testClient.TestClientTry. Could you provide the logs and the the source of the File?
Http 500 can mean many things. In Spring security I think that can mean that you didnt have the appropriate authentication to reach the resource. Without knowing much about your server side its hard to say what the problem is or how to solve it.
What kind of technology did you have at the server?
HTTP status code 500 usually means that the web server code crashes.
If HttpURLConnection#getResponseCode() and error and HttpURLConnectionof#getErrorStream() instead of (to the status code to determine in advance), it is necessary to read. It can in other words, information about the problem.
Host if blocked you, you have got the code 4NN State, as the more 401 and 403rd