Why does the following Code returns -1? Seems that the request failed.
public static void main(String[] args)
{
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet("http://www.google.de");
HttpResponse response;
try
{
response = httpClient.execute(httpGet);
HttpEntity entity = response.getEntity();
EntityUtils.consume(entity);
// Prints -1
System.out.println(entity.getContentLength());
}
catch (ClientProtocolException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
httpGet.releaseConnection();
}
}
And is it possible to get the response as String?
Try running
Header[] headers = response.getAllHeaders();
for (Header header : headers) {
System.out.println(header);
}
It will print
Date: Tue, 10 Sep 2013 19:10:04 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=dad7e2356ddb3b7a:FF=0:TM=1378840204:LM=1378840204:S=vQcLzVPbOOTxfvL4; expires=Thu, 10-Sep-2015 19:10:04 GMT; path=/; domain=.google.de
Set-Cookie: NID=67=S11HcqAV454IGRGMRo-AJpxAPxClJeRs4DRkAJQ5vI3YBh4anN3qS0EVeiYX_4XDTGN-mY86xTBoJ3Ncca7eNSdtGjcaG31pbCOuqsZEQMWwKn-7-6Dnizx395snehdA; expires=Wed, 12-Mar-2014 19:10:04 GMT; path=/; domain=.google.de; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic
Transfer-Encoding: chunked
This is not a problem, the page you requested simply doesn't provide a Content-Length header in its response. As such, the HttpEntity#getContentLength() returns -1.
EntityUtils has a number of methods, some of which return a String.
Running curl more recently produces
> curl --head http://www.google.de
HTTP/1.1 200 OK
Date: Fri, 03 Apr 2020 15:38:18 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2020-04-03-15; expires=Sun, 03-May-2020 15:38:18 GMT; path=/; domain=.google.de; Secure
Set-Cookie: NID=201=H8GdKY8_vE5Ehy6qSkmQru13HqdGEj2tvZUFqvTDAVBxFoL4POI0swPtfI45v1TBjrJuAAfbcNMUddniIf9HHituCAFwUqmUFMDwxDYK5qUlcWiB1A64OcGp6PTT6LKur2r_3z-ToSvLf8RZhKWdny6E8SaArMpkaOqUEWp4aoQ; expires=Sat, 03-Oct-2020 15:38:18 GMT; path=/; domain=.google.de; HttpOnly
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding
The headers contain a Transfer-Encoding value of chunked. With chunked, the response contains "chunks" preceded by their length. An HTTP client uses those to read the entire response.
The HTTP Specification states that the Content-Length header should not be present when Transfer-Encoding has a value of chunked and MUST be ignored if it is.
Please notice that response header name Transfer-Encoding. Its value is chunked which means data is deliveryed block by block. Transfer-Encoding: chunked and Content-Length does not turn out at the same time.
There are two reason.
Server does not want sent content length.
Or server do not know the content length when it flush a big size data whose size is large than server's buffer.
So when there is no content length header, you can find the size of each chunked block before body of content. For example:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=8A7461DDA53B4C4DD0E89D73219CB5F8; Path=/
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Date: Wed, 18 Mar 2015 07:10:05 GMT
11
helloworld!
3
123
0
Above headers and content tell us, there are two block data. The size of first block is 11. the size of second block is 3. So the content length is 14 at all.
regards,
Xici
If you really want to get the content length without caring about the content, you can do this.
EntityUtils.toByteArray(httpResponse.getEntity()).length
Related
To be more specific, I mean specifically to just consume the HTTP headers over the network and stop the communication before the client receives the response body.
Example
Client makes a request
GET / HTTP/1.1
Host: example.com
Connection: Keep-Alive
User-Agent: Apache-HttpClient/UNAVAILABLE (Java/1.8.0_262)
Accept-Encoding: gzip,deflate
Then the response over the network is just
HTTP/1.1 200 OK
Date: Wed, 23 Sep 2020 22:41:21 GMT
Server: Apache
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: public, max-age=10800
Content-Language: en
Vary: Cookie,Accept-Encoding
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Age: 1865
grace: none
Content-Length: 9213
Connection: keep-alive
Accept-Ranges: bytes
Http protocol has six method, one of the methods is 'HEAD'. You can try use HEAD method instead of GET method.
And another stupid way : declare a web interface, and return null string.Like this:
// a web interface
String result = "";
return result;
I am using the Dank Memer imgen api to manipulate images.
To perform the GET request, i used the Unirest-Java. (https://kong.github.io/unirest-java/)
The code looks like this:
HttpResponse<JsonNode> response = Unirest.get("https://dankmemer.services/api/changemymind")
.header("Authorization", "tokenhere")
.queryString("text", "I am a human")
.asJson();
If i request the Headers with response.getHeaders() it returns this:
Date: Mon, 02 Dec 2019 17:07:08 GMT
Content-Type: image/jpeg
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=dd737927432f802b76c89b2fa8ee72e171575306428; expires=Wed, 01-Jan-20 17:07:08 GMT; path=/; domain=.dankmemer.services; HttpOnly; Secure
Cache-Control: public, max-age=43200
Expires: Tue, 03 Dec 2019 05:07:08 GMT
X-RateLimit-Limit: 5
X-RateLimit-Remaining: 4
X-RateLimit-Reset: 1575306429645
X-Global-RateLimit-Limit: 300
X-Global-RateLimit-Remaining: 299
X-Global-RateLimit-Reset: 1575306488643
CF-Cache-Status: DYNAMIC
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 53eee53a48a5d709-FRA
However i need to get the response body which returns null when i request it.
response.getBody();
// returns null.
// Also returns null when i put: response.getBody.toString();
How do i display the body?
I tried getting the response as a file and store it.
This worked perfectly.
HttpResponse<File> response = Unirest.get("https://dankmemer.services/api/changemymind")
.header("Authorization", "tokenhere")
.queryString("text", "i am a human" )
.asFile("C:\\Users\\PC\\Some Folder\\changemymind.jpg");
When we reload the Nginx Ingress config, we get the NoHttpResponseException for some of our POST requests. This does not occur in neither OkHttp client or just plain ab -c 100 -n 1000 https://...
Using 4.5.7, the latest one, and disabled the Gzip compression for visibility. Put a break point in DefaultHttpResponseParser in:
#Override
protected HttpResponse parseHead(
final SessionInputBuffer sessionBuffer) throws IOException, HttpException {
//read out the HTTP status string
int count = 0;
ParserCursor cursor = null;
do {
// clear the buffer
this.lineBuf.clear();
final int i = sessionBuffer.readLine(this.lineBuf);
if (i == -1 && count == 0) {
// The server just dropped connection on us
throw new NoHttpResponseException("The target server failed to respond");
}
When an error occurs, we observe the buffer has the following contents:
0
1.1 200 OK
Server: nginx/1.15.5
Date: Tue, 19 Mar 2019 08:51:27 GMT
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Strict-Transport-Security: max-age=15724800; includeSubDomains
10
{"success":true}
But for the regular requests, it has the following contents, which makes more sense:
HTTP/1.1 200 OK
Server: nginx/1.15.5
Date: Tue, 19 Mar 2019 08:52:30 GMT
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Strict-Transport-Security: max-age=15724800; includeSubDomains
10
{"success":true}
Now, I am not sure what is wrong, because both okhttp and ab works correctly. Tried many versions, but it seems to remain.
I use java (and http client, or other lib). I Want doing a GET request to a web server (sample http://MyDomain:8080/login.jsp).
Now, I have <html> .... </html> and any headers.
How find any informations (Tomcat 7.0.75, Linux 2016.09 v2.5.0, Java 8.0.233.3...) of target Web server by java httpClient?
My java code is:
public class GetWebServerInfo {
public static void main(String[] args) throws ClientProtocolException,
IOException {
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet("http://www.mydomain...");
HttpResponse response = client.execute(request);
System.out.println(response);
Header[] headers = response.getAllHeaders();
for (Header header : headers) {
System.out.println("" + header);
}
}
}
My response in console is:
Date: Sun, 12 Feb 2017 00:11:41 GMT
Server: Apache-Coyote/1.1
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Application-Context: application:prod:8080
Last-Modified: Sat, 11 Feb 2017 23:36:56 GMT
Accept-Ranges: bytes
Content-Type: text/html;charset=UTF-8
Content-Length: 3426
Keep-Alive: timeout=60, max=100
Connection: Keep-Alive
I do not have Tomcat 8, ....
Different servers may or may not return headers telling you about themselves. eg.
Server:Microsoft-IIS/8.5
X-Powered-By:ARR/3.0, ASP.NET
or
server:Oracle-Application-Server-11g
x-powered-by:Servlet/2.5 JSP/2.1
You can't demand information and I don't think many servers will be telling you about their operating system. "Why is that any of your business?"
I recommend you print out all the returned headers and make requests to different servers and see what you get, and what can be of use to you.
How i can i display all the HTTP Headers when using the DefaultHTTPClient?
I'm writing an Java Desktop app which has different HTTP connections to the same website running through different proxies all in different threads. These threads are all posting to one website using an HttpURLConnection but are intended to appear as different users, all with different session cookies. I have written a basic cookie manager to allow each thread to have different sessions, by iterating through the http response headers and storing those with the key "Set-Cookie". The relevant code is below:
private void addCookie(String cookie) { //adds received cookie to cookie string
if (cookies.contains(cookie)) return; //we have this exact cookie
processor.printSys("New Cookie: " + cookie);
cookies += (cookie + "; "); //add the new cookie followed by a semicolon
}
private void storeCookies(URLConnection con) { //iterates through headers, adding new cookies
String headerName = null;
for (int i = 0; i < con.getHeaderFields().size(); i ++) { //print headers for analysis
System.out.println(con.getHeaderFieldKey(i) + ": " + con.getHeaderField(i));
}
for (int i=1; (headerName = con.getHeaderFieldKey(i)) != null; i++) if (headerName.equals("Set-Cookie")) addCookie(con.getHeaderField(i));
}
private void setCookies(URLConnection conn) { //add stored cookies to next HttpURLConnection
if (cookies.isEmpty()) {conn.setRequestProperty("Cookie", ""); return;}
conn.setRequestProperty("Cookie", cookies);
processor.printSys("Cookies Set: " + cookies);
}
This cookie manager was working flawlessly before and would properly store cookies, printing to the console the below text for each thread when storeCookies(con) was called.
null: HTTP/1.1 200 OK
Server: nginx
Date: Sun, 10 Jan 2016 20:57:16 GMT
Content-Type: text/javascript; charset=utf-8
Content-Length: 977
Connection: keep-alive
Status: 200 OK
X-XHR-Current-Location: /shop/169618/add
X-UA-Compatible: IE=Edge,chrome=1
ETag: "3a5199771b11612b34cba523cd4af312"
Set-Cookie: cart=FOO; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT
Set-Cookie: pure_cart=BAR; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT
Set-Cookie: _supreme_session=BAZ; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT; HttpOnly
X-Request-Id: 20ef370b74e478b6484bb8d507a33a3a
X-Runtime: 0.021299
X-Rack-Cache: invalidate, pass
Accept-Ranges: bytes
X-Varnish: 2380939932
Age: 0
Via: 1.1 varnish
Cache-Control: private, max-age=0, must-revalidate
Order 1: New Cookie: cart=FOO; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT (3:57:16 PM)
Order 1: New Cookie: pure_cart=BAR; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT (3:57:16 PM)
Order 1: New Cookie: _supreme_session=BAZ; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT; HttpOnly (3:57:16 PM)
However, I implemented a JFX WebView in my UI in a different thread an instantly noticed a change. All threads were storing the same session cookies and even when I explicitly set the cookies to be blank this was overridden. The WebView was implemented as shown below
Platform.runLater(() -> {
webView = new WebView();
webView.setZoom(.5);
jfxPanel.setScene(new Scene(webView));
webView.getEngine().load(url);
});
This WebView would cause the same storeCookies(con) call to print:
null: HTTP/1.1 200 OK
Server: nginx
Date: Sun, 10 Jan 2016 20:57:16 GMT
Content-Type: text/javascript; charset=utf-8
Content-Length: 977
Connection: keep-alive
Status: 200 OK
X-XHR-Current-Location: /shop/169618/add
X-UA-Compatible: IE=Edge,chrome=1
ETag: "3a5199771b11612b34cba523cd4af312"
Set-Cookie: cart=FOO; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT
Set-Cookie: pure_cart=BAR; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT
Set-Cookie:
X-Request-Id: 20ef370b74e478b6484bb8d507a33a3a
X-Runtime: 0.021299
X-Rack-Cache: invalidate, pass
Accept-Ranges: bytes
X-Varnish: 2380939932
Age: 0
Via: 1.1 varnish
Cache-Control: private, max-age=0, must-revalidate
Order 1: New Cookie: cart=FOO; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT (3:57:16 PM)
Order 1: New Cookie: pure_cart=BAR; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT (3:57:16 PM)
Order 1: New Cookie:
Upon packet analysis with WireShark I determined these fields weren't blank, the _supreme_session cookie was still there it just could no longer be accessed in Java. Even worse, the session cookie of the JFX WebView was being passed in as the session cookie for every thread posting through HttpUrlConnection. Clearly something is going on in the background and overriding my cookie manager, setting and storing all cookies for any HttpURLConnection. When I comment out the WebView the issue is solved, so I just need to know what is going on in the background and how to disable it. Thanks, Ben