Develop Reference

How to make an HTTP request in Java and ignore the body

To be more specific, I mean specifically to just consume the HTTP headers over the network and stop the communication before the client receives the response body.
Example
Client makes a request
GET / HTTP/1.1
Host: example.com
Connection: Keep-Alive
User-Agent: Apache-HttpClient/UNAVAILABLE (Java/1.8.0_262)
Accept-Encoding: gzip,deflate
Then the response over the network is just
HTTP/1.1 200 OK
Date: Wed, 23 Sep 2020 22:41:21 GMT
Server: Apache
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: public, max-age=10800
Content-Language: en
Vary: Cookie,Accept-Encoding
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Age: 1865
grace: none
Content-Length: 9213
Connection: keep-alive
Accept-Ranges: bytes

Http protocol has six method, one of the methods is 'HEAD'. You can try use HEAD method instead of GET method.
And another stupid way : declare a web interface, and return null string.Like this:
// a web interface
String result = "";
return result;

How can i get the json body of this GET request?

I am using the Dank Memer imgen api to manipulate images.
To perform the GET request, i used the Unirest-Java. (https://kong.github.io/unirest-java/)
The code looks like this:
HttpResponse<JsonNode> response = Unirest.get("https://dankmemer.services/api/changemymind")
.header("Authorization", "tokenhere")
.queryString("text", "I am a human")
.asJson();
If i request the Headers with response.getHeaders() it returns this:
Date: Mon, 02 Dec 2019 17:07:08 GMT
Content-Type: image/jpeg
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=dd737927432f802b76c89b2fa8ee72e171575306428; expires=Wed, 01-Jan-20 17:07:08 GMT; path=/; domain=.dankmemer.services; HttpOnly; Secure
Cache-Control: public, max-age=43200
Expires: Tue, 03 Dec 2019 05:07:08 GMT
X-RateLimit-Limit: 5
X-RateLimit-Remaining: 4
X-RateLimit-Reset: 1575306429645
X-Global-RateLimit-Limit: 300
X-Global-RateLimit-Remaining: 299
X-Global-RateLimit-Reset: 1575306488643
CF-Cache-Status: DYNAMIC
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 53eee53a48a5d709-FRA
However i need to get the response body which returns null when i request it.
response.getBody();
// returns null.
// Also returns null when i put: response.getBody.toString();
How do i display the body?

I tried getting the response as a file and store it.
This worked perfectly.
HttpResponse<File> response = Unirest.get("https://dankmemer.services/api/changemymind")
.header("Authorization", "tokenhere")
.queryString("text", "i am a human" )
.asFile("C:\\Users\\PC\\Some Folder\\changemymind.jpg");

Running external program within java produces different results from terminal command

Hey I'm trying to run a command within java:
String curlCommand=String.format("curl --max-time %d --socks5 %s -A \"%s\" -i %s",maxWait,proxy ,getAgent(), myurl);
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(curlCommand);
For some reason executing this line would result in cookies to be denied.
however, when I run the curl program from terminal I have no problem and everything works as it should be
here is the result if I run the program Java
Date: Tue, 03 Mar 2015 19:20:38 GMT
X-Li-Pop: prod-lva1
X-LI-UUID: 8C548z0TyBMwY002tCoAAA==
Set-Cookie: denial-reason-code=3,2,8; Max-Age=5
Set-Cookie: denial-client-ip=xxxxxxx; Max-Age=5
Content-Length: 1852
Content-Type: text/html
here is the result if I run exact command using curl in terminal:
HTTP/1.1 200 OK
Server: Play
X-SSR-Engine-Init: <xxxxxxx>
X-FS-UUID: 0e69d33f4913c813b05f1aefb42a0000
X-Page-Speed: 1
Date: Tue, 03 Mar 2015 19:21:27 GMT
Content-Length: 31113
Content-Type: text/html; charset=utf-8
X-Frame-Options: sameorigin
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Li-Fabric: prod-lva1
Strict-Transport-Security: max-age=0
Set-Cookie: JSESSIONID="ajax:4773794645302404176"; Path=/; Domain=<xxxxxx>; HTTPOnly
Set-Cookie: bcookie="v=2&da2801ec-e7e2-4b04-8244-401d6ac0e7c3"; domain=xxxxx; Path=/; Expires=Fri, 03-Mar-2017 06:58:59 GMT
Set-Cookie: bscookie="v=1&2015030319212713ed4940-eb5d-4860-8c30-2eec4819157aAQHMGWOb0Fiv_feTewpb6Rw3TGGAGQ_Q"; domain=<xxxxxx>; Path=/; Secure; Expires=Fri, 03-Mar-2017 06:58:59 GMT; HttpOnly
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache, no-store
Connection: keep-alive
X-Li-Pop: prod-lva1
X-LI-UUID: DmnTP0kTyBOwXxrvtCoAAA==
Set-Cookie: lidc="b=VB97:g=149:u=1:i=1425410487:t=1425496887:s=AQHAOhdkFxryw00Wo64LkfjsYb8Q1hqZ"; Expires=Wed, 04 Mar 2015 19:21:27 GMT; domain=<xxxxx>; Path=/
So My question is why should it be different at all, am I missing something out here?

Alright I finally solved it.
the reason is Runtime runtime = Runtime.getRuntime();
it should not be used. as it just messes up with the parameters (eventhough the print out of the command looks fine, it just didn't send the right command to the OS) instead I should have used ProcessBuilder class. it worked like a charm.

Why is my Java app only fetching an old version of an online file?

I have a file online with information about some Minecraft blocks. When I first made this test file, I gave it three rows and a header expiration date of next Sunday (whenever that may be). My Java app fetched this no problem!
However, now I have inserted three more rows into this small database and changed the expiration date to last week, but my Java app still displays the original 3! When I visit the page in a browser, it gives me the full, current table. How come the Java app is still only fetching the old version?
The key code:
InputStream in;
URLConnection urlc = url.openConnection(); // url is a valid java.net.URL object
urlc.setAllowUserInteraction(false);
urlc.setDoInput(true);
urlc.setDoOutput(false);
urlc.setRequestProperty("User-Agent", "BHMI/3.0.0 (+http://prog.BHStudios.org/BHMI) Java/" + System.getProperty("java.version") + "(" + System.getProperty("java.vm.name") + ")"); // GoDaddy blocks Java clients, so we must have a custom user agent string
urlc.setDefaultUseCaches(false);
urlc.setUseCaches(false);
urlc.connect();
System.out.println("Connection successful! Database expires " + new Date(urlc.getExpiration()));
in = urlc.getInputStream();
int data;
StringBuilder sb = new StringBuilder();
while ((data = in.read()) != -1)
sb.append((char) data);
System.out.println("RAW DATA:\r\n"+sb);
Sample output:
Connection successful! Database expires Tue Nov 26 00:09:05 EST 2013
RAW DATA:
minecraft:air,Air,0,0,,
minecraft:stone,Stone,1,0,2,
minecraft:grass,Grass,2,0,,
I cleared the Java network cache through Windows control panel, and all caches and temporary files on my local machine with CCleaner, but this still happens. Heck, it happens across machines, so it can't be that. I've cleared all edge caches from my server, so it also can't be that.
I've even tried downloading the file after telling my browser to use my Java app's User-Agent string, and it fetched all 5 lines.
Request Headers
From my Java app:
GET /http/bhstudios/v2/prog/bhmi/database/get HTTP/1.1
User-Agent: BHMI/3.0.0 (+http://prog.BHStudios.org/BHMI) Java/1.7.0_45(Java HotSpot(TM) 64-Bit Server VM)
Cache-Control: no-cache, must-revalidate, max-age=0, no-store
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Connection: close
Host: BHStudios.org
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
From Chrome, spoofing the same User-Agent string:
GET /prog/bhmi/database/get/ HTTP/1.1
Host: prog.bhstudios.org
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: BHMI/3.0.0 (+http://prog.BHStudios.org/BHMI) Java/1.7.0_45(Java HotSpot(TM) 64-Bit Server VM)
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: __cfduid=dc9d0394ed55ebb1214fcbb5fc825626b1385426208553; visitorId=5293ed2b758cb1b5620000b0
Response Headers
From my Java app:
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Tue, 26 Nov 2013 02:17:39 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d4432e3d81cf9e5b9393f2cca483e4b2d1385432256651; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.bhstudios.org; HttpOnly
X-Powered-By: ASP.NET
X-UA-Compatible: chrome=IE8
CF-RAY: d33155416660862
Note that suspicious cookie expiration expires=Mon, 23-Dec-2019 23:50:00 GMT. Could this be the cause?
I also note that, when fetching from Chrome and using the same User-Agent string as my app, the header is:
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Wed, 27 Nov 2013 17:30:01 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache, must-revalidate, max-age=0, no-store
Pragma: no-cache
Expires: Mon, 18 Nov 2013 10:30:01 America/Phoenix
Content-Description: File Transfer
Content-Disposition: attachment; filename=BHMI_Items_Vanilla_172.csv
Content-Transfer-Encoding: base64
X-Powered-By: ASP.NET
X-UA-Compatible: chrome=IE8
CF-RAY: d408b3c56320098
Content-Encoding: gzip
which is the intended header, with an expiration date of last week.

You have to put in your request header info that you are looking for data that are not cached:
urlc.setRequestProperty("Cache-Control","no-cache, must-revalidate"); //HTTP 1.1
urlc.setRequestProperty("Pragma","no-cache"); //HTTP 1.0

... I was requesting the wrong file.
Sorry for wasting your time >.<
As the header shows, I was addressing an old URL scheme, /http/bhstudios/v2/prog/bhmi/database/get, when I wanted /prog/bhmi/database/get

Apache HttpClient response content length returns -1

Why does the following Code returns -1? Seems that the request failed.
public static void main(String[] args)
{
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet("http://www.google.de");
HttpResponse response;
try
{
response = httpClient.execute(httpGet);
HttpEntity entity = response.getEntity();
EntityUtils.consume(entity);
// Prints -1
System.out.println(entity.getContentLength());
}
catch (ClientProtocolException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
httpGet.releaseConnection();
}
}
And is it possible to get the response as String?

Try running
Header[] headers = response.getAllHeaders();
for (Header header : headers) {
System.out.println(header);
}
It will print
Date: Tue, 10 Sep 2013 19:10:04 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=dad7e2356ddb3b7a:FF=0:TM=1378840204:LM=1378840204:S=vQcLzVPbOOTxfvL4; expires=Thu, 10-Sep-2015 19:10:04 GMT; path=/; domain=.google.de
Set-Cookie: NID=67=S11HcqAV454IGRGMRo-AJpxAPxClJeRs4DRkAJQ5vI3YBh4anN3qS0EVeiYX_4XDTGN-mY86xTBoJ3Ncca7eNSdtGjcaG31pbCOuqsZEQMWwKn-7-6Dnizx395snehdA; expires=Wed, 12-Mar-2014 19:10:04 GMT; path=/; domain=.google.de; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic
Transfer-Encoding: chunked
This is not a problem, the page you requested simply doesn't provide a Content-Length header in its response. As such, the HttpEntity#getContentLength() returns -1.
EntityUtils has a number of methods, some of which return a String.
Running curl more recently produces
> curl --head http://www.google.de
HTTP/1.1 200 OK
Date: Fri, 03 Apr 2020 15:38:18 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2020-04-03-15; expires=Sun, 03-May-2020 15:38:18 GMT; path=/; domain=.google.de; Secure
Set-Cookie: NID=201=H8GdKY8_vE5Ehy6qSkmQru13HqdGEj2tvZUFqvTDAVBxFoL4POI0swPtfI45v1TBjrJuAAfbcNMUddniIf9HHituCAFwUqmUFMDwxDYK5qUlcWiB1A64OcGp6PTT6LKur2r_3z-ToSvLf8RZhKWdny6E8SaArMpkaOqUEWp4aoQ; expires=Sat, 03-Oct-2020 15:38:18 GMT; path=/; domain=.google.de; HttpOnly
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding
The headers contain a Transfer-Encoding value of chunked. With chunked, the response contains "chunks" preceded by their length. An HTTP client uses those to read the entire response.
The HTTP Specification states that the Content-Length header should not be present when Transfer-Encoding has a value of chunked and MUST be ignored if it is.

Please notice that response header name Transfer-Encoding. Its value is chunked which means data is deliveryed block by block. Transfer-Encoding: chunked and Content-Length does not turn out at the same time.
There are two reason.
Server does not want sent content length.
Or server do not know the content length when it flush a big size data whose size is large than server's buffer.
So when there is no content length header, you can find the size of each chunked block before body of content. For example:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=8A7461DDA53B4C4DD0E89D73219CB5F8; Path=/
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Date: Wed, 18 Mar 2015 07:10:05 GMT
11
helloworld!
3
123
0
Above headers and content tell us, there are two block data. The size of first block is 11. the size of second block is 3. So the content length is 14 at all.
regards,
Xici

If you really want to get the content length without caring about the content, you can do this.
EntityUtils.toByteArray(httpResponse.getEntity()).length