Develop Reference

How to make an HTTP request in Java and ignore the body

To be more specific, I mean specifically to just consume the HTTP headers over the network and stop the communication before the client receives the response body.
Example
Client makes a request
GET / HTTP/1.1
Host: example.com
Connection: Keep-Alive
User-Agent: Apache-HttpClient/UNAVAILABLE (Java/1.8.0_262)
Accept-Encoding: gzip,deflate
Then the response over the network is just
HTTP/1.1 200 OK
Date: Wed, 23 Sep 2020 22:41:21 GMT
Server: Apache
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: public, max-age=10800
Content-Language: en
Vary: Cookie,Accept-Encoding
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Age: 1865
grace: none
Content-Length: 9213
Connection: keep-alive
Accept-Ranges: bytes

Http protocol has six method, one of the methods is 'HEAD'. You can try use HEAD method instead of GET method.
And another stupid way : declare a web interface, and return null string.Like this:
// a web interface
String result = "";
return result;

JavaFx WebView Storing and Manipulating Session Cookies of HttpURLConnection

I'm writing an Java Desktop app which has different HTTP connections to the same website running through different proxies all in different threads. These threads are all posting to one website using an HttpURLConnection but are intended to appear as different users, all with different session cookies. I have written a basic cookie manager to allow each thread to have different sessions, by iterating through the http response headers and storing those with the key "Set-Cookie". The relevant code is below:
private void addCookie(String cookie) { //adds received cookie to cookie string
if (cookies.contains(cookie)) return; //we have this exact cookie
processor.printSys("New Cookie: " + cookie);
cookies += (cookie + "; "); //add the new cookie followed by a semicolon
}
private void storeCookies(URLConnection con) { //iterates through headers, adding new cookies
String headerName = null;
for (int i = 0; i < con.getHeaderFields().size(); i ++) { //print headers for analysis
System.out.println(con.getHeaderFieldKey(i) + ": " + con.getHeaderField(i));
}
for (int i=1; (headerName = con.getHeaderFieldKey(i)) != null; i++) if (headerName.equals("Set-Cookie")) addCookie(con.getHeaderField(i));
}
private void setCookies(URLConnection conn) { //add stored cookies to next HttpURLConnection
if (cookies.isEmpty()) {conn.setRequestProperty("Cookie", ""); return;}
conn.setRequestProperty("Cookie", cookies);
processor.printSys("Cookies Set: " + cookies);
}
This cookie manager was working flawlessly before and would properly store cookies, printing to the console the below text for each thread when storeCookies(con) was called.
null: HTTP/1.1 200 OK
Server: nginx
Date: Sun, 10 Jan 2016 20:57:16 GMT
Content-Type: text/javascript; charset=utf-8
Content-Length: 977
Connection: keep-alive
Status: 200 OK
X-XHR-Current-Location: /shop/169618/add
X-UA-Compatible: IE=Edge,chrome=1
ETag: "3a5199771b11612b34cba523cd4af312"
Set-Cookie: cart=FOO; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT
Set-Cookie: pure_cart=BAR; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT
Set-Cookie: _supreme_session=BAZ; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT; HttpOnly
X-Request-Id: 20ef370b74e478b6484bb8d507a33a3a
X-Runtime: 0.021299
X-Rack-Cache: invalidate, pass
Accept-Ranges: bytes
X-Varnish: 2380939932
Age: 0
Via: 1.1 varnish
Cache-Control: private, max-age=0, must-revalidate
Order 1: New Cookie: cart=FOO; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT (3:57:16 PM)
Order 1: New Cookie: pure_cart=BAR; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT (3:57:16 PM)
Order 1: New Cookie: _supreme_session=BAZ; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT; HttpOnly (3:57:16 PM)
However, I implemented a JFX WebView in my UI in a different thread an instantly noticed a change. All threads were storing the same session cookies and even when I explicitly set the cookies to be blank this was overridden. The WebView was implemented as shown below
Platform.runLater(() -> {
webView = new WebView();
webView.setZoom(.5);
jfxPanel.setScene(new Scene(webView));
webView.getEngine().load(url);
});
This WebView would cause the same storeCookies(con) call to print:
null: HTTP/1.1 200 OK
Server: nginx
Date: Sun, 10 Jan 2016 20:57:16 GMT
Content-Type: text/javascript; charset=utf-8
Content-Length: 977
Connection: keep-alive
Status: 200 OK
X-XHR-Current-Location: /shop/169618/add
X-UA-Compatible: IE=Edge,chrome=1
ETag: "3a5199771b11612b34cba523cd4af312"
Set-Cookie: cart=FOO; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT
Set-Cookie: pure_cart=BAR; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT
Set-Cookie:
X-Request-Id: 20ef370b74e478b6484bb8d507a33a3a
X-Runtime: 0.021299
X-Rack-Cache: invalidate, pass
Accept-Ranges: bytes
X-Varnish: 2380939932
Age: 0
Via: 1.1 varnish
Cache-Control: private, max-age=0, must-revalidate
Order 1: New Cookie: cart=FOO; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT (3:57:16 PM)
Order 1: New Cookie: pure_cart=BAR; path=/; expires=Mon, 11-Jan-2016 20:57:16 GMT (3:57:16 PM)
Order 1: New Cookie:
Upon packet analysis with WireShark I determined these fields weren't blank, the _supreme_session cookie was still there it just could no longer be accessed in Java. Even worse, the session cookie of the JFX WebView was being passed in as the session cookie for every thread posting through HttpUrlConnection. Clearly something is going on in the background and overriding my cookie manager, setting and storing all cookies for any HttpURLConnection. When I comment out the WebView the issue is solved, so I just need to know what is going on in the background and how to disable it. Thanks, Ben

Why is my Java app only fetching an old version of an online file?

I have a file online with information about some Minecraft blocks. When I first made this test file, I gave it three rows and a header expiration date of next Sunday (whenever that may be). My Java app fetched this no problem!
However, now I have inserted three more rows into this small database and changed the expiration date to last week, but my Java app still displays the original 3! When I visit the page in a browser, it gives me the full, current table. How come the Java app is still only fetching the old version?
The key code:
InputStream in;
URLConnection urlc = url.openConnection(); // url is a valid java.net.URL object
urlc.setAllowUserInteraction(false);
urlc.setDoInput(true);
urlc.setDoOutput(false);
urlc.setRequestProperty("User-Agent", "BHMI/3.0.0 (+http://prog.BHStudios.org/BHMI) Java/" + System.getProperty("java.version") + "(" + System.getProperty("java.vm.name") + ")"); // GoDaddy blocks Java clients, so we must have a custom user agent string
urlc.setDefaultUseCaches(false);
urlc.setUseCaches(false);
urlc.connect();
System.out.println("Connection successful! Database expires " + new Date(urlc.getExpiration()));
in = urlc.getInputStream();
int data;
StringBuilder sb = new StringBuilder();
while ((data = in.read()) != -1)
sb.append((char) data);
System.out.println("RAW DATA:\r\n"+sb);
Sample output:
Connection successful! Database expires Tue Nov 26 00:09:05 EST 2013
RAW DATA:
minecraft:air,Air,0,0,,
minecraft:stone,Stone,1,0,2,
minecraft:grass,Grass,2,0,,
I cleared the Java network cache through Windows control panel, and all caches and temporary files on my local machine with CCleaner, but this still happens. Heck, it happens across machines, so it can't be that. I've cleared all edge caches from my server, so it also can't be that.
I've even tried downloading the file after telling my browser to use my Java app's User-Agent string, and it fetched all 5 lines.
Request Headers
From my Java app:
GET /http/bhstudios/v2/prog/bhmi/database/get HTTP/1.1
User-Agent: BHMI/3.0.0 (+http://prog.BHStudios.org/BHMI) Java/1.7.0_45(Java HotSpot(TM) 64-Bit Server VM)
Cache-Control: no-cache, must-revalidate, max-age=0, no-store
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Connection: close
Host: BHStudios.org
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
From Chrome, spoofing the same User-Agent string:
GET /prog/bhmi/database/get/ HTTP/1.1
Host: prog.bhstudios.org
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: BHMI/3.0.0 (+http://prog.BHStudios.org/BHMI) Java/1.7.0_45(Java HotSpot(TM) 64-Bit Server VM)
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: __cfduid=dc9d0394ed55ebb1214fcbb5fc825626b1385426208553; visitorId=5293ed2b758cb1b5620000b0
Response Headers
From my Java app:
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Tue, 26 Nov 2013 02:17:39 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d4432e3d81cf9e5b9393f2cca483e4b2d1385432256651; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.bhstudios.org; HttpOnly
X-Powered-By: ASP.NET
X-UA-Compatible: chrome=IE8
CF-RAY: d33155416660862
Note that suspicious cookie expiration expires=Mon, 23-Dec-2019 23:50:00 GMT. Could this be the cause?
I also note that, when fetching from Chrome and using the same User-Agent string as my app, the header is:
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Wed, 27 Nov 2013 17:30:01 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache, must-revalidate, max-age=0, no-store
Pragma: no-cache
Expires: Mon, 18 Nov 2013 10:30:01 America/Phoenix
Content-Description: File Transfer
Content-Disposition: attachment; filename=BHMI_Items_Vanilla_172.csv
Content-Transfer-Encoding: base64
X-Powered-By: ASP.NET
X-UA-Compatible: chrome=IE8
CF-RAY: d408b3c56320098
Content-Encoding: gzip
which is the intended header, with an expiration date of last week.

You have to put in your request header info that you are looking for data that are not cached:
urlc.setRequestProperty("Cache-Control","no-cache, must-revalidate"); //HTTP 1.1
urlc.setRequestProperty("Pragma","no-cache"); //HTTP 1.0

... I was requesting the wrong file.
Sorry for wasting your time >.<
As the header shows, I was addressing an old URL scheme, /http/bhstudios/v2/prog/bhmi/database/get, when I wanted /prog/bhmi/database/get

Apache HttpClient response content length returns -1

Why does the following Code returns -1? Seems that the request failed.
public static void main(String[] args)
{
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet("http://www.google.de");
HttpResponse response;
try
{
response = httpClient.execute(httpGet);
HttpEntity entity = response.getEntity();
EntityUtils.consume(entity);
// Prints -1
System.out.println(entity.getContentLength());
}
catch (ClientProtocolException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
httpGet.releaseConnection();
}
}
And is it possible to get the response as String?

Try running
Header[] headers = response.getAllHeaders();
for (Header header : headers) {
System.out.println(header);
}
It will print
Date: Tue, 10 Sep 2013 19:10:04 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=dad7e2356ddb3b7a:FF=0:TM=1378840204:LM=1378840204:S=vQcLzVPbOOTxfvL4; expires=Thu, 10-Sep-2015 19:10:04 GMT; path=/; domain=.google.de
Set-Cookie: NID=67=S11HcqAV454IGRGMRo-AJpxAPxClJeRs4DRkAJQ5vI3YBh4anN3qS0EVeiYX_4XDTGN-mY86xTBoJ3Ncca7eNSdtGjcaG31pbCOuqsZEQMWwKn-7-6Dnizx395snehdA; expires=Wed, 12-Mar-2014 19:10:04 GMT; path=/; domain=.google.de; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic
Transfer-Encoding: chunked
This is not a problem, the page you requested simply doesn't provide a Content-Length header in its response. As such, the HttpEntity#getContentLength() returns -1.
EntityUtils has a number of methods, some of which return a String.
Running curl more recently produces
> curl --head http://www.google.de
HTTP/1.1 200 OK
Date: Fri, 03 Apr 2020 15:38:18 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2020-04-03-15; expires=Sun, 03-May-2020 15:38:18 GMT; path=/; domain=.google.de; Secure
Set-Cookie: NID=201=H8GdKY8_vE5Ehy6qSkmQru13HqdGEj2tvZUFqvTDAVBxFoL4POI0swPtfI45v1TBjrJuAAfbcNMUddniIf9HHituCAFwUqmUFMDwxDYK5qUlcWiB1A64OcGp6PTT6LKur2r_3z-ToSvLf8RZhKWdny6E8SaArMpkaOqUEWp4aoQ; expires=Sat, 03-Oct-2020 15:38:18 GMT; path=/; domain=.google.de; HttpOnly
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding
The headers contain a Transfer-Encoding value of chunked. With chunked, the response contains "chunks" preceded by their length. An HTTP client uses those to read the entire response.
The HTTP Specification states that the Content-Length header should not be present when Transfer-Encoding has a value of chunked and MUST be ignored if it is.

Please notice that response header name Transfer-Encoding. Its value is chunked which means data is deliveryed block by block. Transfer-Encoding: chunked and Content-Length does not turn out at the same time.
There are two reason.
Server does not want sent content length.
Or server do not know the content length when it flush a big size data whose size is large than server's buffer.
So when there is no content length header, you can find the size of each chunked block before body of content. For example:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=8A7461DDA53B4C4DD0E89D73219CB5F8; Path=/
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Date: Wed, 18 Mar 2015 07:10:05 GMT
11
helloworld!
3
123
0
Above headers and content tell us, there are two block data. The size of first block is 11. the size of second block is 3. So the content length is 14 at all.
regards,
Xici

If you really want to get the content length without caring about the content, you can do this.
EntityUtils.toByteArray(httpResponse.getEntity()).length

Unable to cache images served by Spring MVC

I am trying to serve some assets using a Spring MVC controller. My assets are database managed and thus have to be served this way. The service looks up the metadata of the asset from the database, reads the file from file system and builds the response.
Here is how my controller looks like.
#Controller
#RequestMapping("/assets")
public class AssetController {
#Autowired
private AssetService assetService;
#RequestMapping("/{assetName:.+}")
public ResponseEntity<byte[]> getAsset(#PathVariable("assetName") String assetName) throws FileNotFoundException, IOException {
Asset asset = assetService.findByName(assetName);
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.valueOf(asset.getContentType()));
headers.setCacheControl("max-age=1209600");
headers.setLastModified(asset.getModifiedOn().getTime()); // always in the past
return new ResponseEntity<byte[]>(assetService.toBytes(asset), headers, OK);
}
}
Seems simple and straightforward enough? One would hope to see the browser caching the images. But despite trying all combinations of Cache-Control, Expires, Last-Modified-On and ETag, I have had no success.
Below are the HTTP headers (irrelevant headers removed) spit out during two successive requests.
GET /adarshr-web/assets/Acer.png HTTP/1.1
Host: localhost:8080
Pragma: no-cache
Cache-Control: no-cache
HTTP/1.1 200 OK
Cache-Control: max-age=1209600
Last-Modified: Sun, 21 Jul 2013 11:56:32 GMT
Content-Type: image/png
Date: Tue, 23 Jul 2013 21:22:58 GMT
----------------------------------------------------------
GET /adarshr-web/assets/Acer.png HTTP/1.1
Host: localhost:8080
If-Modified-Since: Sun, 21 Jul 2013 11:56:32 GMT
Cache-Control: max-age=0
HTTP/1.1 200 OK <-- Why not 304 Not Modified?
Cache-Control: max-age=1209600
Last-Modified: Sun, 21 Jul 2013 11:56:32 GMT
Content-Type: image/png
Date: Tue, 23 Jul 2013 21:23:03 GMT
However, when I try the same sequence (Ctrl + F5 for first request and F5 for subsequent ones) on URLs such as
http://www.google.co.uk/images/srpr/logo4w.png (Google's logo)
http://fbstatic-a.akamaihd.net/rsrc.php/v2/yI/r/0PsXdTWc41M.png (Facebook's mobile image)
I see the headers such as these (shown for the Facebook URL) which indicate that the response is being cached by the browser.
GET /rsrc.php/v2/yI/r/0PsXdTWc41M.png HTTP/1.1
Host: fbstatic-a.akamaihd.net
Pragma: no-cache
Cache-Control: no-cache
HTTP/1.1 200 OK
Content-Type: image/png
Last-Modified: Sat, 15 Jun 2013 00:48:42 GMT
Cache-Control: public, max-age=31535893
Expires: Wed, 23 Jul 2014 21:27:47 GMT
Date: Tue, 23 Jul 2013 21:29:34 GMT
----------------------------------------------------------
GET /rsrc.php/v2/yI/r/0PsXdTWc41M.png HTTP/1.1
Host: fbstatic-a.akamaihd.net
If-Modified-Since: Sat, 15 Jun 2013 00:48:42 GMT
Cache-Control: max-age=0
HTTP/1.1 304 Not Modified <-- Note this
Content-Type: image/png
Last-Modified: Sat, 15 Jun 2013 00:48:42 GMT
Cache-Control: public, max-age=31535892
Expires: Wed, 23 Jul 2014 21:27:47 GMT
Date: Tue, 23 Jul 2013 21:29:35 GMT
Notes:
I don't have an <mvc:resources /> section in my Spring config since I am doing exactly the same in my controller. Even adding it doesn't make any difference.
I don't have a org.springframework.web.servlet.mvc.WebContentInterceptor defined in the Spring config again for the reasons above. I have tried adding one with no gain.
I have tried all methods explained in https://developers.google.com/speed/docs/best-practices/caching.
I can replicate this across all browsers.

You'll have to implement the check of the last modified, fortunately Spring makes that pretty easy.
From the Spring Framework Reference
#RequestMapping
public String myHandleMethod(WebRequest webRequest, Model model) {
long lastModified = // 1. application-specific calculation
if (request.checkNotModified(lastModified)) {
// 2. shortcut exit - no further processing necessary
return null;
}
// 3. or otherwise further request processing, actually preparing content
model.addAttribute(...);
return "myViewName";
}