Develop Reference

How to parse the web content using HttpURLConnection and TagNode class?

I would like to take some data from www.groupon.pl website. To do this I used code below:
String currentUrl = "https://www.groupon.pl/browse/radom";
URL urlObj = new URL(currentUrl);
HttpURLConnection urlConnection = (HttpURLConnection) urlObj.openConnection();
urlConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0");
urlConnection.connect();
final HtmlCleaner cleaner = new HtmlCleaner();
final TagNode tagNodeRoot = cleaner.clean(urlConnection.getInputStream());
The problem is that after the last line:
final TagNode tagNodeRoot = cleaner.clean(urlConnection.getInputStream());
The program doesn't want go further. I think that maybe the problem is with:
urlConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0");
Or maybe I use some and old API which is not supported any more? I use debugger and it doesn't stepping out. When I run this code normally without any debugger it also stops. No error at the console. I don't know maybe it is waiting for the response. Could you tell me what is wrong? Thank you.

JSoup.connect with some requests has 403 error

I try to get HTML source by Jsoup.connect of this page: https://bitskins.com/?market_hash_name=SSG+08+%7C+DARK+WATER+%28Field-Tested%29&is_stattrak=0&has_stickers=0&sort_by=bumped_at&order=desc
but, I have the error: Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403, URL=https://bitskins.com/?market_hash_name=SSG+08+%7C+DARK+WATER+%28Field-Tested%29&is_stattrak=0&has_stickers=0&sort_by=bumped_at&order=desc
My code is:
Document doc = Jsoup.connect("https://bitskins.com/?market_hash_name=SSG+08+%7C+DARK+WATER+%28Field-Tested%29&is_stattrak=0&has_stickers=0&sort_by=bumped_at&order=desc")
.data(":authority", "bitskins.com")
.data(":method", "GET")
.data(":path", "/?market_hash_name=SSG+08+%7C+DARK+WATER+%28Field-Tested%29&is_stattrak=0&has_stickers=0&sort_by=bumped_at&order=desc")
.data(":scheme", "https")
.data("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
.data("accept-encoding", "gzip, deflate, sdch, br")
.data("accept-language:", "ru,en-US;q=0.8,en;q=0.6")
.data("cache-control", "max-age=0")
.data("cookie", "__cfduid=d76231c8cccdbd5303a7d4feeb3f3a11f1466541718; _gat=1; _ga=GA1.2.1292204706.1466541721; request_method=POST; _session_id=5dc49c7814d5087ac51f9d9da20b2680")
.data("dnt", "1")
.data("upgrade-insecure-requests", "1")
.data("user-agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36")
.post();
What is the problem???

The problem is, .data() adds to the form data, not the header. So you need to use the appropriate methods to set the related information. Refer to below to fix your code:
To set the header:
.header("key", "value")
To set the form data:
.data("key", "value")
To set user agent:
.userAgent("Mozilla...")

How do i recreate a html request header?

I'm currently writing a Java Application that remotely controls my Roku. I found this website and used it to control my Roku. From Chromes developer tools i watched its data traffic and found the html request that controlled the Roku. The Header was this.
POST /keydown/Play HTTP/1.1
Host: 192.xxx.x.82:8060
Connection: keep-alive
Content-Length: 0
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://remoku.tv
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://remoku.tv/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
I then tried to recreate this POST request within Java and it ended up looking like this:
HttpURLConnection urlConn;
URL url = new URL("html://192.xxx.x.82:8060/keydown/Play");
urlConn = (HttpURLConnection) url.openConnection();
urlConn.setRequestProperty("Connection", "keep-alive");
urlConn.setRequestProperty("Content-Length", "0");
urlConn.setRequestProperty("Cache-Control", "max-age=0");
urlConn.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
urlConn.setRequestProperty("Origin", "http://192.xxx.x.254");
urlConn.setRequestProperty("Upgrade-Insecure-Requests", "1");
urlConn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36");
urlConn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
urlConn.setRequestProperty("Referer", "http://192.xxx.x.254");
urlConn.setRequestProperty("Accept-Encoding", "gzip, deflate");
urlConn.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
I'm not 100% sure this is the correct way to recreate the request because it does not have the same effect as the the original (working). However, this may be because I changed a few minor details that may have actually be important. So my question to you is if this the correct way to recreate a request and if it is why is it not working? If not what is? Any help is appreciated.

Thanks to tgkprog's comment i edited my code to this:
HttpURLConnection urlConn;
URL url = new URL("http://192.xxx.x.82:8060/keypress/Right");
urlConn = (HttpURLConnection) url.openConnection();
urlConn.setRequestMethod("POST");
urlConn.setDoOutput(true);
try(DataOutputStream wr = new DataOutputStream(urlConn.getOutputStream())) {
wr.writeChars("");
}
System.out.println(urlConn.getResponseCode());
and now it works perfectly and I can control my Roku the problem was i was not using the correct keys in the header as they where not in caps lock in Chrome (edit: they are not needed).

URL: fetch Last Modified only (without content)

Is there a way to fetch just Last Modified header by using a HttpConnection?
When I use this code:
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
long lastModified = httpCon.getLastModified();
In the log files of the webserver I see:
a.b.c.d - - [26/Dec/2015:10:25:50 +0100] "GET /file.txt HTTP/1.1" 200 484 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"
If I understand it correct, the GET request fetches the whole content as well. Is there a possibility to perform HEAD request to retrieve the headers only?

You can set the HTTP method, see HttpURLConnection:
Set the method for the URL request, one of:
GET
POST
HEAD
OPTIONS
PUT
DELETE
TRACE
are legal, subject to protocol restrictions. The default method is GET.
Example:
HttpURLConnection httpUrlConnection = (HttpURLConnection) url.openConnection();
httpUrlConnection.setRequestMethod("HEAD");
long lastModified = httpUrlConnection.getLastModified();

HttpURLConnection respond 404 when file clearly exist

I have a java program that is trying to read any arbitrary file from URL. However, it return an 404 error when the file clearly exist, try it for yourself with the URL. What is wrong?
URL url = new URL("http://images.all-free-download.com/images/graphiclarge/blue_abstract_background_310971.jpg");
HttpURLConnection myHTTPConTest = null;
myHTTPConTest = (HttpURLConnection) url.openConnection();
int responseCode = myHTTPConTest.getResponseCode(); // Returns 404
Added user-agent, no change:
myHTTPConTest.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401");

I think you had a bad luck and you just hit the server restart time or something like that.
Because it returns 200 now.