Develop Reference

How to parse the web content using HttpURLConnection and TagNode class?

I would like to take some data from www.groupon.pl website. To do this I used code below:
String currentUrl = "https://www.groupon.pl/browse/radom";
URL urlObj = new URL(currentUrl);
HttpURLConnection urlConnection = (HttpURLConnection) urlObj.openConnection();
urlConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0");
urlConnection.connect();
final HtmlCleaner cleaner = new HtmlCleaner();
final TagNode tagNodeRoot = cleaner.clean(urlConnection.getInputStream());
The problem is that after the last line:
final TagNode tagNodeRoot = cleaner.clean(urlConnection.getInputStream());
The program doesn't want go further. I think that maybe the problem is with:
urlConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0");
Or maybe I use some and old API which is not supported any more? I use debugger and it doesn't stepping out. When I run this code normally without any debugger it also stops. No error at the console. I don't know maybe it is waiting for the response. Could you tell me what is wrong? Thank you.

HttpURLConnection respond 404 when file clearly exist

I have a java program that is trying to read any arbitrary file from URL. However, it return an 404 error when the file clearly exist, try it for yourself with the URL. What is wrong?
URL url = new URL("http://images.all-free-download.com/images/graphiclarge/blue_abstract_background_310971.jpg");
HttpURLConnection myHTTPConTest = null;
myHTTPConTest = (HttpURLConnection) url.openConnection();
int responseCode = myHTTPConTest.getResponseCode(); // Returns 404
Added user-agent, no change:
myHTTPConTest.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401");

I think you had a bad luck and you just hit the server restart time or something like that.
Because it returns 200 now.

Program can not login into website

This is my code:
URL url = new URL("http://superchillin.com/login2.php");
HttpURLConnection urlConnection = (HttpURLConnection)url.openConnection();
urlConnection.setUseCaches(false);
urlConnection.setRequestMethod("POST");
String data = "email="+URLEncoder.encode(name, "UTF-8")+"&password="+URLEncoder.encode(pass, "UTF-8");
urlConnection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
urlConnection.setRequestProperty("Accept-Encoding", "gzip,deflate");
urlConnection.setRequestProperty("Accept-Language", "en-US,en;q=0.8,lt;q=0.6");
urlConnection.setRequestProperty("Cache-Control", "max-age=0");
urlConnection.setRequestProperty("Connection", "keep-alive");
urlConnection.setRequestProperty("Content-Length", Integer.toString(data.getBytes().length));
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
urlConnection.addRequestProperty("Cookie", "place=1");
urlConnection.addRequestProperty("Cookie", "lvca_unique_user=1");
urlConnection.setRequestProperty("Host", "superchillin.com");
urlConnection.setRequestProperty("Origin", "http://superchillin.com");
urlConnection.setRequestProperty("Referer", "http://superchillin.com/login.php");
urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36");
urlConnection.setDoOutput(true);
urlConnection.setDoInput(true);
urlConnection.setInstanceFollowRedirects(true);
DataOutputStream wr = new DataOutputStream(urlConnection.getOutputStream());
wr.writeBytes(data);
wr.flush();
wr.close();
After that code I only read the response. It redirects me to "login.php" and is trying to set cookie "place=1"...
Connecting via browser works great. The reason for so many headers is I thought they may be the problem so I copied all headers from which I see when using a browser.
The response code is 200.
I also noticed that if password or email is incorrect, there's a message saying that in HTML which i retrieve.
When I use a browser I get redirected to index.php and cookie "auth" is set. So that's what I'm expecting from my program aswell. Curently I get redirected back to "login.php".

There is no universal answer to this question, I'm afraid. What you're asking is "why does the remote server not return an auth cookie when I send this exact request?" And that depends entirely on what the server's documentation says about those requests, whether it has any bugs in its implementation, etc.
If you don't have access to the server's own source and logs, then you'll likely have to get by with experimentation. Use something like Firebug or Chrome's Developer Tools to capture the exact requests sent by the browser with the login works successfully. Since these text strings are the only thing the remote server sees, if you replicate them exactly with your Java program you will(/should) get exactly the same responses.
If you think you're sending the same requests from Java and find that you're still not getting the expected responses, there must be some difference. Try recording the network traffic with something like Wireshark in order to see exactly what your app is sending - and then address the differences.
And if you get to the point where e.g. a redirect isn't being followed, and you're not sure how to do that with a URLConnection - then that's a good concrete question to ask.

login to http website java

I am trying to login to a http website for the first time and I am having a hard time understanding the proper format for sending arguments. I have looked at other examples and they don't seem to work for me so I thought I would see if someone can explain this to me. At this point my code seems to do absolutely nothing but here it is...
HttpURLConnection url= (HttpURLConnection)new URL("http://www.myameego.com/index2.php?do=login").openConnection();
url.setDoOutput(true);
url.setRequestMethod("POST");
OutputStreamWriter writer = new OutputStreamWriter(url.getOutputStream());
writer.write("X-Mapping-fjhppofk=6A991610BA398B3A39F4B491D5382BB4;
PHPSESSID=kbo25e08t3qvu08l1shkq8kk94; userName=coled; pass=ed45d626b07112a8a501d9672f3b92796a6754b8d8d9cb4c617fec9774889220; clientID=129; X-Mapping-fjhppofk=DCE62FE972E1EF2F12D0060EC74C3681; PHPSESSID=ukeo21oldb5pqsntu7kl8j3b96");
writer.flush();
I downloaded an http sniffer thinking that I could read what the browser was sending. that is how I got the write() line, it is the cookie that was sent by explorer. I also viewed the source code for the login screen and found a block of code near the bottom that looks like its responsible for login.
http://www.myameego.com/index2.php?do=login
Can someone tell me how I would go about hooking into this interface I don't understand how this works. if it helps this is the full packet from my manual login through the browser. I got it from my http sniffer.
Host Name: www.myameego.com
Method: POST
Path: /index2.php?do=login
User Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; NP06)
Response Code: 302
Response String: found
Content Type: text/html; charset=UTF-8
Referer: http://www.myameego.com/index.php?do=login
Transfer Encoding: chunked
Server: Apache
Content Length: 17817
Connection: Keep-Alive
Cache Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Location: /Ameego/index.php
Cookie: X-Mapping-fjhppofk=6A991610BA398B3A39F4B491D5382BB4; PHPSESSID=kbo25e08t3qvu08l1shkq8kk94; userName=coled; pass=ed45d626b07112a8a501d9672f3b92796a6754b8d8d9cb4c617fec9774889220; clientID=129; X-Mapping-fjhppofk=DCE62FE972E1EF2F12D0060EC74C3681; PHPSESSID=ukeo21oldb5pqsntu7kl8j3b96
URL: http://www.myameego.com/index2.php?do=login
How can I make a packet like the one above? any guidance would be greatly appreciated.
I Looked into that link you posted and the http sniffer shows that the POST request is being called but the cookie line doesn't match up with that of the manual browser request.
HttpURLConnection httpConnection = (HttpURLConnection)new URL("http://www.myameego.com/index2.php?do=login").openConnection();
httpConnection.setDoOutput(true);
httpConnection.setRequestMethod("POST");
httpConnection.setRequestProperty("Accept-Charset","UTF-8");
httpConnection.setRequestProperty("User-Agent","Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; NP06)");
httpConnection.setRequestProperty("Content-Type","application/x-www-form-urlencoded;charset=UTF-8");
String info = String.format("user=%s&coled=%s",URLEncoder.encode("user","UTF-8"),URLEncoder.encode("coled","UTF-8"));
info += String.format("pass=%s&MYPASS=%s",URLEncoder.encode("pass","UTF-8"),URLEncoder.encode("MYPASS","UTF-8"));
info += String.format("clientID=%s&129=%s",URLEncoder.encode("clientID","UTF-8"),URLEncoder.encode("129","UTF-8"));
info += String.format("login=%s&Sign In=%s",URLEncoder.encode("login","UTF-8"),URLEncoder.encode("Sign In","UTF-8"));
httpConnection.setRequestProperty("Cookie",info);
OutputStream output = httpConnection.getOutputStream();
output.write(info.getBytes("UTF-8"));
int x;
while((x = httpConnection.getInputStream().read()) != -1)System.out.print((char)x);
my Cookie:
user=user&coled=coledpass=pass&MYPASS=MYPASSclientID=clientID&129=129login=login&Sign In=Sign+In
browsers cookie:
X-Mapping-fjhppofk=6A991610BA398B3A39F4B491D5382BB4; PHPSESSID=112tg9i4afau5i382hui705553
anyone know what I may be missing here?

With Jsoup this should be simple like this:
Connection.Response response = Jsoup.connect("http://www.myameego.com/index2.php?do=login")
.method(Connection.Method.GET)
.execute();
Document page = Jsoup.connect("http://www.myameego.com/index2.php?do=login")
.data("user", "login")
.data("pass", "password")
.data("clientID", "123456")
.cookies(response.cookies())
.post();
Gathered with Google Chrome Developer Tools

What is the proper way of setting headers in a URLConnection?

My code is like the following:
URLConnection cnx = address.openConnection();
cnx.setAllowUserInteraction(false);
cnx.setDoOutput(true);
cnx.addRequestProperty("User-Agent",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
InputStream is = cnx.getInputStream();
Is it ok if I set the headers before I get the InputStream? Will my header be sent, or will the server see the default URLConnection's user-agent ( if any ) ?

The headers must be set prior to getting the InputStream to have any affect - an IllegalStateException will be thrown if the connection is already open.
As far as the User-Agent header specifically, it should be sent if it has been set.
See the URLConnection JavaDoc.

To answer the question, the code is correct. The moment getInputStream(), an HTTP get is sent to the target server.
A side-note on user-agent, if you don't set it, URLConnection will send the default one anyway, which is:
User-Agent: Java/1.6.0_24 (varies depending on your java version)

I'd advise against using low-level constructs such as URLConnection. There are plenty of libraries for sending HTTP requests, with the most prominent being Apache HTTP Client.