I'm working on a java program that will need to log into a ASP.NET web form, then once authenticated, download a file. Normal HTTP GET/POST is not a problem, but it appears that ASP is not giving me a SESSION ID when I connect from java, but it is from the browser.
When I look at the header information in Firefox, I see the cookies being set from the initial login, but then the page is immediately redirected over to a new URL. I'm not sure if it matters, but the page it redirects to after login contains iframes. I've tried loading both the main page and the iframe src inside, but neither give me the cookie in the header.
//Pull up the login page, extract out the hidden input variables __VIEWSTATE, __EVENTVALIDATION
URL url = new URL(loginPage);
HttpURLConnection conn = null;
conn = (HttpURLConnection) url.openConnection();
//This reads the page line-by-line and extracts out all the values from hidden input fields
Map<String,String> formFields = getViewstate(conn);
//Now re-open the URL to actually submit the POST data
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
DataOutputStream out = new DataOutputStream(conn.getOutputStream());
String postValues = URLEncoder.encode("txtUsername", "UTF-8") + "=" + URLEncoder.encode(uid, "UTF-8");
postValues += "&" + URLEncoder.encode("txtPassword", "UTF-8") + "=" + URLEncoder.encode(pwd, "UTF-8");
postValues += "&" + URLEncoder.encode("__EVENTTARGET", "UTF-8") + "=" + URLEncoder.encode("", "UTF-8");
postValues += "&" + URLEncoder.encode("__VIEWSTATE", "UTF-8") + "=" + URLEncoder.encode(formFields.get("viewstate"), "UTF-8");
postValues += "&" + URLEncoder.encode("__EVENTVALIDATION", "UTF-8") + "=" + URLEncoder.encode(formFields.get("eventvalidation"), "UTF-8");
out.writeBytes(postValues);
out.flush();
out.close();
//At this point looking at Firefox sniffer data, it should be sending back the cookie
//However there is no Set-Cookie in the header fields
for (int i = 1; (key = conn.getHeaderFieldKey(i)) != null; i++) {
// get ASP.NET_SessionId from cookie
if (key.equalsIgnoreCase("set-cookie")) {
sessionId = conn.getHeaderField(key);
sessionId = sessionId.substring(0, sessionId.indexOf(";"));
}
}
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = rd.readLine()) != null) {
//The page it prints out is the page it was redirected to when logged in through the browser
System.out.println(line);
}
rd.close();
//At this point, it was a successful login, but I never got the cookie so I'm stuck
HttpClient, which I believe HtmlUnit is based on, has the lower level functionality I think you're looking for. Handles cookies well, though if you need more, then Kurt is right in that you should look for something with more functionality. If you actually need to get full browser functionality, you could try something like Selenium/Webdriver that actually automates a browser under programmatic control.
It looks like the site you are trying to access relies on Cookies which are not supported by HttpURLConnection. A way around this issue is to use a library like HtmlUnit which simulates a browser (supports cookies, javascript, etc..).
Related
I'm working on a program that queries Google Safe Browsing for certain urls, but I'm getting an error that I don't think I should be getting.
I'm sending the following request:
2
http://google.com
http://facebook.com
via POST to: https://sb-ssl.google.com/safebrowsing/api/lookup?client=api&apikey=[KEY]&appver=1.5.2&pver=3.1
However, I'm getting a 403 response.
This is what the documentation says for HTTP POST lookup errors:
The server generates the following HTTP error codes for the POST request:
•200: AT LEAST ONE of the queried URLs are matched in either the phishing, malware, or unwanted software lists. The actual results are returned through the response body.
•204: NONE of the queried URLs matched the phishing, malware, or unwanted software lists, and no response body is returned.
•400: Bad Request—The HTTP request was not correctly formed.
•401: Not Authorized—The API key is not authorized.
•503: Service Unavailable—The server cannot handle the request. Besides the normal server failures, this could also indicate that the client has been “throttled” for sending too many requests.
The response code 403 isn't listed, yet I'm getting it.
I have triple-checked my API-key and made sure the API is enabled for my project. I'm using a Server-key, but I also tried a Browser-key.
I tried doing a GET request also, and that did work, but I cannot get POST to work. What's going on?
Here is my code:
try {
String baseURL="https://sb-ssl.google.com/safebrowsing/api/lookup";
String arguments = "";
arguments +=URLEncoder.encode("client", "UTF-8") + "=" + URLEncoder.encode("api", "UTF-8") + "&";
arguments +=URLEncoder.encode("apikey", "UTF-8") + "=" + URLEncoder.encode("[KEY]", "UTF-8") + "&";
arguments +=URLEncoder.encode("appver", "UTF-8") + "=" + URLEncoder.encode("1.5.2", "UTF-8") + "&";
arguments +=URLEncoder.encode("pver", "UTF-8") + "=" + URLEncoder.encode("3.1", "UTF-8");
// Construct the url object representing cgi script
URL url = new URL(baseURL + "?" + arguments);
// Get a URLConnection object, to write to POST method
HttpURLConnection connect = (HttpURLConnection) url.openConnection();
connect.setRequestMethod("POST");
// Specify connection settings
connect.setDoInput(true);
connect.setDoOutput(true);
// Get an output stream for writing
OutputStream output = connect.getOutputStream();
PrintStream pout = new PrintStream (output);
pout.print("2");
pout.println();
pout.print("http://www.google.com");
pout.println();
pout.print("http://www.facebook.com");
pout.close();
BufferedReader in = new BufferedReader(new InputStreamReader(connect.getInputStream()));
String decodedString;
while ((decodedString = in.readLine()) != null) {
System.out.println("w: " + decodedString);
}
in.close();
} catch(Exception e) {
e.printStackTrace();
}
I found the error. The CGI parameter was incorrect. It should have been key and not apikey. Still weird that you get an undocumented response-code though.
So there is this line of code
String data = URLEncoder.encode("image", "UTF-8") + "=" + URLEncoder.encode(Base64.encodeBase64String(baos.toByteArray()).toString(), "UTF-8");
data += "&" + URLEncoder.encode("key", "UTF-8") + "=" + URLEncoder.encode(YOUR API KEY GOES HERE, "UTF-8");
and when I registered for the Imgur API I was given a client_id and a client_secret and was wondering which one I use for where it says "YOUR API KEY GOES HERE" also in the first part in the second line where it says "key" what do I enter there? Also is the site to upload it http://imgur.com/api/upload because I have seen a few different ones.
try this out:
public static String getImgurContent(String clientID) throws Exception {
URL url;
url = new URL("https://api.imgur.com/3/image");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
String data = URLEncoder.encode("image", "UTF-8") + "="
+ URLEncoder.encode(IMAGE_URL, "UTF-8");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Authorization", "Client-ID " + clientID);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
conn.connect();
StringBuilder stb = new StringBuilder();
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
// Get the response
BufferedReader rd = new BufferedReader(
new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
stb.append(line).append("\n");
}
wr.close();
rd.close();
return stb.toString();
}
was almost like humpty dumpty, getting every piece back together, codes from everywhere, at least it worked as expected, its a shame they don't have examples...
enjoy.
ps: ou can also make with FILES (haven't tried yet) but you need to convert an image to base64 and then to utf8 (to replace the url)
edit, use this instead of the URL, so you can upload files:
//create base64 image
BufferedImage image = null;
File file = new File(imageDir);
//read image
image = ImageIO.read(file);
ByteArrayOutputStream byteArray = new ByteArrayOutputStream();
ImageIO.write(image, "png", byteArray);
byte[] byteImage = byteArray.toByteArray();
String dataImage = Base64.encode(byteImage);
String data = URLEncoder.encode("image", "UTF-8") + "="
+ URLEncoder.encode(dataImage, "UTF-8");
The site to upload to is - https://api.imgur.com/3/image or you can alternatively use the same link with "upload" instead of image.
I am currently trying to use the Imgur API myself and although I have not got it completely right yet (I can't seem to parse the URL response) I have looked at quite a few code examples for it. Are you definitely using version 3 of the API?
Because the homepage of the API says that you should give your client ID in this format "Authorization Client-ID YOUR_CLIENT_ID", not using "key" like you are.
Have a look at http://api.imgur.com/
Edit: you might find the following useful - Anonymous Uploading File object to Imgur API (JSON) gives Authentication Error 401
I am trying to login to a website and get page source of a page site after I login to the web site with java URLConnection. The problem I am facing is I can't maintain session so server gives me this warning and doesn't let me to get connected:
This system requires the use of HTTP cookies to verify authorization information.
Our system has detected that your browser has disabled HTTP cookies, or does not support them.
Please refer to the Help page in your browser for more information on how to correctly configure your browser for use with this system.
At first I am trying to send empty cookie to let server to understand I am handling sessions but it doesn't give me session id either.
This is my source code:
try {
// Construct data
String data = URLEncoder.encode("usr", "UTF-8") + "=" + URLEncoder.encode("usr", "UTF-8");
data += "&" + URLEncoder.encode("password", "UTF-8") + "=" + URLEncoder.encode("pass", "UTF-8");
// Send data
URL url = new URL("https://loginsite.com");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
conn.setRequestProperty("Cookie", "SESSID=");
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
wr.close();
rd.close();
String headerName=null;
for (int i=1; (headerName = conn.getHeaderFieldKey(i))!=null; i++) {
if (headerName.equals("Set-Cookie")) {
String cookie = conn.getHeaderField(i);
System.out.println(cookie.split(";", 2)[0]);
}
}
} catch (Exception e) {
}
You should use an HTTP library which handles session management and other details of the HTTP protocol for you, e.g. supports Cookies and things like Keep-Alive, Proxies etc. out of the box. Try Apache HttpComponents
I'm receiving the following SSLHandshakeException when trying to send a push notification through the C2DM servers.
javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate for: https://android.apis.google.com/c2dm/send
The code to send the message is as follows, and is running on App Engine. Everything works fine when I use cURL, so I know that the server authentication code and device registration ID are correct.
public static void sendHttpPostToC2dmService(String msg, PrintWriter out) {
String authCode = "XXXX";
String regID = "YYYY";
try {
URL url = new URL("https://android.apis.google.com/c2dm/send");
String data = URLEncoder.encode("registration_id", "UTF-8") + "="
+ URLEncoder.encode(regID, "UTF-8");
data += "&"
+ URLEncoder.encode("Authorization: GoogleLogin auth",
"UTF-8") + "="
+ URLEncoder.encode(authCode, "UTF-8");
data += "&" + URLEncoder.encode("collapse_key", "UTF-8") + "="
+ URLEncoder.encode("something", "UTF-8");
data += "&" + URLEncoder.encode("data.message", "UTF-8") + "="
+ URLEncoder.encode(msg, "UTF-8");
out.println("data=" + data);
HttpURLConnection connection = (HttpURLConnection) url
.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
OutputStreamWriter writer = new OutputStreamWriter(
connection.getOutputStream());
writer.write(data);
writer.close();
out.println("responseCode=" + connection.getResponseCode());
if (connection.getResponseCode() == HttpURLConnection.HTTP_OK) {
String responseLine = new BufferedReader(new InputStreamReader(
connection.getInputStream())).readLine();
out.println("responseLine=" + responseLine);
} else {
// Server returned HTTP error code.
}
} catch (MalformedURLException e) {
out.println("MalformedURL");
out.println(e.getMessage());
} catch (IOException e) {
out.println("IOException");
out.println(e.getMessage());
e.printStackTrace(out);
}
}
It seems like others have also had this problem, but I haven't been able to find a clear solution (at least not one I'm able to understand). Appreciate any help.
You need to use HttpsURLConnection instead of HttpURLConnection.
AFAIK, HttpURLConnection doesn't verify hostnames.
If you are in the testing phase, you could use the non-secure url to push a notification.
http://android.apis.google.com/c2dm/send
I also got around this issue by adding a certificate validation callback to accept any certificate, but this was on a C# server. I'm sure there is a Java equivalent for this.
In production code you would want to verify the correct certificate or maybe just encrypt the data you are sending.
I'm a bit new to Java and more to connections stuff with it. I'm trying to create a program to connect to a website ("www.buybackprofesional.com") where I would like to download pictures and get some text from cars (after the login you have to enter a plate number to access a car's file).
This is what I have right now, but it always says that the session has expired, I need a way to login using the username and password of the mainpage, am I right? can someone give me some advice? Thanks
Note: I want to do it in Java, maybe I was not clear in the question.
//URL web = new URL("http://www.buybackprofesional.com/DetallePeri.asp?mat=9073FCV&fec=27/07/2010&tipo=C&modelo=4582&Foto=0");
URL web = new URL("http://www.buybackprofesional.com/");
HttpURLConnection con = (HttpURLConnection) web.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; JVM)");
con.setRequestProperty("Pragma", "no-cache");
con.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
A colleage helped me with this so I'll post the code that works:
public static URLConnection login(String _url, String _username, String _password) throws IOException, MalformedURLException {
String data = URLEncoder.encode("Usuario", "UTF-8") + "=" + URLEncoder.encode(_username, "UTF-8");
data += "&" + URLEncoder.encode("Contrase", "UTF-8") + "=" + URLEncoder.encode(_password, "UTF-8");
// Send data
URL url = new URL(_url);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
wr.close();
return conn;
}
This will submit the form info on the page I need and after that, using cookies I can stay connected!
To connect to a website using java consider using httpunit or httpcore (offered by apache). They handle sessions much better then you (or I) could do on your own.
Edit: Fixed the location of the link. Thanks for the correction!