I'm currently trying to log in to a php server using HttpURLConnection. I have been successfully able to connect to the server/website as the response code I'm getting back is 200.
getPageContent has a response code of 200 while sendPost has a response code of 0. I am unsure about which getRequestProperty's are necessary and which aren't. I have looked at the elements and response and request headers and form data to get the correct information for the properties I did include however.
I can't really make too much sense of the string coming out of getPageContent as it is quite long but it does seem to be correct. getFormParams could possibly be outputting an incorrect string but information such as the username and password are included so I am unsure.
I'm stuck as to what's wrong or right in my attempt. Am I getting the form parameters correctly? And how do I send the login data correctly?
I call:
String page = getPageContent(URL);
String PostParams = getFormParams( page, "user", "pass" );
sendPost( URL, PostParams );
And the functions are:
private String getPageContent(String url) throws Exception {
URL obj = new URL( url );
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
conn.setRequestProperty("User-Agent", AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
if (cookies != null) {
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
MainActivity.globalV.responsecode = responseCode;
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
setCookies(conn.getHeaderFields().get("Set-Cookie"));
MainActivity.globalV.getPageContent = responseCode;
return response.toString();
}
public String getFormParams(String html, String username, String password)
throws UnsupportedEncodingException {
Document doc = Jsoup.parse(html);
Elements inputElements = doc.getElementsByTag("input");
List<String> paramList = new ArrayList<String>();
for (Element inputElement : inputElements) {
String key = inputElement.attr("name");
String value = inputElement.attr("value");
if (key.equals("user_username"))
value = username;
else if (key.equals("user_password"))
value = password;
paramList.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
// build parameters list
StringBuilder result = new StringBuilder();
for (String param : paramList) {
if (result.length() == 0) {
result.append(param);
} else {
result.append("&" + param);
}
}
return result.toString();
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL( url );
HttpURLConnection conn = ( HttpsURLConnection) obj.openConnection();
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "host website");
conn.setRequestProperty("User-Agent", AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Referer", "referer url");
conn.setRequestProperty("Content-Type", "text/html");
conn.setRequestProperty("Content-Length", Integer.toString(postParams.length()));
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
MainActivity.globalV.sendPost = conn.getResponseCode();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
}
Related
I just started working with JHipster and I created a .jh file with the entities I want. I would like to fill all the entities by doing an HTTP request to an existing HTTP:PORT that already has my needed information.
I would like to know if it's possible? I'm currently trying with this method that I found online:
private static void sendPOST() throws IOException {
// URL obj = new URL(POST_URL);
Authenticator.setDefault(new Authenticator() {
#Override
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("admin", "admin".toCharArray());
}
});
String url = "http://localhost:8080/api/table-simple-tasks";
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
// HttpURLConnection con = (HttpURLConnection) URL.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
// For POST only - START
con.setDoOutput(true);
OutputStream os = con.getOutputStream();
os.write(POST_PARAMS.getBytes());
os.flush();
os.close();
// For POST only - END
int responseCode = con.getResponseCode();
System.out.println("POST Response Code :: " + responseCode);
if (responseCode == HttpURLConnection.HTTP_OK) { //success
BufferedReader in = new BufferedReader(new InputStreamReader(
con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// print result
System.out.println(response.toString());
} else {
System.out.println("POST request not worked");
}
}
I would like to login to amazon seller central using java and get all the feedbacks that has been submitted for a seller. I followed the approach mentioned at http://www.mkyong.com/java/how-to-automate-login-a-website-java-example/, but no success.
Below is the code that I am using. Please let me know what wrong I am doing:
public class TestStringSplit {
private List<String> cookies;
private HttpsURLConnection conn;
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception {
String url = "https://sellercentral.amazon.in/gp/feedback-manager/view-all-feedback.html";
TestStringSplit http = new TestStringSplit();
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
// 1. Send a "GET" request, so that you can extract the form's data.
HashMap<String, String> hm = http.GetPageContent(url);
url = hm.keySet().iterator().next();
String page = hm.get(url);
String postParams = http.getFormParams(page, "xxx#gmail.com", "abcdef#123");
// 2. Construct above post's content and then send a POST request for
// authentication
http.sendPost(url, postParams);
// 3. success then go to gmail.
// String result =
// (http.GetPageContent(targetURL)).entrySet().iterator().next().getValue();
// System.out.println(result);
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// Acts like a browser
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "sellercentral.amazon.in");
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
// conn.setRequestProperty("Referer",
// "https://accounts.google.com/ServiceLoginAuth");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", Integer.toString(postParams.length()));
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + postParams);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
System.out.println(response.toString());
}
private HashMap<String, String> GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
if (cookies != null) {
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
setCookies(conn.getHeaderFields().get("Set-Cookie"));
System.out.println("redirected url: " + conn.getURL());
HashMap<String, String> hm = new HashMap<>();
hm.put(conn.getURL().toString(), response.toString());
return hm;
}
public String getFormParams(String html, String username, String password) throws UnsupportedEncodingException {
System.out.println("Extracting form's data...");
Document doc = Jsoup.parse(html);
// Google form id
Element loginform = doc.getElementById("ap_signin_form");
Elements inputElements = loginform.getElementsByTag("input");
List<String> paramList = new ArrayList<String>();
for (Element inputElement : inputElements) {
String key = inputElement.attr("name");
String value = inputElement.attr("value");
if (key.equalsIgnoreCase("Email"))
value = username;
else if (key.equalsIgnoreCase("Password"))
value = password;
paramList.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
// build parameters list
StringBuilder result = new StringBuilder();
for (String param : paramList) {
if (result.length() == 0) {
result.append(param);
} else {
result.append("&" + param);
}
}
return result.toString();
}
public List<String> getCookies() {
return cookies;
}
public void setCookies(List<String> cookies) {
this.cookies = cookies;
}
}
I have the following method that works fine, but it throws an exception when the server returns 403 code which results in the method to never return the server response.
public String ping(String lat, String lon)
{
StringBuffer response = null;
try
{
String url = "https://api.mysite.com";
URL urlObj = new URL(url);
HttpsURLConnection con = null;
if (useProxy)
{
con = (HttpsURLConnection) urlObj.openConnection(proxy);
}
else
{
con = (HttpsURLConnection) urlObj.openConnection();
}
// add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Content-Type", "application/json; charset=utf-8");
con.setRequestProperty("Host", urlObj.getHost());
con.setRequestProperty("Connection", "Keep-Alive");
// con.setRequestProperty("Accept-Encoding", "gzip");
String urlParameters = "{\"lat\":" + lat + ",\"lon\":" + lon + "}";
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
// System.out.println("\nSending 'POST' request to URL : " + url);
// System.out.println("Post parameters : " + urlParameters);
// System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
response = new StringBuffer();
while ((inputLine = in.readLine()) != null)
{
response.append(inputLine);
}
in.close();
// print result
// System.out.println(response.toString());
}
catch (Exception e)
{
e.printStackTrace();
}
return response.toString();
}
How can I make this return the server response no matter what the server returns and even if the server responds with 403 code (or any other response code)?
You only need to add an if block to make the code more robust like below:
public String ping(String lat, String lon)
{
StringBuffer response = null;
try
{
String url = "https://api.mysite.com";
URL urlObj = new URL(url);
HttpsURLConnection con = null;
if (useProxy)
{
con = (HttpsURLConnection) urlObj.openConnection(proxy);
}
else
{
con = (HttpsURLConnection) urlObj.openConnection();
}
// add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Content-Type", "application/json; charset=utf-8");
con.setRequestProperty("Host", urlObj.getHost());
con.setRequestProperty("Connection", "Keep-Alive");
// con.setRequestProperty("Accept-Encoding", "gzip");
String urlParameters = "{\"lat\":" + lat + ",\"lon\":" + lon + "}";
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
response = readResponse(con.getInputStream());
} else {
response = readResponse(con.getErrorStream());
}
// print result
System.out.println(response.toString());
}
catch (Exception e)
{
e.printStackTrace();
}
return response.toString();
}
private StringBuffer readResponse(InputStream in) {
BufferedReader in = new BufferedReader(new InputStreamReader(in));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null)
{
response.append(inputLine);
}
in.close();
return response;
}
HTH.
I have a java method which should get Set-Cookie property for following login into webpage. But the conn.getHeaderFields().get("Set-Cookie") does not return anything. Any advice?
private String GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "s-CZ,cs;q=0.8,en;q=0.6");
if (cookies != null) {
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream(),"UTF-8"));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
System.out.println(conn.getHeaderFields().get("Set-Cookie")); //print for testing
setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
Whole program: http://pastebin.com/3nB682L7
Anyone?..:-)
Recent Java versions have "fixed" URLConnection to hide cookies that are marked HttpOnly, and I don't think there's a setting to disable that. I would recommend using HttpClient from Apache HttpComponents.
Im trying to automate a login using java, and have used this example: http://www.mkyong.com/java/how-to-automate-login-a-website-java-example/ to help me do it with another website. The code is below, and so is the output. My question is, what does the error mean, and how do I fix it?
public class testing {
private List<String> cookies;
private HttpsURLConnection conn;
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception {
String url = "https://www.studentinvestor.org/secure/login.php?dest=http://www.studentinvestor.org/stock-list.php";
String companies = "http://www.studentinvestor.org/stock-list.php";
testing http = new testing();
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
// 1. Send a "GET" request, so that you can extract the form's data.
String page = http.GetPageContent(url);
String postParams = http.getFormParams(page, "username", ",password");
// 2. Construct above post's content and then send a POST request for
// authentication
http.sendPost(url, postParams);
// 3. success then go to gmail.
String result = http.GetPageContent(companies);
System.out.println(result);
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// Acts like a browser
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "www.studentinvestor.org");
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Content-Type", "text/html");
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + postParams);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// System.out.println(response.toString());
}
private String GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");
if (cookies != null) {
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
}
public String getFormParams(String html, String username, String password)
throws UnsupportedEncodingException {
System.out.println("Extracting form's data...");
Document doc = Jsoup.parse(html);
// Google form id
Element loginform = doc.getElementById("loginsubmitted");
Elements inputElements = loginform.getElementsByTag("label");
List<String> paramList = new ArrayList<String>();
for (Element inputElement : inputElements) {
String key = inputElement.attr("name");
String value = inputElement.attr("value");
if (key.equals("team-name"))
value = username;
else if (key.equals("team-password"))
value = password;
paramList.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
// build parameters list
StringBuilder result = new StringBuilder();
for (String param : paramList) {
if (result.length() == 0) {
result.append(param);
} else {
result.append("&" + param);
}
}
return result.toString();
}
public List<String> getCookies() {
return cookies;
}
public void setCookies(List<String> cookies) {
this.cookies = cookies;
}
}
OUTPUT
Sending 'GET' request to URL : https://www.studentinvestor.org/secure/login.php? dest=http://www.studentinvestor.org/stock-list.php
Response Code : 200
Extracting form's data...
Sending 'POST' request to URL : https://www.studentinvestor.org/secure/login.php? dest=http://www.studentinvestor.org/stock-list.php
Post parameters :
Response Code : 200
Exception in thread "main" java.lang.ClassCastException: sun.net.www.protocol.http.HttpURLConnection cannot be cast to javax.net.ssl.HttpsURLConnection
at testing.GetPageContent(testing.java:98)
at testing.main(testing.java:44)
Therefore the error message is:
Exception in thread "main" java.lang.ClassCastException: sun.net.www.protocol.http.HttpURLConnection cannot be cast to javax.net.ssl.HttpsURLConnection
at testing.GetPageContent(testing.java:98)
at testing.main(testing.java:44)
Stupid me! If you get this error, make sure your URLs are HTTPS, not just http or www!
Look at the exception : sun.net.www.protocol.http.HttpURLConnection cannot be cast to javax.net.ssl.HttpsURLConnection
Here the protocol of required Url should be Https instead of HTTP.
so using the below code will solve your issue:
String companies = "https://www.studentinvestor.org/stock-list.php";