I have a java method which should get Set-Cookie property for following login into webpage. But the conn.getHeaderFields().get("Set-Cookie") does not return anything. Any advice?
private String GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "s-CZ,cs;q=0.8,en;q=0.6");
if (cookies != null) {
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream(),"UTF-8"));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
System.out.println(conn.getHeaderFields().get("Set-Cookie")); //print for testing
setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
Whole program: http://pastebin.com/3nB682L7
Anyone?..:-)
Recent Java versions have "fixed" URLConnection to hide cookies that are marked HttpOnly, and I don't think there's a setting to disable that. I would recommend using HttpClient from Apache HttpComponents.
Related
I try to request a transaction list from paypal but I allways get HTTP-Code 400
public void getTransactionList(String accessToken)
{
try
{
URL url = new URL(
"https://api.sandbox.paypal.com/v1/reporting/transactions"
+ "?start_date=2018-01-01T00:00:00Z&end_date=2018-04-01T00:00:00Z"
+ "&fields=all&page_size=100&page=1");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Authorization", "Bearer " + accessToken);
conn.setRequestProperty("Content-Type", "application/json");
conn.setRequestProperty("Accept-Language", "en_US");
conn.setUseCaches(false);
conn.setDoOutput(true);
String line;
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = reader.readLine()) != null)
{
System.out.println(line);
}
reader.close();
}
catch (Throwable e)
{
e.printStackTrace();
//throw new ActionException(e);
}
}
The Exception is
java.io.IOException: Server returned HTTP response code: 400 for URL: https://api.sandbox.paypal.com/v1/reporting/transactions?start_date=2018-01-01T00:00:00-000&end_date=2018-04-02T00:00:00-000&fields=all&page_size=100&page=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at de.crefo.workflow.paypal.http.HttpRestClient.getTransactionList(HttpRestClient.java:42)
at de.crefo.workflow.paypal.http.HttpRestClient.main(HttpRestClient.java:112)
The Access-Token request, I do in a similar way, is working perfectly fine and gives me a valid token.
What am I doing wrong?
EDIT. Changed dateformat in URL, see comments.
Error description:
java.lang.NoSuchMethodError: org.apache.http.HttpHost.getAddress()Ljava/net/InetAddress;
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:102)
Could anyone help me here....what i am missing ???
use below sample code :
// HTTP POST request
private void sendPost() throws Exception {
String url = "https://selfsolve.apple.com/wcResults.do";
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
String urlParameters = "sn=C02G8416DRJM&cn=&locale=&caller=& num=12345";
// Send post request
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + urlParameters);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
}
}
I am trying to invoke a url through java using java.net.HttpURLConnection.
Below is the code.
I get 401 as response. The url is up.
// HTTP GET request
private void sendGet() throws Exception {
String url = "http://10.10.200.151:8720/scheduler/stat.go?opt1=0&opt2=0&opt3=0";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
//add request header
con.setRequestProperty("User-Agent", USER_AGENT);
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
//print result
System.out.println(response.toString());
}
Is there something missing.
HTTP status 401 indicate that the request requires authentication.
Perhaps this url need login, and the server check this by your cookie.
I'm currently trying to log in to a php server using HttpURLConnection. I have been successfully able to connect to the server/website as the response code I'm getting back is 200.
getPageContent has a response code of 200 while sendPost has a response code of 0. I am unsure about which getRequestProperty's are necessary and which aren't. I have looked at the elements and response and request headers and form data to get the correct information for the properties I did include however.
I can't really make too much sense of the string coming out of getPageContent as it is quite long but it does seem to be correct. getFormParams could possibly be outputting an incorrect string but information such as the username and password are included so I am unsure.
I'm stuck as to what's wrong or right in my attempt. Am I getting the form parameters correctly? And how do I send the login data correctly?
I call:
String page = getPageContent(URL);
String PostParams = getFormParams( page, "user", "pass" );
sendPost( URL, PostParams );
And the functions are:
private String getPageContent(String url) throws Exception {
URL obj = new URL( url );
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
conn.setRequestProperty("User-Agent", AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
if (cookies != null) {
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
MainActivity.globalV.responsecode = responseCode;
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
setCookies(conn.getHeaderFields().get("Set-Cookie"));
MainActivity.globalV.getPageContent = responseCode;
return response.toString();
}
public String getFormParams(String html, String username, String password)
throws UnsupportedEncodingException {
Document doc = Jsoup.parse(html);
Elements inputElements = doc.getElementsByTag("input");
List<String> paramList = new ArrayList<String>();
for (Element inputElement : inputElements) {
String key = inputElement.attr("name");
String value = inputElement.attr("value");
if (key.equals("user_username"))
value = username;
else if (key.equals("user_password"))
value = password;
paramList.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
// build parameters list
StringBuilder result = new StringBuilder();
for (String param : paramList) {
if (result.length() == 0) {
result.append(param);
} else {
result.append("&" + param);
}
}
return result.toString();
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL( url );
HttpURLConnection conn = ( HttpsURLConnection) obj.openConnection();
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "host website");
conn.setRequestProperty("User-Agent", AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.8");
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Referer", "referer url");
conn.setRequestProperty("Content-Type", "text/html");
conn.setRequestProperty("Content-Length", Integer.toString(postParams.length()));
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
MainActivity.globalV.sendPost = conn.getResponseCode();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
}
Im trying to automate a login using java, and have used this example: http://www.mkyong.com/java/how-to-automate-login-a-website-java-example/ to help me do it with another website. The code is below, and so is the output. My question is, what does the error mean, and how do I fix it?
public class testing {
private List<String> cookies;
private HttpsURLConnection conn;
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception {
String url = "https://www.studentinvestor.org/secure/login.php?dest=http://www.studentinvestor.org/stock-list.php";
String companies = "http://www.studentinvestor.org/stock-list.php";
testing http = new testing();
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
// 1. Send a "GET" request, so that you can extract the form's data.
String page = http.GetPageContent(url);
String postParams = http.getFormParams(page, "username", ",password");
// 2. Construct above post's content and then send a POST request for
// authentication
http.sendPost(url, postParams);
// 3. success then go to gmail.
String result = http.GetPageContent(companies);
System.out.println(result);
}
private void sendPost(String url, String postParams) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// Acts like a browser
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "www.studentinvestor.org");
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Content-Type", "text/html");
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(postParams);
wr.flush();
wr.close();
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + postParams);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// System.out.println(response.toString());
}
private String GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");
if (cookies != null) {
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
}
public String getFormParams(String html, String username, String password)
throws UnsupportedEncodingException {
System.out.println("Extracting form's data...");
Document doc = Jsoup.parse(html);
// Google form id
Element loginform = doc.getElementById("loginsubmitted");
Elements inputElements = loginform.getElementsByTag("label");
List<String> paramList = new ArrayList<String>();
for (Element inputElement : inputElements) {
String key = inputElement.attr("name");
String value = inputElement.attr("value");
if (key.equals("team-name"))
value = username;
else if (key.equals("team-password"))
value = password;
paramList.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
// build parameters list
StringBuilder result = new StringBuilder();
for (String param : paramList) {
if (result.length() == 0) {
result.append(param);
} else {
result.append("&" + param);
}
}
return result.toString();
}
public List<String> getCookies() {
return cookies;
}
public void setCookies(List<String> cookies) {
this.cookies = cookies;
}
}
OUTPUT
Sending 'GET' request to URL : https://www.studentinvestor.org/secure/login.php? dest=http://www.studentinvestor.org/stock-list.php
Response Code : 200
Extracting form's data...
Sending 'POST' request to URL : https://www.studentinvestor.org/secure/login.php? dest=http://www.studentinvestor.org/stock-list.php
Post parameters :
Response Code : 200
Exception in thread "main" java.lang.ClassCastException: sun.net.www.protocol.http.HttpURLConnection cannot be cast to javax.net.ssl.HttpsURLConnection
at testing.GetPageContent(testing.java:98)
at testing.main(testing.java:44)
Therefore the error message is:
Exception in thread "main" java.lang.ClassCastException: sun.net.www.protocol.http.HttpURLConnection cannot be cast to javax.net.ssl.HttpsURLConnection
at testing.GetPageContent(testing.java:98)
at testing.main(testing.java:44)
Stupid me! If you get this error, make sure your URLs are HTTPS, not just http or www!
Look at the exception : sun.net.www.protocol.http.HttpURLConnection cannot be cast to javax.net.ssl.HttpsURLConnection
Here the protocol of required Url should be Https instead of HTTP.
so using the below code will solve your issue:
String companies = "https://www.studentinvestor.org/stock-list.php";