So, I want to get data from aspx page. And I need to do the follow steps
1)Send GET request and get __VIEWSTATE and __EVENTVALIDATION.
2)Send POST request with parameters (to login) then update __VIEWSTATE and __EVENTVALIDATION
3)Send POST request with parameters (to choose field) then update variables too
4)Send POST request with parameters (to press button)
5)Parse page content.
Page content dynamically loaded after script.
I tryed to use HttpUrlConnection
public String sendGet(String url) throws Exception {
StringBuilder result = new StringBuilder();
URL ur = new URL(url);
HttpURLConnection connection = (HttpURLConnection) ur.openConnection();
connection.setRequestMethod("GET");
connection.setUseCaches(false);
connection.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36");
connection.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
connection.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputLine;
while ((inputLine = br.readLine()) != null) {
result.append(inputLine);
}
br.close();
return result.toString();
}
public String sendPost(String url, List<NameValuePair> formParams) throws Exception {
URL ur = new URL(url);
HttpURLConnection connection = (HttpURLConnection) ur.openConnection();
connection.setReadTimeout(10000);
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setConnectTimeout(10000);
connection.setRequestMethod("POST");
for (HttpCookie cookie : this.cookies) {
connection.addRequestProperty("Cookie", cookie.toString());
}
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setRequestProperty("Content-Language", "en-US");
connection.connect();
OutputStream os = connection.getOutputStream();
BufferedWriter writer = new BufferedWriter(
new OutputStreamWriter(os, "UTF-8"));
writer.write(getQuery(formParams));
writer.flush();
writer.close();
os.close();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String response = "";
String line;
while ((line = br.readLine()) != null) {
response += line;
}
System.out.println(Jsoup.parse(response).text());
updateViewState(response);
updateSubSession(response);
return response;
}
And I stayed in the second step. POST and GET subSession is different , but Cookie is the same :). So, EVENTVALIDATION and VIEWSTATE is invalid, and I get "login.aspx" page again. This is how I get Cookie
List<HttpCookie> cookies;
public void init() throws Exception {
CookieManager manager = new CookieManager();
manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(manager);
String html = sendGet(url);
updateViewState(html);
updateSubSession(html);
CookieStore cookieJar = manager.getCookieStore();
cookies=cookieJar.getCookies();
}
So, I tryed to send POST with subSession.
public String sendPost(String url, List<NameValuePair> formParams) throws Exception {
URL ur = new URL(url+subSession);
HttpURLConnection connection = (HttpURLConnection) ur.openConnection();
And...
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 500 for URL: http://www.aogc2.state.ar.us:8080/DWClient/Login.aspx?DWSubSession=9852&v=1589
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1840)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at com.company.UrlConnection.sendPost(UrlConnection.java:111)
at com.company.UrlConnection.login(UrlConnection.java:138)
at com.company.UrlConnection.start(UrlConnection.java:35)
at com.company.Main.main(Main.java:11)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
I thought that I was wrong to get cookie and I tryed to get cookie like this:
connection.getHeaderFields().get("Set-Cookie");
but Cookie were null .
Where I was wrong? Thank you a lot!
Related
I've this code in Java 8
String u = "https://test.com?WSDL";
URL url = new URL(u);
System.out.println("Link: " + u);
SSLContext sc = SSLContext.getInstance("TLSv1");
sc.init(null, null, new java.security.SecureRandom());
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
con.setSSLSocketFactory(sc.getSocketFactory());
System.out.println("After url connection");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
con.connect();
BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream(), Charset.forName("UTF-8")));
String input;
while ((input = br.readLine()) != null) {
System.out.println(input);
}
br.close();
And I get Error 403 forbidden when I connect to a ii's web service. How can I get Internet explorer subcode error (es. 403.1 execute access forbidden, 403.2 Read access forbidden)?
I want to delete the API from url : "http://dublr024vm.devlab.ibm.com:60633/B2BiAPIs/svc/cadigitalcertificates/_id:tibco_ssl" , I am getting java.io.IOException: Server returned HTTP response code: 403 for URL. I have passed the correct authentication details too.
Below is the part of my code:
public static void call_me() throws Exception {
InputStream is = null;
String url = "http://dublr024vm.devlab.ibm.com:60633/B2BiAPIs/svc/cadigitalcertificates/_id:tibco_ssl";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.29 Safari/537.36");
con.setDoOutput(true);
System.setProperty("http.agent", "Chrome");
// optional default is DELETE
con.setRequestMethod("DELETE");
//add request header
con.setRequestProperty("Content-Type", "application/json");
con.setRequestProperty("Accept", "application/json");
byte[] message = "admin:password".getBytes("UTF-8");
String encoding = DatatypeConverter.printBase64Binary(message);
con.setRequestProperty("Authorization", "Basic "+new String(encoding));
con.connect();
is = obj.openConnection().getInputStream();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'DELETE' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader br = new BufferedReader(new InputStreamReader(is));
String line, responseText = "";
while ((line = br.readLine()) != null) {
System.out.println(line);
responseText += line;
}
br.close();
con.disconnect();
}
I an new to java and I need help. I am trying to login to a website using java, and everything seems to be fine until now. I get the response back and everything, but It doesn't attempt to log in, which is kind of weird..
The response I get when I run the code is:
Sending 'POST' request to URL : http://mrpropop.com/login
Post parameters : login=admin&password=admin
Response Code : 200
+website html/css code
Here is my code:
package practice;
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.CookieHandler;
import java.net.CookieManager;
import java.net.HttpURLConnection;
import java.net.URL;
public class login {
private HttpURLConnection conn;
public login() {
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
}
public String sendPost(String url, String params) throws Exception {
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
// Acts like a browser
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "mrpropop.com");
conn.setRequestProperty("User-Agent",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36");
conn.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("Content-Length", Integer.toString(params.length()));
conn.setDoOutput(true);
conn.setDoInput(true);
// Send post request
DataOutputStream wr = new DataOutputStream(conn.getOutputStream());
wr.writeBytes(params);
wr.flush();
wr.close();
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + params);
System.out.println("Response Code : " + responseCode);
return getResponse();
}
private String getResponse() throws IOException {
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line = br.readLine();
StringBuilder sb = new StringBuilder();
while (line != null) {
sb.append(line);
line = br.readLine();
}
return sb.toString();
}
public static void main(String[] argv) throws Exception {
login login = new login();
// 1. login first
String loginUrl = "http://mrpropop.com/login";
String loginParams = "login=admin&password=admin";
login.sendPost(loginUrl, loginParams);
// Post request
String apiUrl = loginUrl;
String apiParams = loginParams;
System.out.println(login.sendPost(apiUrl, apiParams));
}
}
What is wrong here? Thanks in advance!
I suppose you want to do something when you get a code 200.
For example setting a cookie.
The response code of 200 suggests you have successfully logged in. You can display the contents by querying the HTTP body from the response object.
I am trying to scrape this link but failing to do so as it takes a soap request and return results as HTML via soap response.
Here's my Code that is not working properly.
Connection.Response response = Jsoup.connect("http://www.itatonline.in:8080/itat/jsp/runBirt2.jsp?subAction=showReoprt&__report=CaseDetails1_DELHI.rptdesign&searchWhat=searchByAssName&Serial%20No=&Appeal%20No=&Assessee%20Name=&AssType=DontKnow&appealDate=&Bench=AGR").method(Method.GET).timeout(30000).execute();
String url = "http://www.itatonline.in:8080/itat/jsp/runBirt2.jsp?subAction=showReoprt&__report=CaseDetails1_DELHI.rptdesign&searchWhat=searchByAssName&Serial%20No=&Appeal%20No=&Assessee%20Name=k&AssType=DontKnow&appealDate=&Bench=AGR&__sessionId=20151210_231624_579";
String rawData = "<soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/>"
+ "<soap:Body><GetUpdatedObjects xmlns=\"http://schemas.eclipse.org/birt\">"
+ "<Operation><Target><Id>Document</Id><Type>Document</Type></Target><Operator>GetPage</Operator><Oprand><Name>Appeal No</Name><Value></Value></Oprand><Oprand><Name>__isdisplay__AppealNo</Name><Value></Value></Oprand><Oprand><Name>Serial No</Name><Value></Value></Oprand><Oprand><Name>__isdisplay__Serial No</Name><Value></Value></Oprand><Oprand><Name>Assessee Name</Name><Value>k</Value></Oprand><Oprand><Name>__isdisplay__Assessee Name</Name><Value>k</Value></Oprand><Oprand><Name>searchWhat</Name><Value>searchByAssName</Value></Oprand><Oprand><Name>__isdisplay__searchWhat</Name><Value>searchByAssName</Value></Oprand><Oprand><Name>AssType</Name><Value>DontKnow</Value></Oprand><Oprand><Name>__isdisplay__AssType</Name><Value>DontKnow</Value></Oprand><Oprand><Name>appealDate</Name><Value></Value></Oprand><Oprand><Name>__isdisplay__appealDate</Name><Value></Value></Oprand><Oprand><Name>Bench</Name><Value>AGR</Value></Oprand><Oprand><Name>__isdisplay__Bench</Name><Value>AGR</Value></Oprand><Oprand><Name>__page</Name><Value>1</Value></Oprand><Oprand><Name>__svg</Name><Value>true</Value></Oprand><Oprand><Name>__page</Name><Value>1</Value></Oprand><Oprand><Name>__taskid</Name><Value>2015-11-9-23-16-22-34</Value></Oprand></Operation></GetUpdatedObjects></soap:Body></soap:Envelope>";
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "text/xml; charset=UTF-8");
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
con.setRequestProperty("Cookie", response.cookies().toString().replaceAll("[{}]", ""));
con.setRequestProperty("Host", "www.itatonline.in:8080");
con.setRequestProperty("Referer", "http://www.itatonline.in:8080/itat/jsp/runBirt2.jsp?subAction=showReoprt&__report=CaseDetails1_DELHI.rptdesign&searchWhat=searchByAssName&Serial%20No=&Appeal%20No=&Assessee%20Name=k&AssType=DontKnow&appealDate=&Bench=AGR");
con.setRequestProperty("SOAPAction", "");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.3; rv:42.0) Gecko/20100101 Firefox/42.0");
con.setRequestProperty("X-Prototype-Version", "1.4.0");
con.setRequestProperty("X-Requested-With", "XMLHttpRequest");
con.setRequestProperty("request-type", "SOAP");
// Send post request
con.setDoOutput(true);
OutputStreamWriter w = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
w.write(rawData);
w.close();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer resp = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
resp.append(inputLine);
}
in.close();
System.out.println(resp);
`
I am making a facebook software and for it i need to make a fql query to get friendlist. I use the below code
String url = "https://graph.facebook.com/fql?q=SELECT uid, name, pic_square FROM user WHERE uid = me() OR uid IN (SELECT uid2 FROM friend WHERE uid1 = me())&access_token="+access;
url = url.replace(" ", "%20");
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("POST");
//add request header
String USER_AGENT = "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36";
con.setRequestProperty("User-Agent", USER_AGENT);
int responseCode = con.getResponseCode();
System.out.println(responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
But i always seem to get a 400 error.I have tried replace spaces by %20 but to no effect.
When i open the link in the browser, it opens good.
I feel ashamed.The solution is really the POST line which should be GET.
String url = "https://graph.facebook.com/fql?q=SELECT uid, name, pic_square FROM user WHERE uid = me() OR uid IN (SELECT uid2 FROM friend WHERE uid1 = me())&access_token="+access;
url = url.replace(" ", "%20");
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// optional default is GET
con.setRequestMethod("GET");
//add request header
String USER_AGENT = "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36";
con.setRequestProperty("User-Agent", USER_AGENT);
int responseCode = con.getResponseCode();
System.out.println(responseCode);
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();