I try to access an ASPX-website where subsequent pages are returned based on
post data. Unfortunately all my attempts to get the following pages fail.
Hopefully, someone here has an idea where to find the error!
In step one I read the session ID from the cookie as well as the value of the
viewstate variable in the returned html page. Step two intends to send it
back to the server to get the desired page.
Sniffing the data in the webbrowser gives
Host=www.geocaching.com
User-Agent=Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100618
Iceweasel/3.5.9 (like Firefox/3.5.9)
Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language=en-us,en;q=0.5
Accept-Encoding=gzip,deflate
Accept-Charset=ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive=300
Connection=keep-alive
Referer=http://www.geocaching.com/seek/nearest.aspx?state_id=149
Cookie=Send2GPS=garmin; BMItemsPerPage=200; maprefreshlock=true; ASP.
NET_SessionId=c4jgygfvu1e4ft55dqjapj45
Content-Type=application/x-www-form-urlencoded
Content-Length=4099
POSTDATA=__EVENTTARGET=ctl00%24ContentBody%24pgrBottom%
24lbGoToPage_3&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPD[...]2Xg%3D%
3D&language=on&logcount=on&gpx=on
Currently, my script looks like this
import java.net.*;
import java.io.*;
import java.util.*;
import java.security.*;
import java.net.*;
public class test1 {
public static void main(String args[]) {
// String loginWebsite="http://geocaching.com/login/default.aspx";
final String loginWebsite = "http://www.geocaching.com/seek/nearest.aspx?state_id=159";
final String POST_CONTENT_TYPE = "application/x-www-form-urlencoded";
// step 1: get session ID from cookie
String sessionId = "";
String viewstate = "";
try {
URL url = new URL(loginWebsite);
String key = "";
URLConnection urlConnection = url.openConnection();
if (urlConnection != null) {
for (int i = 1; (key = urlConnection.getHeaderFieldKey(i)) != null; i++) {
// get ASP.NET_SessionId from cookie
// System.out.println(urlConnection.getHeaderField(key));
if (key.equalsIgnoreCase("set-cookie")) {
sessionId = urlConnection.getHeaderField(key);
sessionId = sessionId.substring(0, sessionId.indexOf(";"));
}
}
BufferedReader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
// get the viewstate parameter
String aLine;
while ((aLine = in.readLine()) != null) {
// System.out.println(aLine);
if (aLine.lastIndexOf("id=\"__VIEWSTATE\"") > 0) {
viewstate = aLine.substring(aLine.lastIndexOf("value=\"") + 7, aLine.lastIndexOf("\" "));
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(sessionId);
System.out.println("\n");
System.out.println(viewstate);
System.out.println("\n");
// String goToPage="3";
// step2: post data to site
StringBuilder htmlResult = new StringBuilder();
try {
String encoded = "__EVENTTARGET=ctl00$ContentBody$pgrBottom$lbGoToPage_3" + "&" + "__EVENTARGUMENT=" + "&"
+ "__VIEWSTATE=" + viewstate;
URL url = new URL(loginWebsite);
URLConnection urlConnection = url.openConnection();
urlConnection = url.openConnection();
// Specifying that we intend to use this connection for input
urlConnection.setDoInput(true);
// Specifying that we intend to use this connection for output
urlConnection.setDoOutput(true);
// Specifying the content type of our post
urlConnection.setRequestProperty("Content-Type", POST_CONTENT_TYPE);
// urlConnection.setRequestMethod("POST");
urlConnection.setRequestProperty("Cookie", sessionId);
urlConnection.setRequestProperty("Content-Type", "text/html");
DataOutputStream out = new DataOutputStream(urlConnection.getOutputStream());
out.writeBytes(encoded);
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String aLine;
while ((aLine = in.readLine()) != null) {
System.out.println(aLine);
}
} catch (MalformedURLException e) {
// Print out the exception that occurred
System.err.println("Invalid URL " + e.getMessage());
} catch (IOException e) {
// Print out the exception that occurred
System.err.println("Unable to execute " + e.getMessage());
}
}
}
Any idea what's wrong? Any help is very appreciated!
Update
Thank you for the fast reply!
I switched to use the HttpURLConnection instead of the URLConnection which implements the setRequestMethod(). I also corrected the minor mistakes you mentioned, e.g. removed the obsolete first setRequestProperty call.
Unfortunately this doesn’t change anything... I think I set all relevant parameters but still get the first page of the list, only. It seems that the "__EVENTTARGET=ctl00$ContentBody$pgrBottom$lbGoToPage_3" is ignored. I don't have any clues why.
Internally, the form on the website looks like this:
<form name="aspnetForm" method="post" action="nearest.aspx?state_id=159" id="aspnetForm">
It is called by the following javascript:
<script type="text/javascript">
//<![CDATA[
var theForm = document.forms['aspnetForm'];
if (!theForm) {
theForm = document.aspnetForm;
}
function __doPostBack(eventTarget, eventArgument) {
if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
theForm.__EVENTTARGET.value = eventTarget;
theForm.__EVENTARGUMENT.value = eventArgument;
theForm.submit();
}
}
//]]>
</script>
Hopefully, this helps to find a solution?
Greetings
maik.
Do you actually want to GET or POST? If you want to POST, then you may need the setRequestMethd() line.
You're setting Content-Type twice -- I think you may need to combine these into one line.
Then, don't close the output stream before you try and read from the input stream.
Other than that, is there any more logging you can put in/clues you can give as to what way it's going wrong in?
Hey use following code
String userAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0";
org.jsoup.nodes.Document jsoupDoc = Jsoup.connect(url).timeout(15000).userAgent(userAgent).referrer("http://calendar.legis.ga.gov/Calendar/?chamber=House").ignoreContentType(true)
.data("__EVENTTARGET", eventtarget).data("__EVENTARGUMENT", eventarg).data("__VIEWSTATE", viewState).data("__VIEWSTATEGENERATOR", viewStateGenarator)
.data("__EVENTVALIDATION", viewStateValidation).parser(Parser.xmlParser()).post();
Related
I'm having issues trying to send over a JSON string to a REST API. Long story short, I'm taking user input in a form, sending it over to a java servlet to validate and work with it a bit, and then trying to send it to an endpoint.
I have the following method being called on in my doPost method in my servlet, I am using printwriter pw to be able to read back data being returned in my response in the browser console at this point.
String jsonData = //JSON STRING HERE\\
String username = //USERNAME\\
String password = //PASSWORD\\
String endpointURL = //ENDPOINT URL HERE\\
pw.println(sendJson(jsonData, username, password));
private String sendJSON(String jsonData, String usrname, String usrpass) {
try {
String auth = usrname + ":" + usrpass;
byte[] encodedAuth = Base64.encodeBase64(auth.getBytes(StandardCharsets.UTF_8));
String authHeaderValue = "Basic " + new String(encodedAuth);
URL url = new URL(endpointURL);
HttpURLConnection http = (HttpURLConnection)url.openConnection();
http.setConnectTimeout(5000);
http.setReadTimeout(5000);
http.setRequestMethod("POST");
http.setRequestProperty("Content-Type", "application/json; utf-8");
http.setRequestProperty("Authorization", authHeaderValue);
http.setDoOutput(true);
//POST Json to URL using HttpURLConnection
//try(OutputStream os = http.getOutputStream()) {
OutputStream os = http.getOutputStream();
byte[] input = jsonData.getBytes("utf-8");
os.write(input, 0, input.length);
//}
/*String responseBody;
try(BufferedReader br = new BufferedReader(
new InputStreamReader(http.getInputStream(), "utf-8"))) {
StringBuilder response = new StringBuilder();
String responseLine = null;
while ((responseLine = br.readLine()) != null) {
response.append(responseLine.trim());
}
//System.out.println(response.toString());
responseBody = response.toString();
return responseBody;
}
return responseBody;*/
BufferedReader in = new BufferedReader(
new InputStreamReader(http.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
return response.toString();
} catch (IOException e) {
return e.toString();
}
}
}
I was having issues with the try's so I rewrote it to try and just get functionality right away first. Right now I'm receiving "java.io.IOException: Server Returned HTTP response code: 500 for URL: //URL HERE\"
Would anybody have any tips to point me in the right direction? I feel like I'm just missing like a small piece of the puzzle at this point, and I'm having a hard time finding any tutorials showing what it is that I'm trying to do. Thank you so much to anyone for any tips/pointers!
Made sure I was able to authenticate and that wasn't the issue by just connecting and returning:
int statusCode = http.getResponseCode();
String statusCodeString = Integer.toString(statusCode);
return statusCodeString;
This worked fine, received 403 response when setting wrong password/username and 400 response when I change to correct.
I attempted using HttpClient as well instead, but was having issues trying to get that to work at all. I also had an error earlier with week trying to do this with a certificate error, but after reimporting the cert to my cacerts file this was resolved (unrelated to this issue I believe).
I'm trying to open my university's website to read their menu. I've written a version that reads the menu given the link directly to the menu link, but I want to pull it back a little so I can read the menu from the website and not the direct link (in case the link ever changes).
Here is the URL I am opening:
https://nccudining.sodexomyway.com/dining-choices/index.html
Whenever I open the link to the website, this is the output that I get:
302
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
The URL it outputs is the mobile version of the website, but when I try to use that URL, it outputs nothing.
This is my code:
import java.io.*;
import java.net.*;
public class test
{
public static void main( String[] args )
{
URL url = null;
try
{
url = new URL("https://nccudining.sodexomyway.com/dining-choices/index.html");
HttpURLConnection test = (HttpURLConnection) url.openConnection();
test.setInstanceFollowRedirects(true);
test.connect();
System.out.println(test.getResponseCode());
} catch ( MalformedURLException e1 )
{
System.out.println("URL cannot be opened.");
return;
}
BufferedReader in = null;
try
{
in = new BufferedReader(new InputStreamReader(url.openStream()));
} catch ( IOException e )
{
System.out.println("Error");
}
String inputLine;
try
{
while ((inputLine = in.readLine()) != null)
{
System.out.println(inputLine);
}
} catch ( IOException e )
{
System.out.println("Error");
}
}
}
I apologize for all the try/catch loops. I don't want to just throw an IOException from the main from the get-go because I've heard that's bad practice. Anyway, this code just opens the URL, sets up a connection so I can make sure the URL actually exists, and try to read the HTML of it. It works on any other site I've tried it on, including google.
My question is why will my code not read the correct source code of the website? Is it something wrong with my code (I figured adding in the HttpsURLConnection and allowing redirects would work) or is it just the website, and is there anything I can do to bypass that aside from just opening the weekly menu's page?
Solution found! Thanks to #ShayHaned for the fixes. I added the following lines to the HttpURLConnection so I got a 200 response code rather than a 302:
test = (HttpURLConnection) url.openConnection();
test.setRequestMethod("GET");
test.setRequestProperty("User-Agent", "Mozilla/5.0");
test.setInstanceFollowRedirects(true);
Then I changed the InputStream from opening the stream from the URL to getting the input stream from the HttpURLConnection, as shown:
BufferedReader in = new BufferedReader(new InputStreamReader(test.getInputStream()));
That gave me the HTML I was looking for.
You are just missing the appropriate headers for http communication to work safely and securely. You can add a few Headers to make sure that you get the desired response
HttpURLConnection test = (HttpURLConnection) url.openConnection();
test.addRequestProperty( "User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko" );
test.addRequestProperty( "Accept" , "text/html,application/xhtml+xml,application/xml,image/png, image/svg+xml,;q=0.9,*/*;q=0.8");
test.addRequestProperty( "Accept-Charset" , "ISO-8859-1,utf-8;q=0.7,*;q=0.3");
test.addRequestProperty( "Accept-Language" , "en-US,en;q=0.8" );
test.addRequestProperty( "Connection" , "close" );
test.setRequestMethod("GET");
test.setInstanceFollowRedirects(true);
test.connect();
// Nopes DONT TRY THIS
//in = new BufferedReader(new InputStreamReader(url.openStream()));
in = new BufferedReader( new InputStreamReader( test.getInputStream() ) );
String htmlContent = "";
for( String inputLine = ""; ( inputLine = in.readLine() ) != null; )
htmlContent += inputLine;
System.out.println( htmlContent );
Instead of in = new BufferedReader(new InputStreamReader(url.openStream() ) ); , please try in = new BufferedReader(new InputStreamReader(test.getInputStream() ) ); , because it sounds pretty logical to open your InputStream from the actual HttpURLConnection object .
If you really want to understand the http header part try https://en.wikipedia.org/wiki/List_of_HTTP_header_fields for a detailed description of http headers and usage
I have researched extensively and cannot find a solution. I have been using the solutions provided to other users and it does not seem to work for me.
My java code:
public class Post {
public static void main(String[] args) {
String name = "Bobby";
String address = "123 Main St., Queens, NY";
String phone = "4445556666";
String data = "";
try {
// POST as urlencoded is basically key-value pairs
// create key=value&key=value.... pairs
data += "name=" + URLEncoder.encode(name, "UTF-8");
data += "&address=" +
URLEncoder.encode(address, "UTF-8");
data += "&phone=" +
URLEncoder.encode(phone, "UTF-8");
// convert string to byte array, as it should be sent
byte[] dataBytes = data.toString().getBytes("UTF-8");
// open a connection to the site
URL url = new URL("http://xx.xx.xx.xxx/yyy.php");
HttpURLConnection conn =
(HttpURLConnection) url.openConnection();
// tell the server this is POST & the format of the data.
conn.setDoOutput(true);
conn.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
conn.setRequestMethod("POST");
conn.setFixedLengthStreamingMode(dataBytes.length);
conn.getOutputStream().write(dataBytes);
conn.getInputStream();
// Print out the echo statements from the php script
BufferedReader in = new BufferedReader(
new InputStreamReader(url.openStream()));
String line;
while((line = in.readLine()) != null)
System.out.println(line);
in.close();
} catch(Exception e) {
e.printStackTrace();
}
}
}
and the php
<?php
echo $_POST["name"];
?>
The output I receive is an empty line. I tested to see if it was a php/server side issue by making an html form that sends data over to a similar script and prints the data on the screen and that worked. But, for the life of me, I cannot get this to work with a remote client.
I am using Ubuntu server and Apache.
Thank you in advance.
The problem is actually in what you read as output. You are doing two requests:
1)conn.getInputStream(); - sends POST request with desired body
2)BufferedReader in = new BufferedReader(
new InputStreamReader(url.openStream())); - sends empty GET request (!!)
Change it to:
// ...
conn.getOutputStream().write(dataBytes);
BufferedReader in = new BufferedReader(
new InputStreamReader(conn.getInputStream()));
and see result.
I have a simple http service with JSON payload I want to test with a Java test harness.
Initially I set up a client using Basic Auth which works fine; server certificate is in the trustStore and I'm supplying username/password in the code. I send the request, I get the correct response.
public static void main(String[] args) {
// TODO Auto-generated method stub
String xxxURL = new String("https://www.xxx.yyy/zzz/AdminServlet?data=");
String username = new String("username");
String password = new String("password");
String authString = new String (username+":"+password);
String apiList = new String("{\"apiVersion\":\"1.4\",\"method\":\"api.list\",\"params\":{}}"); // Create JSON string. A bit ugly
try
{
System.setProperty("javax.net.ssl.trustStore","C:\\workspace\\http_client_test\\security\\cacerts");
String jsonStr = apiList;
URL url = new URL(xxxURL + URLEncoder.encode(jsonStr, "UTF-8") );
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection(); // create connection object
String encoded = Base64.encodeBase64String(authString.getBytes());
httpConn.setRequestProperty("Authorization", "Basic "+encoded);
httpConn.setRequestMethod("GET");
httpConn.connect(); // open connection
BufferedReader in = new BufferedReader(new InputStreamReader(httpConn.getInputStream()));
String temp = null;
StringBuilder sb = new StringBuilder();
while((temp = in.readLine()) != null)
{
sb.append(temp).append(" ");
}
String result = sb.toString();
System.out.println("result = " + result);
}
catch(Exception e)
{
e.printStackTrace();
}
}
I want to do the same test using mutual authentication. I have set up the keystore and truststore on both server and client, and imported the necessary certificates on each.
My problem is I cannot find out how to tell the HttpURLConnection that I want mutual certificate authentication.
I tried :
public class Test3 {
public static void main(String[] args) {
// TODO Auto-generated method stub
String xxxURL = new String("https://www.xxx.yyy/zzz/AdminServlet?data=");
String apiList = new String("{\"apiVersion\":\"1.4\",\"method\":\"api.list\",\"params\":{}}"); // Create JSON string. A bit ugly
try
{
System.setProperty("javax.net.ssl.trustStore","C:\\workspace\\http_client_test\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStorePassword","changeit");
System.setProperty("javax.net.ssl.keyStore","C:\\workspace\\http_client_test\\security\\keystore.jks");
System.setProperty("javax.net.ssl.keyStorePassword","password");
String jsonStr = apiList;
URL url = new URL(netThingsURL + URLEncoder.encode(jsonStr, "UTF-8") );
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection(); // create connection object
String encoded = Base64.encodeBase64String(authString.getBytes());
httpConn.setRequestProperty("Authorization", "?????????"); // ????????
httpConn.setRequestMethod("GET");
httpConn.connect(); // open connection
BufferedReader in = new BufferedReader(new InputStreamReader(httpConn.getInputStream()));
String temp = null;
StringBuilder sb = new StringBuilder();
while((temp = in.readLine()) != null)
{
sb.append(temp).append(" ");
}
String result = sb.toString();
System.out.println("result = " + result);
}
catch(Exception e)
{
e.printStackTrace();
}
}
}
What should I have here : httpConn.setRequestProperty("Authorization", "?????????"); I realise I may need more than just this 1 line. I tried various resources to find appropriate values but drew a blank. I tried various intuitive arguments but get a '401' error or NoSuchAlgorithException.
Any help, code, links to resources is greatly appreciated.
My problem is I cannot find out how to tell the HttpURLConnection that I want mutual certificate authentication.
You can't. You have to configure the server to ask for the client certificate. All you can do at the client is specify where the client certificate is. You can't force it to be sent. Only the server can do that.
I'm trying to download a file from
http://aula.au.dk/main/document/document.php?action=download&id=%2F%D8velsesvejledning+2012.pdf
but it dosen't appear to be a pdf, when i try downloading it with this code
import java.io.*;
import java.net.*;
public class DownloadFile {
public static void download(String address, String localFileName) throws IOException {
URL url1 = new URL(address);
byte[] ba1 = new byte[1024];
int baLength;
FileOutputStream fos1 = new FileOutputStream(localFileName);
try {
// Contacting the URL
System.out.print("Connecting to " + url1.toString() + " ... ");
URLConnection urlConn = url1.openConnection();
// Checking whether the URL contains a PDF
if (!urlConn.getContentType().equalsIgnoreCase("application/pdf")) {
System.out.println("FAILED.\n[Sorry. This is not a PDF.]");
} else {
try {
// Read the PDF from the URL and save to a local file
InputStream is1 = url1.openStream();
while ((baLength = is1.read(ba1)) != -1) {
fos1.write(ba1, 0, baLength);
}
fos1.flush();
fos1.close();
is1.close();
} catch (ConnectException ce) {
System.out.println("FAILED.\n[" + ce.getMessage() + "]\n");
}
}
} catch (NullPointerException npe) {
System.out.println("FAILED.\n[" + npe.getMessage() + "]\n");
}
}
}
Can you help me out here?
http://aula.au.dk/main/document/document.php?action=download&id=%2F%D8velsesvejledning+2012.pdf is not a pdf. The website gives an error this is why the script doesn't work:
SQL error in file /data/htdocs/dokeos184/www/main/inc/tool_navigation_menu.inc.php at line 70
As Marti said, the root cause of the problem is the fact that the script fails. I tested your program on a working pdf link, it works just fine.
This wouldn't have helped you in this case, but HttpURLConnection is a specialized subclass of URLConnection that makes communications with an http server a lot easier - eg direct access to error codes, etc.
HttpURLConnection urlConn = (HttpURLConnection) url1.openConnection();
// check the responsecode for e.g. errors (4xx or 5xx)
int responseCode = urlConn.getResponseCode();
2 step process with 2 libraries.
// 1. Use Jsoup to get the response.
Response response= Jsoup.connect(location)
.ignoreContentType(true)
// more method calls like user agent, referer, timeout
.execute();
// 2. Use Apache Commons to write the file
FileUtils.writeByteArrayToFile(new File(path), response.bodyAsBytes());