I'm developing an application which needs to be able to login to a website on a user's behalf and do some html scraping. Like many other developers, app engine is giving me trouble when it comes to cookie management. The server I'm logging in to sends a redirect after the initial POST, which then sends another redirect to the final landing page. As far as I can tell the purpose is for the server to verify cookies are working. I've stitched together the following helper class from other answers on SO.
public class Utilities {
public static String smartPost(String url, String data) throws IOException {
// storage for cookies between redirects
Map<String, String> cookies = new HashMap<String, String>();
HttpURLConnection connection;
StringBuilder response = new StringBuilder();
response.append(url);
URL resource = new URL(url);
connection = (HttpURLConnection) resource.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
connection.setRequestProperty("Content-Length",
"" + Integer.toString(data.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches(false);
connection.setInstanceFollowRedirects(false);
connection.setDoInput(true);
connection.setDoOutput(true);
// Send request
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(data);
wr.flush();
wr.close();
url = connection.getHeaderField("location");
while (url != null) {
// Get Cookies
getCookiesFromConnection(connection, cookies);
URL redirectResource = new URL(url);
response.append(url);
connection = (HttpURLConnection) redirectResource.openConnection();
connection.setRequestMethod("GET");
addCookiesToConnection(connection, cookies);
connection.setInstanceFollowRedirects(false);
connection.setUseCaches(false);
connection.setDoInput(true);
url = connection.getHeaderField("location");
connection.disconnect();
}
// Arrived at final location
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
response.append(line);
response.append('\r');
}
rd.close();
return response.toString();
}
static void addCookiesToConnection(HttpURLConnection c,
Map<String, String> storage) {
StringBuilder cookieStringBuilder = new StringBuilder();
for (Entry<String, String> e : storage.entrySet()) {
cookieStringBuilder.append(e.getKey());
cookieStringBuilder.append("=");
cookieStringBuilder.append(e.getValue());
cookieStringBuilder.append(";");
}
c.setRequestProperty("Cookies", cookieStringBuilder.toString());
}
static void getCookiesFromConnection(HttpURLConnection c,
Map<String, String> storage) {
Map<String, List<String>> headers = c.getHeaderFields();
for (Entry<String, List<String>> e : headers.entrySet()) {
if (e.getKey().equalsIgnoreCase("Set-Cookie")) {
for (String cookieHeader : e.getValue()) {
String cookie = cookieHeader.substring(0,
cookieHeader.indexOf(";"));
String key = cookie.substring(0, cookie.indexOf("="));
String value = cookie.substring(cookie.indexOf("=") + 1);
storage.put(key, value);
}
}
}
}
}
My goal was to handle the redirects manually and pass the cookies through to the final page. It works fine on the development server, but I don't think it's my code doing the work, but rather the default behavior on the local server. Anyone have experience implementing this kind of functionality on the production server? I'm pretty inexperienced with the java.net package, so I could be awfully far from a solution.
I originally tried implementing this in Go, but I had the same results and figured it was just my utter lack of experience with Go. Java will be easier for html scraping anyway because of Jsoup, but I'm not opposed to using python or go on this if that will make it easier somehow. It's a pretty small piece of a large project and I'm not too far in to switch.
After struggling with this for a few days, I found this article
Which does exactly what I was trying to do, in python. I've decided to use python for this project and I will user BeautifulSoup for the html scraping. Still unsure what was wrong with my code initially.
Related
I have an Azure webapp (App Service) running with Tomcat. I'd deployed 2 war applications. WAR-1 provides web service call which return a json files using Springboot. WAR-2 is a web application which call this web services in WAR-1. This webapp has system assigned managed identity (or MSI). In addition, this webapp has authentication on with AAD, using Express configuration.
I can access static pages in WAR-2, after authentication through AAD. Now I need to fetch data from WAR-1. I have a servlet which contains code like this:
String subscriptionId = "xxxx";
String testURL = "https://yyy.azurewebsites.net/war1/person/100";
String resourceId = "https://management.azure.com/";
AppServiceMSICredentials credentials = new AppServiceMSICredentials(AzureEnvironment.AZURE);
Azure azure = Azure.configure()
.withLogLevel(LogLevel.BODY_AND_HEADERS)
.authenticate(credentials)
.withSubscription(subscriptionId);
String token = credentials.getToken(resourceId);
HttpURLConnection conn = (HttpURLConnection) new URL(testURL).openConnection();
conn.setDoOutput(true);
conn.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
conn.setRequestMethod("GET");
conn.setRequestProperty("Authorization", "Bearer " + token);
int responseCode = conn.getResponseCode();
OutputStream os = conn.getOutputStream();
....
I do able to get a token, but the response code is 500 when I make the GET call.
So my question is ... is this the correct way to do this call ? I did found an articlehttps://dotnetdevlife.wordpress.com/2018/10/22/call-azure-ad-protected-website-using-managed-service-identity-msi/ similar to this situation but it uses .Net. I cannot find any Java equivalent of this.
I tested at my side, and here are my steps:
1. Two apps in one Azure web app.
App1: https://jackdemoapp1.azurewebsites.net/app1/
App2: https://jackdemoapp1.azurewebsites.net/app2/
2. Configure Authentication/Authorization on Azure portal.
And you can get the client ID by clicking into the details, note it down and we will use it in app2:
3. Configure managed identity on Azure portal
To simplify the test, the app1 will just return a "Hello" string.
4. Code in app2
#ResponseBody
#RequestMapping("/")
public String index() {
JSONObject json = new JSONObject();
try {
AppServiceMSICredentials credential = new AppServiceMSICredentials(AzureEnvironment.AZURE);
// As we want to get token for accessing the aad-protected app, change the
// resource to the client ID you get in step 2
String token = credential.getToken("ac07d701-6f7d-462e-8b67-5dffa1df955f");
json.put("token", token);
// The URL for app1 API
String app1 = "https://jackdemoapp1.azurewebsites.net/app1/";
HttpURLConnection conn = (HttpURLConnection) new URL(app1).openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Authorization", "Bearer " + token);
conn.setDoOutput(true);
conn.setDoInput(true);
// Open the connection
conn.connect();
int code = conn.getResponseCode();
if (code >= 200 && code <= 300) {
try (InputStream inputStream = conn.getInputStream();
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader)) {
StringBuilder stringBuilder = new StringBuilder();
String line = "";
while ((line = bufferedReader.readLine()) != null) {
stringBuilder.append(line);
}
String response = stringBuilder.toString();
json.put("response", response);
}
} else {
json.put("Error", "Response Code" + conn.getResponseCode());
}
conn.disconnect();
} catch (Exception e) {
json.put("Exception", e.getStackTrace());
}
return json.toString();
}
Result
I'm trying to login to a portal. It works using Postman. When I try the same request using plain Java or OkHttp the login fails and I will be redirected to the login page.
HttpUrl.Builder httpBuilder = HttpUrl.parse("https://test58.cashctrl.com/auth/login.html").newBuilder();
httpBuilder.addQueryParameter("JMCF_AUTH_EMAIL", "email");
httpBuilder.addQueryParameter("JMCF_AUTH_PASSWORD", "password");
Request request = new Request.Builder()
.url(httpBuilder.build())
.get()
.build();
I know the Url looks weird but it works this way using Postman or even simply use a browser.
Alternative with plain Java, which I tried:
Map<String, String> parameters = new HashMap<>();
parameters.put(PARAM_EMAIL, EMAIL);
parameters.put(PARAM_PASSWORD, PASSWORD);
URL url = new URL(LOGIN_URL + "?" + ParameterStringBuilder.getParamsString(parameters));
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setInstanceFollowRedirects(true);
con.setDoOutput(true);
DataOutputStream out = new DataOutputStream(con.getOutputStream());
out.writeBytes(ParameterStringBuilder.getParamsString(parameters));
out.flush();
out.close();
int status = con.getResponseCode();
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer content = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
content.append(inputLine + "\n");
}
in.close();
con.disconnect();
System.out.println(status);
System.out.println(content.toString());
Postman must be doing something special or also a browser which I don't see.
I had the same issue, I got to know that Postman has "code" feature. Below the send button you can see the code option it will generate the code for you. There is a list of language to choose from and java is one of them. Do check that out. Also you must be missing the cookie, see the temporary headers in Postman add all in your code and do include the cookie one.
Thanks I hope it helps.
So I have a problem where if I type this link on the browser and hit enter, an activation happens. I just want to do the same through Java. I don't need any kind of response from the URL. It should just do the same as entering the URL on a browser. Currently my code doesn't throw an error, but I don't think its working because the activation is not happening. My code:
public static void enableMachine(String dns){
try {
String req= "http://"+dns+"/username?username=sputtasw";
URL url = new URL(req);
URLConnection connection = url.openConnection();
connection.connect();
/*BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));
String strTemp = "";
while (null != (strTemp = br.readLine())) {
System.out.println(strTemp);
}*/
} catch (Exception ex) {
ex.printStackTrace();
}
}
What's the problem?
If you want to do that with an URLConnection, it isn't sufficient to just open the connection with connect, you also have to send e.g. an HTTP request etc.
That said, i think it would be easier, if you use an HTTP client like the one from Apache HttpComponents (http://hc.apache.org/). Just do a GET request with the HTTP client, this would be the same as visiting the page with a browser (those clients usually also supports redirection etc.).
You may use HttpUrlConnectionClass to do the job:
URL url = new URL("http://my.url.com");
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
httpCon.setRequestProperty("Content-Type", "application/json");
httpCon.setDoOutput(true);
httpCon.setRequestMethod("POST");
String params = "foo=42&bar=buzz";
DataOutputStream wr = new DataOutputStream(httpCon.getOutputStream());
wr.writeBytes(params);
wr.flush();
wr.close();
httpCon.connect();
int responseCode = httpCon.getResponseCode();
You may as well use "GET" request method and just append parameters to the url.
I have java related question...
Website www.stationv3.com gets updated daily (most of the time at least, it's kinda irregular). Every time I connect to a site using address www.stationv3.com (using a browser), it redirects me to it's subpage www.stationv3.com/date_of_latest_update.html
I'm trying to make a program that will pull latest comic from the site, but I am not sure how to find out it's exact address. But I know I'd be able to find out if I could somehow find out where where am I being redirected on every connect. Is that possible with java? I know it can do all sorts of quirky things, but I'm still new to internet related stuff...
I used exact site name just to make it easy for you to check outwhat's going on...
And also, I'm creating a generic code, one which could (with some tinkering) be applyed to any site that functions in that manner.
import java.net.*;
public class ShowStationV3Redirect {
public static void main(String[] args) throws Exception {
URL url = new URL(args[0]);
HttpURLConnection.setFollowRedirects(false);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
System.out.println("Response code = " + connection.getResponseCode());
String header = connection.getHeaderField("location");
if (header != null)
System.out.println("www.stationv3.com redirected to " + header);
}
}
The above code snippet tells you what URL you are being redirected to.
I think you could just fecth:
http://www.stationv3.com/comics/{yyyy}{mm}{dd}sv3.gif
and forget about the redirection problem. You can use this code (not tested indeed):
URL server = new URL("<put here the image URL>");
HttpURLConnection connection = (HttpURLConnection)server.openConnection();
connection.setRequestMethod("GET");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.addRequestProperty("Accept","image/gif");
connection.addRequestProperty("Accept-Encoding", "gzip, deflate");
connection.connect();
InputStream is = connection.getInputStream();
OutputStream os = new FileOutputStream("c:/mycomic.gif");
byte[] buffer = new byte[1024];
int byteReaded = is.read(buffer);
while(byteReaded != -1)
{
os.write(buffer,0,byteReaded);
byteReaded = is.read(buffer);
}
os.close();
I'm a bit new to Java and more to connections stuff with it. I'm trying to create a program to connect to a website ("www.buybackprofesional.com") where I would like to download pictures and get some text from cars (after the login you have to enter a plate number to access a car's file).
This is what I have right now, but it always says that the session has expired, I need a way to login using the username and password of the mainpage, am I right? can someone give me some advice? Thanks
Note: I want to do it in Java, maybe I was not clear in the question.
//URL web = new URL("http://www.buybackprofesional.com/DetallePeri.asp?mat=9073FCV&fec=27/07/2010&tipo=C&modelo=4582&Foto=0");
URL web = new URL("http://www.buybackprofesional.com/");
HttpURLConnection con = (HttpURLConnection) web.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; JVM)");
con.setRequestProperty("Pragma", "no-cache");
con.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
A colleage helped me with this so I'll post the code that works:
public static URLConnection login(String _url, String _username, String _password) throws IOException, MalformedURLException {
String data = URLEncoder.encode("Usuario", "UTF-8") + "=" + URLEncoder.encode(_username, "UTF-8");
data += "&" + URLEncoder.encode("Contrase", "UTF-8") + "=" + URLEncoder.encode(_password, "UTF-8");
// Send data
URL url = new URL(_url);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
wr.close();
return conn;
}
This will submit the form info on the page I need and after that, using cookies I can stay connected!
To connect to a website using java consider using httpunit or httpcore (offered by apache). They handle sessions much better then you (or I) could do on your own.
Edit: Fixed the location of the link. Thanks for the correction!