Develop Reference

Simulate URL entering on java

So I have a problem where if I type this link on the browser and hit enter, an activation happens. I just want to do the same through Java. I don't need any kind of response from the URL. It should just do the same as entering the URL on a browser. Currently my code doesn't throw an error, but I don't think its working because the activation is not happening. My code:
public static void enableMachine(String dns){
try {
String req= "http://"+dns+"/username?username=sputtasw";
URL url = new URL(req);
URLConnection connection = url.openConnection();
connection.connect();
/*BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));
String strTemp = "";
while (null != (strTemp = br.readLine())) {
System.out.println(strTemp);
}*/
} catch (Exception ex) {
ex.printStackTrace();
}
}
What's the problem?

If you want to do that with an URLConnection, it isn't sufficient to just open the connection with connect, you also have to send e.g. an HTTP request etc.
That said, i think it would be easier, if you use an HTTP client like the one from Apache HttpComponents (http://hc.apache.org/). Just do a GET request with the HTTP client, this would be the same as visiting the page with a browser (those clients usually also supports redirection etc.).

You may use HttpUrlConnectionClass to do the job:
URL url = new URL("http://my.url.com");
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
httpCon.setRequestProperty("Content-Type", "application/json");
httpCon.setDoOutput(true);
httpCon.setRequestMethod("POST");
String params = "foo=42&bar=buzz";
DataOutputStream wr = new DataOutputStream(httpCon.getOutputStream());
wr.writeBytes(params);
wr.flush();
wr.close();
httpCon.connect();
int responseCode = httpCon.getResponseCode();
You may as well use "GET" request method and just append parameters to the url.

Trouble posting user credentials with app engine url fetch

I'm developing an application which needs to be able to login to a website on a user's behalf and do some html scraping. Like many other developers, app engine is giving me trouble when it comes to cookie management. The server I'm logging in to sends a redirect after the initial POST, which then sends another redirect to the final landing page. As far as I can tell the purpose is for the server to verify cookies are working. I've stitched together the following helper class from other answers on SO.
public class Utilities {
public static String smartPost(String url, String data) throws IOException {
// storage for cookies between redirects
Map<String, String> cookies = new HashMap<String, String>();
HttpURLConnection connection;
StringBuilder response = new StringBuilder();
response.append(url);
URL resource = new URL(url);
connection = (HttpURLConnection) resource.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
connection.setRequestProperty("Content-Length",
"" + Integer.toString(data.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches(false);
connection.setInstanceFollowRedirects(false);
connection.setDoInput(true);
connection.setDoOutput(true);
// Send request
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(data);
wr.flush();
wr.close();
url = connection.getHeaderField("location");
while (url != null) {
// Get Cookies
getCookiesFromConnection(connection, cookies);
URL redirectResource = new URL(url);
response.append(url);
connection = (HttpURLConnection) redirectResource.openConnection();
connection.setRequestMethod("GET");
addCookiesToConnection(connection, cookies);
connection.setInstanceFollowRedirects(false);
connection.setUseCaches(false);
connection.setDoInput(true);
url = connection.getHeaderField("location");
connection.disconnect();
}
// Arrived at final location
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
response.append(line);
response.append('\r');
}
rd.close();
return response.toString();
}
static void addCookiesToConnection(HttpURLConnection c,
Map<String, String> storage) {
StringBuilder cookieStringBuilder = new StringBuilder();
for (Entry<String, String> e : storage.entrySet()) {
cookieStringBuilder.append(e.getKey());
cookieStringBuilder.append("=");
cookieStringBuilder.append(e.getValue());
cookieStringBuilder.append(";");
}
c.setRequestProperty("Cookies", cookieStringBuilder.toString());
}
static void getCookiesFromConnection(HttpURLConnection c,
Map<String, String> storage) {
Map<String, List<String>> headers = c.getHeaderFields();
for (Entry<String, List<String>> e : headers.entrySet()) {
if (e.getKey().equalsIgnoreCase("Set-Cookie")) {
for (String cookieHeader : e.getValue()) {
String cookie = cookieHeader.substring(0,
cookieHeader.indexOf(";"));
String key = cookie.substring(0, cookie.indexOf("="));
String value = cookie.substring(cookie.indexOf("=") + 1);
storage.put(key, value);
}
}
}
}
}
My goal was to handle the redirects manually and pass the cookies through to the final page. It works fine on the development server, but I don't think it's my code doing the work, but rather the default behavior on the local server. Anyone have experience implementing this kind of functionality on the production server? I'm pretty inexperienced with the java.net package, so I could be awfully far from a solution.
I originally tried implementing this in Go, but I had the same results and figured it was just my utter lack of experience with Go. Java will be easier for html scraping anyway because of Jsoup, but I'm not opposed to using python or go on this if that will make it easier somehow. It's a pretty small piece of a large project and I'm not too far in to switch.

After struggling with this for a few days, I found this article
Which does exactly what I was trying to do, in python. I've decided to use python for this project and I will user BeautifulSoup for the html scraping. Still unsure what was wrong with my code initially.

java and website redirection detection

I have java related question...
Website www.stationv3.com gets updated daily (most of the time at least, it's kinda irregular). Every time I connect to a site using address www.stationv3.com (using a browser), it redirects me to it's subpage www.stationv3.com/date_of_latest_update.html
I'm trying to make a program that will pull latest comic from the site, but I am not sure how to find out it's exact address. But I know I'd be able to find out if I could somehow find out where where am I being redirected on every connect. Is that possible with java? I know it can do all sorts of quirky things, but I'm still new to internet related stuff...
I used exact site name just to make it easy for you to check outwhat's going on...
And also, I'm creating a generic code, one which could (with some tinkering) be applyed to any site that functions in that manner.

import java.net.*;
public class ShowStationV3Redirect {
public static void main(String[] args) throws Exception {
URL url = new URL(args[0]);
HttpURLConnection.setFollowRedirects(false);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
System.out.println("Response code = " + connection.getResponseCode());
String header = connection.getHeaderField("location");
if (header != null)
System.out.println("www.stationv3.com redirected to " + header);
}
}
The above code snippet tells you what URL you are being redirected to.

I think you could just fecth:
http://www.stationv3.com/comics/{yyyy}{mm}{dd}sv3.gif
and forget about the redirection problem. You can use this code (not tested indeed):
URL server = new URL("<put here the image URL>");
HttpURLConnection connection = (HttpURLConnection)server.openConnection();
connection.setRequestMethod("GET");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.addRequestProperty("Accept","image/gif");
connection.addRequestProperty("Accept-Encoding", "gzip, deflate");
connection.connect();
InputStream is = connection.getInputStream();
OutputStream os = new FileOutputStream("c:/mycomic.gif");
byte[] buffer = new byte[1024];
int byteReaded = is.read(buffer);
while(byteReaded != -1)
{
os.write(buffer,0,byteReaded);
byteReaded = is.read(buffer);
}
os.close();

Server returned HTTP response code: 400

I am trying to get an InputStream from a URL. The URL can be a opened from Firefox. It returns a json and I have installed an addon for viewing json in Firefox so I can view it there.
So I tried to get it from Java by:
URL url = new URL(urlString);
URLConnection urlConnection = url.openConnection();
BufferedReader reader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
But it is throwing an IOException in urlConnection.getInputStream().
I also tried:
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
InputStream inputStream = url.openStream();
But no luck.
Any information is appreciable. Thanks in advance.

Thank you everybody. This is a weird problem but at last I solved it.
The URL I am requesting is
http://api.themoviedb.org/2.1/Movie.search/en/json/api_key/a nightmare on elm street
Now browser replaces the spaces between "a nightmare on elm street" by "%20" internally and parses. That is why the requested server can response by that request. But From Java I didn't replaced that spaces by "%20", so it turns into Bad Request, source.
Now it is working.
BufferedReader reader = new BufferedReader(new InputStreamReader(((HttpURLConnection) (new URL(urlString)).openConnection()).getInputStream(), Charset.forName("UTF-8")));

I had a similar issue and my url was:
http://www.itmat.upenn.edu/assets/user-content/documents/ITMAT17. October 10 2017_.pdf
which obviously contained spaces.
These caused java.io.IOException Server returned HTTP response code: 400 in the following code:
java.net.URL url = new URL(urlString);
java.io.InputStream in = url.openStream();
If you copy the above url and paste in browser, you will realize that browser adds '%20' for the spaces. So I did it manually with the following code and the problem is solved.
if(urlString.contains(" "))
urlString = urlString.replace(" ", "%20");
Complete code/answer should be:
if(urlString.contains(" "))
urlString = urlString.replace(" ", "%20");
java.net.URL url = new URL(urlString);
java.io.InputStream in = url.openStream();

are you setting up the connection correctly? here's some code that illustrates how to do this. Note that I am being lazy about exception handling here, this is not production quality code.
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
public class URLFetcher {
public static void main(String[] args) throws Exception {
URL myURL = new URL("http://www.paulsanwald.com/");
HttpURLConnection connection = (HttpURLConnection) myURL.openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(true);
connection.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder results = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
results.append(line);
}
connection.disconnect();
System.out.println(results.toString());
}
}

encode the parameters in the URL as follows:
String messageText = URLEncoder.encode(messageText, "UTF-8");

I a encountered same error. In my case, it was because the sizjwt token in the header was larger than acceptable size by mule soft proxy. One option is to increase the size of acceptable header size in mule soft, or reduce the size of token by removing some of the permissions assigned to the user id

Connect to web that requires user/password

I'm a bit new to Java and more to connections stuff with it. I'm trying to create a program to connect to a website ("www.buybackprofesional.com") where I would like to download pictures and get some text from cars (after the login you have to enter a plate number to access a car's file).
This is what I have right now, but it always says that the session has expired, I need a way to login using the username and password of the mainpage, am I right? can someone give me some advice? Thanks
Note: I want to do it in Java, maybe I was not clear in the question.
//URL web = new URL("http://www.buybackprofesional.com/DetallePeri.asp?mat=9073FCV&fec=27/07/2010&tipo=C&modelo=4582&Foto=0");
URL web = new URL("http://www.buybackprofesional.com/");
HttpURLConnection con = (HttpURLConnection) web.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; JVM)");
con.setRequestProperty("Pragma", "no-cache");
con.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
A colleage helped me with this so I'll post the code that works:
public static URLConnection login(String _url, String _username, String _password) throws IOException, MalformedURLException {
String data = URLEncoder.encode("Usuario", "UTF-8") + "=" + URLEncoder.encode(_username, "UTF-8");
data += "&" + URLEncoder.encode("Contrase", "UTF-8") + "=" + URLEncoder.encode(_password, "UTF-8");
// Send data
URL url = new URL(_url);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
wr.close();
return conn;
}
This will submit the form info on the page I need and after that, using cookies I can stay connected!

To connect to a website using java consider using httpunit or httpcore (offered by apache). They handle sessions much better then you (or I) could do on your own.
Edit: Fixed the location of the link. Thanks for the correction!