Develop Reference

Check if url exists JAVA

I'm trying to check if the url entered by the user actually exists.
Below is what I have tried.
public static Boolean checkURLExists(String urlName)
{
Boolean urlCheck=false;
try{
URL url = new URL(urlName);
HttpURLConnection.setFollowRedirects(false);
HttpURLConnection huc = (HttpURLConnection) url.openConnection();
huc.setRequestMethod("GET");
int responseCode = huc.getResponseCode();
String responseMessage = huc.getResponseMessage();
char a=String.valueOf(Math.abs((long)huc.getResponseCode())).charAt(0);
if ((a == '2' || a == '3')&& (responseMessage.equalsIgnoreCase("ok")||responseMessage.equalsIgnoreCase("found")||responseMessage.equalsIgnoreCase("redirect"))) {
System.out.println("GOOD "+responseCode+" - "+a);
urlCheck=true;
} else {
System.out.println("BAD "+responseCode+" - "+a);
}
}catch(Exception e){
e.printStackTrace();
}
return urlCheck;
}
The issue with the above code is that it returns http://www.gmail.com or http://www.yahoo.co.in etc. as invalid URLs with response code 301 & response message "Moved permanently" but they actually redirects to other url, Is there any way to detect that the url when entered in browser will open a page?
Thank you.

Well the normal behavior of a web browser when it sees a 301 response is to follow the redirect. But you seem to have told your test code NOT to do that. If you want your code to behave (more) like a browser would, change this
HttpURLConnection.setFollowRedirects(false);
to this
HttpURLConnection.setFollowRedirects(true);

How do I authenticate with facebook serverside without using the JS library

At my site a user is allowed to sign in with facebook. When doing that I ask for permission
to post to the users feed. This works like a charm.
When signed in, a user is allowed to write a review and when saving the review the user is asked if the user wants to post the review to the users feed on facebook. Since the post to facebook should be done after the review is saved in my local db, I understand that I need to perform an authentication serverside and then when I have a token I'm able to POST to eg.
http://graph.facebook.com/10XXXX40308/feed
with
message : "This works"
I have been trying to implement the facebook web login as described here:
The steps are:
Perform a request against
https://graph.facebook.com/oauth/authorize?client_id=MY_API_KEY&
redirect_uri=http://www.facebook.com/connect/login_success.html&
scope=publish_stream
Facebook will redirect you to
http://www.facebook.com/connect/login_success.html?
code=MY_VERIFICATION_CODE
Request
https://graph.facebook.com/oauth/access_token?client_id=MY_API_KEY&
redirect_uri=http://www.facebook.com/connect/login_success.html&
client_secret=MY_APP_SECRET&code=MY_VERIFICATION_CODE Facebook will
respond with access_token=MY_ACCESS_TOKEN
When doing 1. in a browser the application behaves accordingly. I get a redirect back from facebook with the MY_VERIFICATION_CODE:
So I try to do it in code like this:
String url = "https://graph.facebook.com/oauth/authorize?client_id="+clientId+"&scope=publish_stream&redirect_uri=http://www.facebook.com/connect/login_success.html";
URL obj = new URL(url);
conn = (HttpURLConnection) obj.openConnection();
conn.setReadTimeout(5000);
conn.setRequestMethod("GET");
System.out.println("Request URL ... " + url);
boolean redirect = false;
// normally, 3xx is redirect
int status = conn.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
System.out.println("Response Code ... " + status);
if (redirect) {
// get redirect url from "location" header field
String newUrl = conn.getHeaderField("Location");
// get the cookie if need, for login
String cookies = conn.getHeaderField("Set-Cookie");
// open the new connnection again
conn = (HttpURLConnection) new URL(newUrl).openConnection();
conn.setRequestProperty("Cookie", cookies);
System.out.println("Redirect to URL : " + newUrl);
}
BufferedReader in = new BufferedReader(
new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer html = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
html.append(inputLine);
}
in.close();
System.out.println("URL Content... \n" + html.toString());
System.out.println("Done");
But what happens is that instead of getting the 302 back I get a 200 back and the login
page in code:
It seems that I have missed a step or do not understand the flow.
What I'm trying to accomplish is to implement a similar call like to janrain's:
https://rpxnow.com/api/v2/facebook/stream.publish
where you are allowed to do this.
Thank you!

I guess rtfm is in place here. The user need to authenticate, so what I'm really trying to do here is to bypass the authentication process. This is of course not allowed. So
how do you solve this?
When the user authenticates I need to save the access token and the expire time so that
I can add that to the request later on.
I think that is the only way...Correct me if i'm wrong.
So in the authentication process i create a regexp:
Pattern pattern = Pattern.compile("access_token=([a-zA-Z0-9]+)&expires=([0-9]+)");
Then in the callback from facebook I extract the token with the regexp:
String accessToken = "";
String expires = "";
Matcher matcher = pattern.matcher(token.trim());
if(matcher.matches()) {
accessToken = matcher.group(1);
expires = matcher.group(2);
} else {
return new JSONObject()
.put("error", "OathBean: accessToken is null");
}
I then call facebook to get the values for the user and return all the values so that I can work with them:
return new JSONObject()
.put("facebookId", facebookId)
.put("firstName", firstName)
.put("lastName", lastName)
.put("email", email)
.put("photo", photo)
.put("accessToken", accessToken)
.put("expires", expires);
Later on when the user wants post a review to facebook. I populate the request and post the review.
Map<String, String> requestData = new HashMap<String, String>();
requestData.put("link",url(newReview));
requestData.put("description", "reviewText");
requestData.put("access_token", credential.getPassword());
String query = createQuery(requestData);
JSONObject result = null;
try {
URL url = new URL("https://graph.facebook.com/"+identifier+"/feed");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.connect();
OutputStreamWriter osw = new OutputStreamWriter(conn.getOutputStream(), "UTF-8");
osw.write(query);
osw.close();
result = new JSONObject(IOUtils.toString(conn.getInputStream()));
}
catch (Exception e) {
log.error("Could not call graph feed to publish for id: "+identifier, e);
}
if(result != null) {
boolean success = StringUtils.isNotBlank(result.getString("id"));
entityManager.persist(new FBPublishEvent(currentUser, newReview, success, result.toString()));
}
If you have a better solution, please share =)

Android 4.0 ICS turning HttpURLConnection GET requests into POST requests

My Galaxy Nexus arrived today, and one of the first things I did was to load my app onto it so I could demonstrate it to my friends. Part of its functionality involves importing RSS Feeds from Google Reader. However, upon trying this, I was getting 405 Method Not Allowed errors.
This problem is Ice Cream Sandwich-specific. The code I've attached works fine on Gingerbread and Honeycomb. I've traced the error down to the moment the connection is made, when the GET request magically turns into a POST request.
/**
* Get the authentication token from Google
* #param auth The Auth Key generated in getAuth()
* #return The authentication token
*/
private String getToken(String auth) {
final String tokenAddress = "https://www.google.com/reader/api/0/token";
String response = "";
URL tokenUrl;
try {
tokenUrl = new URL(tokenAddress);
HttpURLConnection connection = (HttpURLConnection) tokenUrl.openConnection();
connection.setRequestMethod("GET");
connection.addRequestProperty("Authorization", "GoogleLogin auth=" + auth);
connection.setRequestProperty("Content-Type","application/x-www-form-urlendcoded");
connection.setUseCaches(false);
connection.setDoOutput(true);
Log.d(TAG, "Initial method: " + connection.getRequestMethod()); // Still GET at this point
try {
connection.connect();
Log.d(TAG, "Connected. Method is: " + connection.getRequestMethod()); // Has now turned into POST, causing the 405 error
InputStream in = new BufferedInputStream(connection.getInputStream());
response = convertStreamToString(in);
connection.disconnect();
return response;
}
catch (Exception e) {
Log.d(TAG, "Something bad happened, response code was " + connection.getResponseCode()); // Error 405
Log.d(TAG, "Method was " + connection.getRequestMethod()); // POST again
Log.d(TAG, "Auth string was " + auth);
e.printStackTrace();
connection.disconnect();
return null;
}
}
catch(Exception e) {
// Stuff
Log.d(TAG, "Something bad happened.");
e.printStackTrace();
return null;
}
}
Is there anything that could be causing this problem? Could this function be better coded to avoid this problem?
Many thanks in advance.

This behaviour is described in Android Developers: HttpURLConnection
HttpURLConnection uses the GET method by default. It will use POST if
setDoOutput(true) has been called.
What's strange though is that this has not actually been the behaviour until 4.0, so I would imagine it's going to break many existing published apps.
There is more on this at Android 4.0 turns GET into POST.

Removing this line worked for me:
connection.setDoOutput(true);
4.0 thinks with this line it should definitely be POST.

Get rid of this:
connection.setRequestProperty("Content-Type","application/x-www-form-urlendcoded");
This tells the API this is a POST.
UPDATE on how it could be done via HttpClient:
String response = null;
HttpClient httpclient = null;
try {
HttpGet httpget = new HttpGet(yourUrl);
httpget.setHeader("Authorization", "GoogleLogin auth=" + auth);
httpclient = new DefaultHttpClient();
HttpResponse httpResponse = httpclient.execute(httpget);
final int statusCode = httpResponse.getStatusLine().getStatusCode();
if (statusCode != HttpStatus.SC_OK) {
throw new Exception("Got HTTP " + statusCode
+ " (" + httpResponse.getStatusLine().getReasonPhrase() + ')');
}
response = EntityUtils.toString(httpResponse.getEntity(), HTTP.UTF_8);
} catch (Exception e) {
e.printStackTrace();
// do some error processing here
} finally {
if (httpclient != null) {
httpclient.getConnectionManager().shutdown();
}
}

This is one that got me - basically by setting setDoOutput(true) it forces a POST request when you make the connection, even if you specify this is a GET in the setRequestMethod:
HttpURLConnection uses the GET method by default. It will use POST if
setDoOutput(true) has been called. Other HTTP methods (OPTIONS, HEAD,
PUT, DELETE and TRACE) can be used with setRequestMethod(String).
This caught me a while back - very frustrating ...
See http://developer.android.com/reference/java/net/HttpURLConnection.html and go to HTTP Methods heading

I've found that pre-ICS one could get away with making a body-less POST without providing a Content-Length value, however post-ICS you must set Content-Length: 0.

Java function to detect valid webpage

I am trying to write a Java program that will load pages pointed to by valid links and report other links as broken. My problem is that the Java URL will download the appropriate page if the url is valid, and the search-engine results for the url if the url is invalid.
Is there a Java function that detects if the url resolves to a legitimate page . . . thanks very much,
Joel

HttpURLConnection#getResponseCode will give you an HTTP status code

You can get the HTTP response code for a URL like so:
public static int getResponseCode(URL url) throws IOException {
URLConnection conn = url.openConnection();
if (!(conn instanceof HttpURLConnection)) {
throw new IllegalArgumentException("not an HTTP url: " + url);
}
HttpURLConnection httpConn = (HttpURLConnection) conn;
return httpConn.getResponseCode();
}
Now the question is, what do you consider a "valid" webpage? For me, if a URL parses correctly and it's protocol is "http" (or https) and it's response code is in the 200 block or 302 (Found/Redirect) or 304 (Not modified), then it's valid:
public boolean isValidHttpResponseCode(int code) {
return ((code / 100) == 2) || (code == 302) || (code == 304);
}

FileNotFoundException while getting the InputStream object from HttpURLConnection

I am trying to send a post request to a url using HttpURLConnection (for using cUrl in java).
The content of the request is xml and at the end point, the application processes the xml and stores a record to the database and then sends back a response in form of xml string. The app is hosted on apache-tomcat locally.
When I execute this code from the terminal, a row gets added to the db as expected. But an exception is thrown as follows while getting the InputStream from the connection
java.io.FileNotFoundException: http://localhost:8080/myapp/service/generate
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.kodeplay.helloworld.HttpCurl.main(HttpCurl.java:30)
Here is the code
public class HttpCurl {
public static void main(String [] args) {
HttpURLConnection con;
try {
con = (HttpURLConnection) new URL("http://localhost:8080/myapp/service/generate").openConnection();
con.setRequestMethod("POST");
con.setDoOutput(true);
con.setDoInput(true);
File xmlFile = new File("test.xml");
String xml = ReadWriteTextFile.getContents(xmlFile);
con.getOutputStream().write(xml.getBytes("UTF-8"));
InputStream response = con.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response));
for (String line ; (line = reader.readLine()) != null;) {
System.out.println(line);
}
reader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Its confusing because the exception is traced to the line InputStream response = con.getInputStream(); and there doesn't seem to be any file involved for a FileNotFoundException.
When I try to open a connection to an xml file directly, it doesn't throw this exception.
The service app uses spring framework and Jaxb2Marshaller to create the response xml.
The class ReadWriteTextFile is taken from here
Thanks.
Edit:
Well it saves the data in the DB and sends back a 404 response status code at the same time.
I also tried doing a curl using php and print out the CURLINFO_HTTP_CODE which turns out to be 200.
Any ideas on how do I go about debugging this ? Both service and client are on the local server.
Resolved:
I could solve the problem after referring to an answer on SO itself.
It seems HttpURLConnection always returns 404 response when connecting to a url with a non standard port.
Adding these lines solved it
con.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
con.setRequestProperty("Accept","*/*");

I don't know about your Spring/JAXB combination, but the average REST webservice won't return a response body on POST/PUT, just a response status. You'd like to determine it instead of the body.
Replace
InputStream response = con.getInputStream();
by
int status = con.getResponseCode();
All available status codes and their meaning are available in the HTTP spec, as linked before. The webservice itself should also come along with some documentation which overviews all status codes supported by the webservice and their special meaning, if any.
If the status starts with 4nn or 5nn, you'd like to use getErrorStream() instead to read the response body which may contain the error details.
InputStream error = con.getErrorStream();

FileNotFound is just an unfortunate exception used to indicate that the web server returned a 404.

To anyone with this problem in the future, the reason is because the status code was a 404 (or in my case was a 500). It appears the InpuStream function will throw an error when the status code is not 200.
In my case I control my own server and was returning a 500 status code to indicate an error occurred. Despite me also sending a body with a string message detailing the error, the inputstream threw an error regardless of the body being completely readable.
If you control your server I suppose this can be handled by sending yourself a 200 status code and then handling whatever the string error response was.

For anybody else stumbling over this, the same happened to me while trying to send a SOAP request header to a SOAP service. The issue was a wrong order in the code, I requested the input stream first before sending the XML body. In the code snipped below, the line InputStream in = conn.getInputStream(); came immediately after ByteArrayOutputStream out = new ByteArrayOutputStream(); which is the incorrect order of things.
ByteArrayOutputStream out = new ByteArrayOutputStream();
// send SOAP request as part of HTTP body
byte[] data = request.getHttpBody().getBytes("UTF-8");
conn.getOutputStream().write(data);
if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) {
Log.d(TAG, "http response code is " + conn.getResponseCode());
return null;
}
InputStream in = conn.getInputStream();
FileNotFound in this case was an unfortunate way to encode HTTP response code 400.

FileNotFound in this case means you got a 404 from your server - could it be that the server does not like "POST" requests?

FileNotFound in this case means you got a 404 from your server
You Have to Set the Request Content-Type Header Parameter
Set “content-type” request header to “application/json” to send the request content in JSON form.
This parameter has to be set to send the request body in JSON format.
Failing to do so, the server returns HTTP status code “400-bad request”.
con.setRequestProperty("Content-Type", "application/json; utf-8");
Full Script ->
public class SendDeviceDetails extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... params) {
String data = "";
String url = "";
HttpURLConnection con = null;
try {
// From the above URL object,
// we can invoke the openConnection method to get the HttpURLConnection object.
// We can't instantiate HttpURLConnection directly, as it's an abstract class:
con = (HttpURLConnection)new URL(url).openConnection();
//To send a POST request, we'll have to set the request method property to POST:
con.setRequestMethod("POST");
// Set the Request Content-Type Header Parameter
// Set “content-type” request header to “application/json” to send the request content in JSON form.
// This parameter has to be set to send the request body in JSON format.
//Failing to do so, the server returns HTTP status code “400-bad request”.
con.setRequestProperty("Content-Type", "application/json; utf-8");
//Set Response Format Type
//Set the “Accept” request header to “application/json” to read the response in the desired format:
con.setRequestProperty("Accept", "application/json");
//To send request content, let's enable the URLConnection object's doOutput property to true.
//Otherwise, we'll not be able to write content to the connection output stream:
con.setDoOutput(true);
//JSON String need to be constructed for the specific resource.
//We may construct complex JSON using any third-party JSON libraries such as jackson or org.json
String jsonInputString = params[0];
try(OutputStream os = con.getOutputStream()){
byte[] input = jsonInputString.getBytes("utf-8");
os.write(input, 0, input.length);
}
int code = con.getResponseCode();
System.out.println(code);
//Get the input stream to read the response content.
// Remember to use try-with-resources to close the response stream automatically.
try(BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream(), "utf-8"))){
StringBuilder response = new StringBuilder();
String responseLine = null;
while ((responseLine = br.readLine()) != null) {
response.append(responseLine.trim());
}
System.out.println(response.toString());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (con != null) {
con.disconnect();
}
}
return data;
}
#Override
protected void onPostExecute(String result) {
super.onPostExecute(result);
Log.e("TAG", result); // this is expecting a response code to be sent from your server upon receiving the POST data
}
and call it
new SendDeviceDetails().execute("");
you can find more details in this tutorial
https://www.baeldung.com/httpurlconnection-post

The solution:
just change localhost for the IP of your PC
if you want to know this: Windows+r > cmd > ipconfig
example: http://192.168.0.107/directory/service/program.php?action=sendSomething
just replace 192.168.0.107 for your own IP (don't try 127.0.0.1 because it's same as localhost)

Please change
con = (HttpURLConnection) new URL("http://localhost:8080/myapp/service/generate").openConnection();
To
con = (HttpURLConnection) new URL("http://YOUR_IP:8080/myapp/service/generate").openConnection();