Develop Reference

Manually caching Rest without checking headers for ETag and Last-Modified

I'm using Rest to make a call to a remote server that I have no access to. I want to cache received data permanently for offline use without checking Last-Modified or ETag in the headers.
I expected CachingMode.MANUAL mechanism to check if there's an offline content and if there's none, then go online to fetch the content, but it doesn't.
To circumvent this, I had to first use Rest with CachingMode.OFFLINE and if that returns 404, then make another call with CachingMode.SMART.
Shouldn't there be an option of (let's say CachingMode.OFFLINE_FIRST) that checks offline first and if no content then goes online with (CachingMode.SMART)?
Below is my current approach:
Response<Map> response = Rest.get(url)
.cacheMode(CachingMode.OFFLINE)
.queryParam("param", value)
.jsonContent().onErrorCodeJSON(e -> {
throw new RuntimeException(createErrorMessage(e));
}).onError(e -> {
if (e.getResponseCode() == 0 || e.getConnectionRequest().getResponseCode() == 404) {
is404 = true;
return;
}
throw new RuntimeException("Network error. Please check your connection and try again.");
}).timeout(6000).getAsJsonMap();
if (is404) {
is404 = false;
response = Rest.get(url)
.cacheMode(CachingMode.SMART)
.queryParam("param", value)
.jsonContent().onErrorCodeJSON(e -> {
throw new RuntimeException(createErrorMessage(e));
}).onError(e -> {
throw new RuntimeException("Network error. Please check your connection and try again.");
}).timeout(6000).getAsJsonMap();
}

This makes sense. Added support for this in this commit: https://github.com/codenameone/CodenameOne/commit/fd81d979507fb08ee1d595b94df5973b322766a3

Java Get URL Header Alone, Body Not Required

Im using below code to get the cache-control value in header of given URL. I dont want to get the body of the URL. Below request takes 800ms to process. Is there any alteration can be done in below code? Im using Google App Engine for development. Please suggest. Thanks. I like not to add extra jar.
URL obj;
URLConnection conn = null;
String noTransform = "";
obj = new URL(url);
conn = obj.openConnection();
noTransform = conn.getHeaderField("cache-control");
if (noTransform !=null && (noTransform.contains("no-transform") || noTransform.contains("private") )){
news.setIsGoogleLiteURL("false");
return news;
}
else {
news.setIsGoogleLiteURL("false");
return news;
}

Instead of making a GET request, try making a HEAD request.
https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.4

How to confirm the file getdata.php exist in given url

This is my code, currently it just can connect to the url. how to confirm the file is exist(gedata.php). (the url is not real, i change the real url for security reason)
URLConnection conn = new URL("http://testweb/trueweb.com.my/getdata.php").openConnection();
conn.connect();
any idea folk?
EDIT ANSWERED (credit to Hanlet Escaño for providing code):
URLConnection conn = new URL("http://www.google.com.my").openConnection();
conn.connect();
int code = ((java.net.HttpURLConnection)conn).getResponseCode();
if (code == 404)
{
System.out.println("URL not exist");
}
else{
System.out.println("URL Exist!");
}

Try this:
int code = ((java.net.HttpURLConnection)conn).getResponseCode();
Then check if the code is 404 you know your page did not exist:
if (code == 404)
{
...
}

You can check the HTTP headers for a status code. You really only want to look for 200. 200 = page is ok and 404 is page not found

Checking the status of a web page [duplicate]

This question already has answers here:
How to use java.net.URLConnection to fire and handle HTTP requests
(12 answers)
Closed 9 years ago.
Need to make a program that takes a valid URL of a webpage like www.stackoverflow.com/questions and its IP address equivalent. The program will then find that webpage and return the status code of the page to us such as 200 OK and 404 NOT FOUND. If the webpage isn’t reachable, a message should be returned explaining the situation.
Here’s what I have done so far:
interface Result {
public boolean ok ();
public String message (); }
class Page {
public Result check ( String wholeURL ) throws Exception {
throw new Exception ( "Not sure about the rest”); } }
Also if I were to check a page like http://www.stackoverflow.com I’ll create an instance of Page and then do something like this:
Page page = new PageImplementation ();
Result result = page.check ( "http://www.stackoverflow.com:60" );
if ( result.ok () ) { ... }
else { ... }
The object that is returned is an instance of Result, and the “ok” method should return true when the status code is 200 OK but false otherwise. The method “msg” should return the status code as string.

Have a look at the HttpURLConnection class within the JDK or use Apache Http Components.
Basically you try to connect to the url and check the response header or wait for a timeout if the server isn't reachable at all.
With HttpURLConnection it might look like this:
URL url = new URL("http://www.stackoverflow.com");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.connect();
int httpStatusCode = connection.getResponseCode(); //200, 404 etc.

You can use some api like commons http ,
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.*;
import org.apache.commons.httpclient.params.HttpMethodParams;
..........
public Result check ( String fullURL ) throws Exception {
HttpClient client = new HttpClient();
GetMethod method = new GetMethod(url);
int statusCode = client.executeMethod(method);
//Update your result object based on statuscode
}

Checking for a file on a remote server, and if it exists, download it

I have recently been trying to create an updater for my program. The updater is supposed to go to dropbox, look through the files in the 'public' folder, and decide if it exists. It works, and can DOWNLOAD the file, but it can't CHECK if the file exists. I saw this and I thought it was a solution, but it doesn't seem to be working.
Here is the code I'm using to check for the file's existence:
public static boolean exists(String URLName) {
try {
HttpURLConnection.setFollowRedirects(false);
HttpURLConnection con = (HttpURLConnection) new URL(URLName)
.openConnection();
con.setRequestMethod("HEAD");
return (con.getResponseCode() == HttpURLConnection.HTTP_OK);
} catch (Exception e) {
e.printStackTrace();
return false;
}
}
However, it seems to always return true. The files I'm accessing all begin with "App_" and end with ".zip". The only thing that varies is the version, which is in the #.### format.
Here is the FULL code for how I'm checking it:
public static void main(String[] args) throws IOException,
InterruptedException {
double origVersion = 0.008;
double versionTimes = 0.000;
while(exists("http://dl.dropbox.com/u/.../" + "App_"+ String.valueOf(origVersion + versionTimes) + ".zip")) {
versionTimes = round(versionTimes + 0.001);
//origVersion = round(origVersion + 0.001);
System.exit(0);
}
}
public static boolean exists(String URLName) {
try {
HttpURLConnection.setFollowRedirects(false);
// note : you may also need
// HttpURLConnection.setInstanceFollowRedirects(false)
HttpURLConnection con = (HttpURLConnection) new URL(URLName)
.openConnection();
con.setRequestMethod("HEAD");
return (con.getResponseCode() == HttpURLConnection.HTTP_OK);
} catch (Exception e) {
e.printStackTrace();
return false;
}
}
static double round(double d) {
DecimalFormat twoDForm = new DecimalFormat("#.###");
return Double.valueOf(twoDForm.format(d));
}
Sorry about that... That code was too long. Anyway. To test this, right now it will check if version 0.009 is available. Which it is. Its full version is in the variable double origVersion. Now, if you set origVersion to 0.009, it will check for 0.01. Which is fine, except for the fact that App_0.01.zip DOES NOT exist, and yet it still says it does!
I have also looked into wget to solve this problem, by starting up wget with the arguments
THEFILENAME --no-proxy --spider
but that didn't work either. Can anyone help me? I would greatly appreciate it.
I also saw somewhere else that you could establish a connection with the file and if it secures, the file exists. If not, it doesn't. However, I have no idea how to do that. Can anyone bring me out of the dark?
[EDIT]
Also, running the THEFILENAME --no-proxy --spider on wget worked, and outputted the following, when checking for version 0.009:
Spider mode enabled. Check if remote file exists.
--2012-03-16 08:59:55-- http://dl.dropbox.com/u/.../....zip
Resolving dl.dropbox.com... 107.21.103.249, 107.20.135.4, 107.20.198.68, ...
Connecting to dl.dropbox.com|107.21.103.249|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 758067 (740K) [application/zip]
Remote file exists.
And when checking for version 0.01:
Spider mode enabled. Check if remote file exists.
--2012-03-16 09:01:15-- http://dl.dropbox.com/u/.../....zip
Resolving dl.dropbox.com... 107.22.196.64, 50.19.217.32, 174.129.218.194, ...
Connecting to dl.dropbox.com|107.22.196.64|:80... connected.
HTTP request sent, awaiting response... 404 NOT FOUND
Remote file does not exist -- broken link!!!
I also tried reading the output of wget using this, and using if(input.indexOf("404 NOT FOUND") == -1), but still to no avail.

Doing a HEAD request is definitely the right way to check if a file exists on a remote server.
I can't see anything wrong with your exists(String URLName) method so I would check that it is being passed the URL you think it is.
What looks odd to be is this line:
versionTimes = round(versionTimes + 0.001);
You're changing the value of versionTimes in between checking for the existence of the file and printing your message.