Develop Reference

Download file in java when Content-Length is incorrectly set?

I have an android app that downloads and uses a file at runtime. The file is valid as I can download it via the browser and open it up, etc. However my app kept reporting that the file is corrupted.
After investigation I discovered the server (which I have no control over) is returning an incorrect "Content-Length:" (~180 vs ~120000). The header is the culprit as I confirmed the issue by downloading the file with curl - which also resulted in a truncated file.
After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length. To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Anybody know of a way to download a file if the Content-Length is incorrectly set? A browser can.
Here's my latest attempt:
public static void downloadFileFromRemoteUrl(String urlString, String destination){
try {
URL url = new URL(urlString);
File file = new File(destination);
URLConnection urlConnection = url.openConnection();
InputStream inputStream = urlConnection.getInputStream();
byte[] buffer = new byte[1024];
int curLength = 0;
int newLength = 0;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
byteArrayOutputStream.write(buffer, 0, newLength);
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(byteArrayOutputStream.toByteArray());
fos.close();
android.util.Log.d("DB UPDATE", "Done downloading database. Size: " + byteArrayOutputStream.toByteArray().length);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}

After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length.
Nonsense. You are crediting those classes with paranormal powers. How could an output stream possibly become aware of the Content-length header? The URLConnection's input stream is being terminated at the content-length. Correctly.
To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Of course not.
Anybody know of a way to download a file if the Content-Length is incorrectly set?
You could use a Socket and engage in HTTP yourself, which is less trivial than it sounds. But the problem is at the server and that's where it should be fixed. Complain. Or else #Zong Yu is correct and the page is HTML containing JavaScript, say.
NB You don't need to read the entire file into memory:
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
fos.write(buffer, 0, newLength);
}

My final "solution" was to realize I was dealing with a 301 redirect response and not the actual resource! I updated the section that handles my url, checking for a 301 and if exists, update the url. The new url contained the Content-Length that corresponded with the file I was downloading.
// start by creating an http url connection object
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
// determine if this is a redirect
boolean redirect = false;
int status = httpURLConnection.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
// if it is, we need a new url
if (redirect) {
String newUrl = httpURLConnection.getHeaderField("Location");
httpURLConnection = (HttpURLConnection) new URL(newUrl).openConnection();
}

Try Fetch. Fetch is an in app download manager for Android. It's very easy to use. Find the GitHub page here. The project comes with several demos that you can try out. Disclaimer: I'm the creator of Fetch, and it is open source.

Sending Java GET parameters to the server (witout hanging)

I would like to send a GET parameter to a server. I really do not need the InputStream (below), but the request is actually sent when I call "getInputStream". The problem is, this code hangs on getInputStream. The timeout does not apply because the connection is actually established (does not time-out).
What do I need to change so that I'm sending a clean GET to the server without hanging?
URL url = new URL("http://localhost:8888/abc?message=abc"); //[edit]
URLConnection uc = url.openConnection();
uc.setRequestProperty("Accept-Charset", "UTF-8");
uc.setConnectTimeout(1000);
InputStream in = uc.getInputStream();
in.close();
In case it matters, I'm testing with netcat -l as the server instead of using an actual web server. None the less, I would like this code to be very fail-safe so it the server can't adversely effect this code.

I basically gave up in using the URLConnection and wrote the code to use a socket instead. I'm still open for improvements, light-weight posting to a web server is very useful.
URL u = new URL("http://localhost:8888/abc?message=abc");
String get = "";
if (u.getPath() != null)
get += u.getPath();
if (u.getQuery() != null)
get += "?" + u.getQuery();
if (u.getRef() != null)
get += "#" + u.getRef();
Socket socket = new Socket();
socket
.connect(new InetSocketAddress(u.getHost(), u.getPort()),
750);
OutputStream out = socket.getOutputStream();
out.write(("GET " + get + "\n\n").getBytes());
out.close();

POST Requests from a Servlet

I am writing a servlet using eclipse that receives POST request from a client that should do some splitting on the received text, access google geolocation api to get some data and display to the user.
On a localhost, this works perfectly fine. On an actual server (tried with Openshift and CloudBees), this doesn't work. I can see the splitting reply but not the reply from google geolocation service. There is always an error logged into the console from google service. However, the same code works perfectly fine on localhost.
After I receive the POST request in the doPost method of the servlet, I am doing the following to access the Google GeoLocation service:
//Attempting to send data to Google Geolocation Service
URL url;
HttpURLConnection connection = null;
try {
//Create connection
url = new URL("https://www.googleapis.com/geolocation/v1/geolocate?key=MyAPI");
connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
connection.setUseCaches (false);
connection.setDoInput(true);
connection.setDoOutput(true);
//Send request with data (output variable has the JSON data)
DataOutputStream wr = new DataOutputStream (
connection.getOutputStream ());
wr.writeBytes (output);
wr.flush ();
wr.close ();
//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response2 = new StringBuffer();
while((line = rd.readLine()) != null) {
response2.append(line);
response2.append('\r');
}
rd.close();
//Write to Screen using out=response.getWriter();
out.println("Access Point's Location = " + response2.toString());
} catch (Exception e) {
e.printStackTrace();
} finally {
if(connection != null) {
connection.disconnect();
}
Could you tell me why this is happening and how can I make this work? Should I resort to something like AJAX or is there someother work around? I am relatively new to coding and hence, trying to refrain from learning AJAX at this stage. Please let me if there's any other way of getting this to work

Your localhost has your localhost IP as a sending IP. Openshift et al has the Openshift et al IP as a sending IP. So the Google API says "I have only seen that localhost IP twice before, that's fine!", whereas it says "I have seen this Openshift IP millions of times before! NO REPLY FOR YOU!"

java and website redirection detection

I have java related question...
Website www.stationv3.com gets updated daily (most of the time at least, it's kinda irregular). Every time I connect to a site using address www.stationv3.com (using a browser), it redirects me to it's subpage www.stationv3.com/date_of_latest_update.html
I'm trying to make a program that will pull latest comic from the site, but I am not sure how to find out it's exact address. But I know I'd be able to find out if I could somehow find out where where am I being redirected on every connect. Is that possible with java? I know it can do all sorts of quirky things, but I'm still new to internet related stuff...
I used exact site name just to make it easy for you to check outwhat's going on...
And also, I'm creating a generic code, one which could (with some tinkering) be applyed to any site that functions in that manner.

import java.net.*;
public class ShowStationV3Redirect {
public static void main(String[] args) throws Exception {
URL url = new URL(args[0]);
HttpURLConnection.setFollowRedirects(false);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
System.out.println("Response code = " + connection.getResponseCode());
String header = connection.getHeaderField("location");
if (header != null)
System.out.println("www.stationv3.com redirected to " + header);
}
}
The above code snippet tells you what URL you are being redirected to.

I think you could just fecth:
http://www.stationv3.com/comics/{yyyy}{mm}{dd}sv3.gif
and forget about the redirection problem. You can use this code (not tested indeed):
URL server = new URL("<put here the image URL>");
HttpURLConnection connection = (HttpURLConnection)server.openConnection();
connection.setRequestMethod("GET");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
connection.addRequestProperty("Accept","image/gif");
connection.addRequestProperty("Accept-Encoding", "gzip, deflate");
connection.connect();
InputStream is = connection.getInputStream();
OutputStream os = new FileOutputStream("c:/mycomic.gif");
byte[] buffer = new byte[1024];
int byteReaded = is.read(buffer);
while(byteReaded != -1)
{
os.write(buffer,0,byteReaded);
byteReaded = is.read(buffer);
}
os.close();

Blackberry HTTP Connection Issue

I have a simple app written the connects to web service (restful). The app works fine on the blackberry simulator however I'm having problems using it on a blackberry 9300.
I keep getting the error "java.io.ioexception: tunnel down" when the apps attempts to call the web service.
The service I am calling is a simple HTTP post and I'm trying to run this over WIFI (the WIFI connection is working fine for browsing the internet).
I'm using a connection string of "http://127.0.0.1:8080/test/restws;interface=wifi" with the IP address changed to the actual Ip address of the server I'm calling. I can call the restful web service on this server on my laptop browser fine.
The code Im using is similar to below & works fine on the simulator. The only thing im changing between the simulator and the code on the phone is the connection string (using "interface=wifi" as oppose to "deviceside=true")
Is this a code problem or is it a setting I need to change on the handset? Any ideas on what I need to do to overcome this.
Thanks
StreamConnection s = (StreamConnection) Connector
.open(connectionString);
httpConn = (HttpConnection) s;
httpConn.setRequestMethod("POST");
httpConn.setRequestProperty("Content-Length", Integer.toString(postData.length()));
OutputStream output = httpConn.openOutputStream();
output.write(postData.getBytes());
output.flush();
output.close();
String response = httpConn.getResponseMessage();
int statusCode = httpConn.getResponseCode();
if (statusCode != HttpConnection.HTTP_OK) {
}
InputStream is = httpConn.openInputStream();
int ret = 0;
while ((ret = is.read(buf)) > 0) {
os.write(buf, 0, ret);
}
result = new String(os.toByteArray());

I had problems in here .After getting HttpConnection everything is the same i guess. Try this:
ConnectionFactory cf = new ConnectionFactory();
ConnectionDescriptor cd = cf.getConnection("YourUrl");
httpConnector = (HttpConnection) cd.getConnection();
httpConnector.setRequestMethod(HttpConnection.POST);