Develop Reference

In java some redirected urls return the data from redirected page okay but some do not

I have some Java code that takes a url and then returns the data (and at later point BufferedImage is constructed from it, the problem is it works for most urls from particular website but not all.
The urls are actually redirects so for example I pass
//https://coverartarchive.org/release/bdd0e35c-ce68-3f5b-b957-f83ab5846111/front
it will actually redirect to
//https://ia600301.us.archive.org/31/items/mbid-bdd0e35c-ce68-3f5b-b957-f83ab5846111/mbid-bdd0e35c-ce68-3f5b-b957-f83ab5846111-6094238097.jpg
and return the correct data
But if I pass the seemingly very similar url
//http://coverartarchive.org/release/6b105b89-21ee-414a-b98f-b2756c92b0bc/front
Then although this is what is the url seen if pasted into web-browser
//https://ia902907.us.archive.org/33/items/mbid-6b105b89-21ee-414a-b98f-b2756c92b0bc/mbid-6b105b89-21ee-414a-b98f-b2756c92b0bc-3167310704.jpg
My code only returns 169 bytes
If I pass the url it redirects to directly
//https://ia902907.us.archive.org/33/items/mbid-6b105b89-21ee-414a-b98f-b2756c92b0bc/mbid-6b105b89-21ee-414a-b98f-b2756c92b0bc-3167310704.jpg
then it works okay, but I dont have this url so not a solution.
This is quite old code, maybe a better way to do it now, but is my code broken or is the website broken ?
private static byte[] convertUrlToByteArray(URL url) throws IOException
{
//Get imagedata, we want to ensure we just write the data as is as long as in a supported format
URLConnection connection = url.openConnection();
connection.setConnectTimeout(URL_TIMEOUT);
connection.setReadTimeout(URL_TIMEOUT);
// Since you get a URLConnection, use it to get the InputStream
InputStream in = connection.getInputStream();
// Now that the InputStream is open, get the content length
int contentLength = connection.getContentLength();
// To avoid having to resize the array over and over and over as
// bytes are written to the array, provide an accurate estimate of
// the ultimate size of the byte array
ByteArrayOutputStream tmpOut;
if (contentLength != -1)
{
tmpOut = new ByteArrayOutputStream(contentLength);
}
else
{
tmpOut = new ByteArrayOutputStream(16384); // Pick some appropriate size
}
byte[] buf = new byte[1024];
while (true)
{
int len = in.read(buf);
if (len == -1)
{
break;
}
tmpOut.write(buf, 0, len);
}
in.close();
tmpOut.close(); // No effect, but good to do anyway to keep the metaphor alive
return tmpOut.toByteArray();
}

Download file in java when Content-Length is incorrectly set?

I have an android app that downloads and uses a file at runtime. The file is valid as I can download it via the browser and open it up, etc. However my app kept reporting that the file is corrupted.
After investigation I discovered the server (which I have no control over) is returning an incorrect "Content-Length:" (~180 vs ~120000). The header is the culprit as I confirmed the issue by downloading the file with curl - which also resulted in a truncated file.
After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length. To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Anybody know of a way to download a file if the Content-Length is incorrectly set? A browser can.
Here's my latest attempt:
public static void downloadFileFromRemoteUrl(String urlString, String destination){
try {
URL url = new URL(urlString);
File file = new File(destination);
URLConnection urlConnection = url.openConnection();
InputStream inputStream = urlConnection.getInputStream();
byte[] buffer = new byte[1024];
int curLength = 0;
int newLength = 0;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
byteArrayOutputStream.write(buffer, 0, newLength);
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(byteArrayOutputStream.toByteArray());
fos.close();
android.util.Log.d("DB UPDATE", "Done downloading database. Size: " + byteArrayOutputStream.toByteArray().length);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}

After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length.
Nonsense. You are crediting those classes with paranormal powers. How could an output stream possibly become aware of the Content-length header? The URLConnection's input stream is being terminated at the content-length. Correctly.
To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Of course not.
Anybody know of a way to download a file if the Content-Length is incorrectly set?
You could use a Socket and engage in HTTP yourself, which is less trivial than it sounds. But the problem is at the server and that's where it should be fixed. Complain. Or else #Zong Yu is correct and the page is HTML containing JavaScript, say.
NB You don't need to read the entire file into memory:
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
fos.write(buffer, 0, newLength);
}

My final "solution" was to realize I was dealing with a 301 redirect response and not the actual resource! I updated the section that handles my url, checking for a 301 and if exists, update the url. The new url contained the Content-Length that corresponded with the file I was downloading.
// start by creating an http url connection object
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
// determine if this is a redirect
boolean redirect = false;
int status = httpURLConnection.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
// if it is, we need a new url
if (redirect) {
String newUrl = httpURLConnection.getHeaderField("Location");
httpURLConnection = (HttpURLConnection) new URL(newUrl).openConnection();
}

Try Fetch. Fetch is an in app download manager for Android. It's very easy to use. Find the GitHub page here. The project comes with several demos that you can try out. Disclaimer: I'm the creator of Fetch, and it is open source.

How to download an HttpResponse into a file?

My android app uses an API which sends a multipart HTTP request. I am successfully getting the response like so:
post.setEntity(multipartEntity.build());
HttpResponse response = client.execute(post);
The response is the contents of an ebook file (usually an epub or mobi). I want to write this to a file with a specified path, lets says "/sdcard/test.epub".
File could be up to 20MB, so it'll need to use some sort of stream, but I can just can't wrap my head around it. Thanks!

well it is a simple task, you need the WRITE_EXTERNAL_STORAGE use permission.. then simply retrieve the InputStream
InputStream is = response.getEntity().getContent();
Create a FileOutputStream
FileOutputStream fos = new FileOutputStream(new File(Environment.getExternalStorageDirectory(), "test.epub"))
and read from is and write with fos
int read = 0;
byte[] buffer = new byte[32768];
while( (read = is.read(buffer)) > 0) {
fos.write(buffer, 0, read);
}
fos.close();
is.close();
Edit, check for tyoo

First is the method:
public HttpResponse setHTTPConnection() throws IOException, NullPointerException, URISyntaxException {
HttpClient client = HttpClientBuilder.create().build();
HttpRequestBase requestMethod = new HttpGet();
requestMethod.setURI(new URI("***FileToDownlod***"));
BasicHttpContext localContext = new BasicHttpContext();
return client.execute(requestMethod, localContext);
}
Second is an actual code:
File downloadedFile = new File("filePathToSave");
HttpResponse fileToDownload = setHTTPConnection();
try {
FileUtils.copyInputStreamToFile(fileToDownload.getEntity().getContent(), downloadedFile);
} finally {
fileToDownload.getEntity().getContent().close();
}
Please make sure to change "filePathToSave" to where to save the file and
"FileToDownlod" to from where to download accordingly.
"filePathToSave" is where you want to save the file, if you choose to save it locally then you can simply point to the desktop but don't forget to give a name to your file like "/Users/admin/Desktop/downloaded.pdf" (in Mac).
"FileToDownlod" is in the form of URL e.g "https://www.doesntexist.com/sample.pdf"
Don't panic as the second piece will ask to declare in throws clause or catch multiple exceptions. This piece of code was for a specific purpose please customize for your own need.

As you said response is content of book or simple meaning that you are getting response as file but not able to save response as file.
Use this
HttpResponse<Path> response = client.send(request, HttpResponse.BodyHandlers.ofFile(Paths.get("/sdcard/test.epub"));

Android post high res image running out of memory

Good day fellow developers.
I'm busy for android to upload images from a app.
I also got it working (code will follow below).
But when i send large images (10 megapixels) my app crashes with an out-of-memory exception.
A solution for this is to use compression but what if i want to send the full size image?
I think perhaps something with a stream but i'm not familair with streams. Perhaps urlconnection might help to but i really have no idea.
I give the filename the name File[0 to 9999].jpg
The post value with the image date is called Filedata
I give a UID for the post value dropboxid
The code below works but i would love to solve my problem that prevents me from sending high res images.
kind regards
try
{
ByteArrayOutputStream bos = new ByteArrayOutputStream();
bitmap.compress(CompressFormat.JPEG, 100, bos);
byte[] data = bos.toByteArray();
HttpPost postRequest = new HttpPost(URL_SEND);
ByteArrayBody bab = new ByteArrayBody(data, "File" + pad(random.nextInt(9999) + 1) + ".jpg");
MultipartEntity reqEntity = new multipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
reqEntity.addPart("Filedata", bab);
reqEntity.addPart("dropboxId", new StringBody(URLEncoder.encode(uid)));
postRequest.setEntity(reqEntity);
HttpResponse response = httpClient.execute(postRequest);
BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
String sResponse;
StringBuilder s = new StringBuilder();
while((sResponse = reader.readLine()) != null)
{
s = s.append(sResponse);
}
if(d) Log.i(E, "Send response:\n" + s);
}
catch (Exception e)
{
if(d) Log.e(E, "Error while sending: " + e.getMessage());
return ERROR;
}

When using ByteArrayOutputStream and then calling #toByteArray() you've effectively doubled the amount of memory the JPEG is using. ByteArrayOutputStream keeps an internal array with the encoded JPEG and when you call #toByteArray() it allocates a new array & copies the data from the internal buffer.
Consider encoding large bitmaps to a temporary file & using FileOutputStream and FileInputStream to encode and send the image.
Without "uploading" - your app survives "nicely" with the just the huge bitmap in memory I assume?
Edit: FileBody
File img = new File(this is where you put the path of your image)
ContentBody cb = new FileBody(img, "File" + pad(random.nextInt(9999) + 1) + ".jpg", "image/jpg", null);
MultipartEntity reqEntity = new multipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
reqEntity.addPart("Filedata", cb);
reqEntity.addPart("dropboxId", new StringBody(URLEncoder.encode(uid)));

Uploading a photo via imgur on android programatically

I need help in using, imgur's API, to upload a photo and obviously retrieve a link.
IMGUR API:
http://api.imgur.com/resources_anon
I'm able to get the URI for my image required to be uploaded but how can I implement the api above,
I've downloaded mime4j and httpmime and added them to the libraries, but I can't seem to understand how to use them,
I looked at this but its confused me :
Sending images using Http Post

Just from having a quick look at imgur and this question, I've come up with (pretty much just combined the two) the following. Let me know if it doesn't work.
Bitmap bitmap = yourBitmapHere;
// Creates Byte Array from picture
ByteArrayOutputStream baos = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, baos); // Not sure whether this should be jpeg or png, try both and see which works best
URL url = new URL("http://api.imgur.com/2/upload");
//encodes picture with Base64 and inserts api key
String data = URLEncoder.encode("image", "UTF-8") + "=" + URLEncoder.encode(Base64.encode(baos.toByteArray(), Base64.DEFAULT).toString(), "UTF-8");
data += "&" + URLEncoder.encode("key", "UTF-8") + "=" + URLEncoder.encode(YOUR_API_KEY, "UTF-8");
// opens connection and sends data
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
Edit: It seems we need to pass Base64.DEFAULT as the second option to Base64.encode. Updated the example above.
Edit 2: Can you use the following code, based upon the oracle site, and report back what it outputs:
BufferedReader in = new BufferedReader(
new InputStreamReader(
conn.getInputStream()));
String inputLine;
while ((inputLine = ic.readLine()) != null)
System.out.println(inputLine);
in.close();