Develop Reference

Why is my binary data bigger after getting it from the webserver?

I need to serve a binary file through a web service implemented in Python/Django. The problem is, that when I compare the original file with the transferred file with vbindiff I see trailing bytes on the transferred file, sadly rendering it useless.
The Binary File is accessed saved by a client in Java with:
HttpURLConnection userdataConnection = null;
URL userdataUrl = null;
try {
userdataUrl = new URL("http://localhost:8000/app/vuforia/10");
userdataConnection = (HttpURLConnection) userdataUrl.openConnection();
userdataConnection.setRequestMethod("GET");
userdataConnection.setRequestProperty("Content-Type", "application/octet-stream");
userdataConnection.connect();
InputStream userdataStream = new BufferedInputStream(userdataConnection.getInputStream());
try (ByteArrayOutputStream fileStream = new ByteArrayOutputStream()) {
byte[] buffer = new byte[4094];
while (userdataStream.read(buffer) != -1) {
fileStream.write(buffer);
}
byte[] fileBytes = fileStream.toByteArray();
try (FileOutputStream fos = new FileOutputStream("./test.dat")) {
fos.write(fileBytes);
}
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
I think that HttpURLConnection.getInputStream only reads the body of the response, or not?
This code serves the data in the backend
in views.py:
if request.method == "GET":
all_data = VuforiaDatabase.objects.all()
data = all_data.get(id=version)
return FileResponse(data.get_dat_bytes())
in models.py:
def get_dat_bytes(self):
return self.dat_upload.open()
How do I go about transferring the binary data 1:1?

You’re ignoring the return value of InputStream.read.
From the documentation:
Returns:
the total number of bytes read into the buffer, or -1 if there is no more data because the end of the stream has been reached.
Your code is assuming that the buffer is filled with every call to userdataStream.read(buffer), instead of checking how many bytes were actually read into buffer.
You don’t need to read from an InputStream at all. Just use Files.copy:
Path file = Paths.get("./test.dat");
try (InputStream userdataStream = new BufferedInputStream(userdataConnection.getInputStream())) {
Files.copy(userdataStream, file, StandardCopyOption.REPLACE_EXISTING);
}

You always write a multiple the 4094 bytes, no matter how many bytes you actually read.
Don't do .write(buffer); write the amount you actually read. This is what userdataStream.read returns you. It can return a number smaller than the buffer size, but still positive.
If you project is using Apache Commons already, you can just use copyInputStreamToFile.
Note: 4K = 4096, not 4094, and it's a ridiculously small buffer, unless you operate something like a smartcard. On a PC, use something like a few hundred kb at least.

Download file in java when Content-Length is incorrectly set?

I have an android app that downloads and uses a file at runtime. The file is valid as I can download it via the browser and open it up, etc. However my app kept reporting that the file is corrupted.
After investigation I discovered the server (which I have no control over) is returning an incorrect "Content-Length:" (~180 vs ~120000). The header is the culprit as I confirmed the issue by downloading the file with curl - which also resulted in a truncated file.
After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length. To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Anybody know of a way to download a file if the Content-Length is incorrectly set? A browser can.
Here's my latest attempt:
public static void downloadFileFromRemoteUrl(String urlString, String destination){
try {
URL url = new URL(urlString);
File file = new File(destination);
URLConnection urlConnection = url.openConnection();
InputStream inputStream = urlConnection.getInputStream();
byte[] buffer = new byte[1024];
int curLength = 0;
int newLength = 0;
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
byteArrayOutputStream.write(buffer, 0, newLength);
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(byteArrayOutputStream.toByteArray());
fos.close();
android.util.Log.d("DB UPDATE", "Done downloading database. Size: " + byteArrayOutputStream.toByteArray().length);
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
}

After some research I concluded that my use of BufferedInputStream to append to a ByteArrayBuffer is autosizing the byte array to the url connections content length.
Nonsense. You are crediting those classes with paranormal powers. How could an output stream possibly become aware of the Content-length header? The URLConnection's input stream is being terminated at the content-length. Correctly.
To circumvent this, I tried to use ByteArrayOutputStream instead, however this solved nothing.
Of course not.
Anybody know of a way to download a file if the Content-Length is incorrectly set?
You could use a Socket and engage in HTTP yourself, which is less trivial than it sounds. But the problem is at the server and that's where it should be fixed. Complain. Or else #Zong Yu is correct and the page is HTML containing JavaScript, say.
NB You don't need to read the entire file into memory:
while((newLength = inputStream.read(buffer))>0)
{
curLength += newLength;
fos.write(buffer, 0, newLength);
}

My final "solution" was to realize I was dealing with a 301 redirect response and not the actual resource! I updated the section that handles my url, checking for a 301 and if exists, update the url. The new url contained the Content-Length that corresponded with the file I was downloading.
// start by creating an http url connection object
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
// determine if this is a redirect
boolean redirect = false;
int status = httpURLConnection.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
// if it is, we need a new url
if (redirect) {
String newUrl = httpURLConnection.getHeaderField("Location");
httpURLConnection = (HttpURLConnection) new URL(newUrl).openConnection();
}

Try Fetch. Fetch is an in app download manager for Android. It's very easy to use. Find the GitHub page here. The project comes with several demos that you can try out. Disclaimer: I'm the creator of Fetch, and it is open source.

Download file hosted on a website

I am trying to download a file that I have hosted on mega. The code is as follows:
AsyncTask.execute(new Runnable() {
#Override
public void run() {
LoggerDebug.d("Prakhar", "Inside run");
try {
URL downloadUrl = new URL(url);
HttpURLConnection httpURLConnection = (HttpURLConnection) downloadUrl.openConnection();
httpURLConnection.setRequestMethod("GET");
httpURLConnection.setDoOutput(true);
httpURLConnection.connect();
File SDCardRoot = Environment.getExternalStorageDirectory();
File downloadFile = new File(SDCardRoot, "downloaded.zip");
FileOutputStream fileOutputStream = new FileOutputStream(downloadFile);
InputStream inputStream = httpURLConnection.getInputStream();
int totalSize = httpURLConnection.getContentLength();
int downloadedSize = 0;
byte[] buffer = new byte[1024];
int bufferLength = 0;
LoggerDebug.d("Prakhar", String.valueOf(totalSize));
while ((bufferLength = inputStream.read(buffer)) > 0 ) {
LoggerDebug.d("Prakhar", String.valueOf(downloadedSize));
fileOutputStream.write(buffer, 0, bufferLength);
downloadedSize += bufferLength;
}
} catch (Exception e) {
e.printStackTrace();
}
}
});
}
Now after I run this code sample I get the following:
Total size is : -1
Downloaded size : 1024
On the mobile I can see the downloaded.zip file, its size is 1.97KB and the original size of the upload is 87.1KB
Where am I going wrong?

So I don't know if you solved your problem yet, but I can see a couple potential issues.
setDoOutput(true) indicates you're going to be uploading according to the documentation here. Specifically, it says:
To upload data to a web server, configure the connection for output using setDoOutput(true).
You're not uploading from what I'm seeing, so this isn't needed, and it's unclear how the server would respond since it appears you're requesting the ability to upload.
A return of -1 indicates an EOF. You're not accessing the file you're intending to, likely due to #3.
You probably can't just download from their site willy-nilly. You need to authenticate somehow. This could be an API key, session id, yadda yadda. They have their own API, and according to this document there is logging in via e-mail in their API. The page doesn't render correctly in GitHub, but you'll need to figure out how to log in. They have an Android app example, but it looks like the API is all native code.

Have you tried changing:
while ((bufferLength = inputStream.read(buffer)) > 0 ) {
to
while ((bufferLength = inputStream.read(buffer)) != null) {

Java REST - call method after POST to REST

I'm using a Java REST service for a file upload.
The file should land on my server, which it does, then move to Amazon S3 bucket.
The upload to the server is fine, but the 2nd call to another method does not work.
I assume because there is a timeout issue?
The code to move the file to amazon works in another app, but I am not able to get it working within my REST project.
Here is the method:
#POST
#Path("/upload")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(#FormDataParam("file") InputStream inputStream,
#FormDataParam("file") FormDataContentDisposition file, #FormDataParam("filename") String filename){
Logger log = Logger.getLogger("Mike");
String response = "";
File f = null;
try {
final String FILE_DESTINATION = "C://uploads//" + file.getFileName();
f = new File(FILE_DESTINATION);
OutputStream outputStream = new FileOutputStream(f);
int size = 0;
byte[] bytes = new byte[1024];
while ((size = inputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, size);
}
outputStream.flush();
outputStream.close();
log.info("upload complete for initial file!");
//move file to Amazon S3 Bucket.
AmazonS3 s3 = new AmazonS3Client(
new ClasspathPropertiesFileCredentialsProvider());
log.info("trying put request");
PutObjectRequest request = new PutObjectRequest("site.address.org","/pdf/PDF_Web_Service/work/"+f.getName(),f);
log.info(f.getName());
log.info(f.getAbsolutePath());
s3.putObject(request);
log.info("put request complete");
response = "File uploaded " + FILE_DESTINATION;
} catch (Exception e) {
e.printStackTrace();
}
return Response.status(200).entity(response).build();
}
Specifically, here is the part not working. I am not getting any log info either:
//move file to Amazon S3 Bucket.
AmazonS3 s3 = new AmazonS3Client(
new ClasspathPropertiesFileCredentialsProvider());
log.info("trying put request");
PutObjectRequest request = new PutObjectRequest("site.address.org","/pdf/PDF_Web_Service/work/"+f.getName(),f);
log.info(f.getName()); log.info(f.getAbsolutePath());
s3.putObject(request); log.info("put request complete");

Michael,
If it's a time-out issue, it's common practice to use guava's Listenable Future to chain your tasks together. What your web sequence will look like then is:
a) Client sends file
b) Server responds with 200 once file completes uploading.
c) Once the server is done loading the file, chain the future to then upload to S3.
Chaining listenable futures is common practice to separate functionality and ensure a time out doesn't occur by breaking up your code and essentially pipe-lining it.
Please let me know if you have any questions!

I moved the Amazon code into the try block and now it works.

Download file in my webserver

I would like to implement a function where you send a URL of a photo and my server will automatically download and store it in a specified folder.
I studied some use cases, but as a beginner in this area of the web, I was a bit lost. I thought about FTP but is not exactly what I want.
Like that, function on my webservice (using Java + Tomcat + AXIS2)
void getPhoto(URL url){
//receive a photo and store at folder /photos
}
but, I don't know what use, I was looking for some httppost or httpget, should I still looking for in this way? Has a dummie sample, to show me the basic way?

I would like to implement a function where you send a URL of a photo and my server will automatically download and store it in a specified folder.
That's not exactly "uploading", but just "downloading".
Just call openStream() on the URL and you've an InputStream which you can do anything with. Writing to a FileOutputStream for example.
InputStream input = url.openStream();
// ...

hey use this code to download.
try {
URL url = new URL(url of file );
URLConnection conection = url.openConnection();
conection.connect();
InputStream input = new BufferedInputStream(url.openStream());
String downloadloc = "D:\"; // or anything
OutputStream output = new FileOutputStream(downloadloc
+ "\name of file.ext");
byte data[] = new byte[1024];
long total = 0;
while ((count = input.read(data)) != -1) {
total += count;
output.write(data, 0, count);
}
output.flush();
output.close();
input.close();
} catch (Exception e) {
}

You want to look at using an HttpURLConnection, call it's 'connect' and 'getInputStream' methods, continually reading from that stream and writing that data to a file with e.g. a FileOutputStream.

To download a file using a URL, as an alternative to what suggested by others, you can take a look to Apache Commons HttpClient.
There is also a well written tutorial.