Develop Reference

Why is my binary data bigger after getting it from the webserver?

I need to serve a binary file through a web service implemented in Python/Django. The problem is, that when I compare the original file with the transferred file with vbindiff I see trailing bytes on the transferred file, sadly rendering it useless.
The Binary File is accessed saved by a client in Java with:
HttpURLConnection userdataConnection = null;
URL userdataUrl = null;
try {
userdataUrl = new URL("http://localhost:8000/app/vuforia/10");
userdataConnection = (HttpURLConnection) userdataUrl.openConnection();
userdataConnection.setRequestMethod("GET");
userdataConnection.setRequestProperty("Content-Type", "application/octet-stream");
userdataConnection.connect();
InputStream userdataStream = new BufferedInputStream(userdataConnection.getInputStream());
try (ByteArrayOutputStream fileStream = new ByteArrayOutputStream()) {
byte[] buffer = new byte[4094];
while (userdataStream.read(buffer) != -1) {
fileStream.write(buffer);
}
byte[] fileBytes = fileStream.toByteArray();
try (FileOutputStream fos = new FileOutputStream("./test.dat")) {
fos.write(fileBytes);
}
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
I think that HttpURLConnection.getInputStream only reads the body of the response, or not?
This code serves the data in the backend
in views.py:
if request.method == "GET":
all_data = VuforiaDatabase.objects.all()
data = all_data.get(id=version)
return FileResponse(data.get_dat_bytes())
in models.py:
def get_dat_bytes(self):
return self.dat_upload.open()
How do I go about transferring the binary data 1:1?

You’re ignoring the return value of InputStream.read.
From the documentation:
Returns:
the total number of bytes read into the buffer, or -1 if there is no more data because the end of the stream has been reached.
Your code is assuming that the buffer is filled with every call to userdataStream.read(buffer), instead of checking how many bytes were actually read into buffer.
You don’t need to read from an InputStream at all. Just use Files.copy:
Path file = Paths.get("./test.dat");
try (InputStream userdataStream = new BufferedInputStream(userdataConnection.getInputStream())) {
Files.copy(userdataStream, file, StandardCopyOption.REPLACE_EXISTING);
}

You always write a multiple the 4094 bytes, no matter how many bytes you actually read.
Don't do .write(buffer); write the amount you actually read. This is what userdataStream.read returns you. It can return a number smaller than the buffer size, but still positive.
If you project is using Apache Commons already, you can just use copyInputStreamToFile.
Note: 4K = 4096, not 4094, and it's a ridiculously small buffer, unless you operate something like a smartcard. On a PC, use something like a few hundred kb at least.

How to know video is successfully uploaded or not?

I have a method which takes a file and upload it on given path.
Here is my service
public String fileUpload(MultipartFile file) throws IOException {
log.debug("uploading video");
File fileUpload = new File(file.getOriginalFilename());
if (!file.isEmpty()) {
InputStream inputStream = file.getInputStream();
byte[] buf = new byte[1024];
FileOutputStream fileOutputStream = new FileOutputStream(new File(
fileUploadPath + File.separator
+ file.getOriginalFilename()));
int numRead = 0;
while ((numRead = inputStream.read(buf)) >= 0) {
fileOutputStream.write(buf, 0, numRead);
}
inputStream.close();
fileOutputStream.close();
}
else {
return Constants.EMPTY_FILE;
}
}
After uploading the file i have to save it information in my database.File size could be 1GB or 2GB.My problem is how would i know the file is fully uploaded or not.So that i can save it status uploaded successfully in my db.
Anyone please help me looking into this ?

You can create a MD5 hash before uploading the file. Take a look at this on creating MD5 hash with JavaScript via How to calculate md5 hash of a file using javascript.
And after the file is completely uploaded, you can use MessageDigest to create another MD5 hash to compare it again the one before the upload. (See example: http://javarevisited.blogspot.com/2013/06/how-to-generate-md5-checksum-for-files.html)

Java checksum md5 on a file situated on a FTP server

I got a program which needs to compare a file that situated on the Local disk and one on a FTP server.
I've decided to go with md5 checksum. I am able to do it with the local file, but I am having problems with the ftp one. Also, I am using Apache FTPClient common.
MessageDigest digest = MessageDigest.getInstance("MD5");
FileInputStream is = new FileInputStream(FTP_listFiles[i]); //ERROR HERE
//FTP_files is a FTPFile from FTPClient apache commons.
byte[] buffer = new byte[8192];
int read = 0;
try {
while( (read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
byte[] md5sum = digest.digest();
BigInteger bigInt = new BigInteger(1, md5sum);
String output = bigInt.toString(16);
System.out.println("MD5: \n" + output);
}
catch(IOException e) {
throw new RuntimeException("Unable to process file for MD5", e);
} finally {
try {
is.close();
}
catch(IOException e) {
throw new RuntimeException("Unable to close input stream for MD5 calculation", e);
}
}
NB: If impossible, do you know any equivilant to md5 but can do the same?

What are trying to achieve? Are.you aware that to compute the md5 locally, you will need to download the full file ?
There is a special ftp extension that provide a server-side md5 to the client but it'd not generally supported. Some server also implement some similar proprietary functionality; you''ll have to check for your specific server.
You may want to have a look at the XCRC or XMD5 or XSHA1 commands, for example.

is it possible to convert .swf file to base64 format in java

I am trying to load a .swf file in my page, i would like to make this load faster by converting it to Base64, rather providing a src. This is working great with image formats by using the below code
Java code
BufferedImage buffImg = ImageIO.read(new File(imagePath));
ImageIO.write(buffImg, imgExtension, bos);
byte[] imageBytes = bos.toByteArray();
BASE64Encoder encoder = new BASE64Encoder();
imageString = encoder.encode(imageBytes);
but this is not working for swf file. is there any possible way to achieve this.
Html
<object width="10" height="10" data="data:application/x-shockwave-flash;base64, RldTCSEAAABIAZAAZAAADAEARBEIAAAAQwIAAP9AAAAA"></object>
thanks in advance.

Trying to get the file in base64 will not speed up the file transfer, it's just the opposite as it will convert the file which is stored in bytes (base256 if it can be said that way) to base64 (64 printable characters), so the final amount of data you will be transfering is more.
The only "win" is that you might be able to load it as part of the page instead of the browser making another call for the swf file, which should be no issue on http 1.1.
Unless you have some other good reason to do this, I would not suggest this kind of practice.
If you have your swf file(s) in a database as a blob, you could just make a servlet which sets the proper contenttype and write the whole file with the ServletOutputStream, without any tags. In your html code, you would have to reference to the servlet instead of a fixed file.
If you still want to convert the file to base64, you shouldn't use some image API, but get the file in a standard way for binary files, here's a sample that should do the job:
http://www.javapractices.com/topic/TopicAction.do?Id=245
You can still do the encoding as you did it once you have a byte array:
File file = new File(imagePath);
log("File size: " + file.length());
byte[] result = null;
try {
InputStream input = new BufferedInputStream(new FileInputStream(file));
result = readAndClose(input);
}
catch (FileNotFoundException ex){
log(ex);
}
BASE64Encoder encoder = new BASE64Encoder();
imageString = encoder.encode(result);
And the readAndClose method:
byte[] readAndClose(InputStream aInput){
byte[] bucket = new byte[32*1024];
ByteArrayOutputStream result = null;
try {
try {
result = new ByteArrayOutputStream(bucket.length);
int bytesRead = 0;
while(bytesRead != -1){
bytesRead = aInput.read(bucket);
if(bytesRead > 0){
result.write(bucket, 0, bytesRead);
}
}
}
finally {
aInput.close();
}
}
catch (IOException ex){
log(ex);
}
return result.toByteArray();
}
This should do the trick, maybe some fine tunings to adapt the code to your specific situation, optimize it and better error handling...

Verify file is copied in Java

I'm working on moving some files to a different directory in my project and it's working great, except for the fact that I can't verify it's moved properly.
I want to verify the length of the copy is the same as the original and then I want to delete the original. I'm closing both FileStreams before I do my verification but it still fails because the sizes are different. Below is my code for closing the streams, verification and deletion.
in.close();
out.close();
if (encCopyFile.exists() && encCopyFile.length() == encryptedFile.length())
encryptedFile.delete();
The rest of the code before this is using a Util to copy the streams, and it's all working fine so really I just need a better verification method.

One wonderful way you can check is to compare md5 hashes. Checking file length doesn't mean they are the same. While md5 hashes doesn't mean they are the same either, it is better than checking the length albeit a longer process.
public class Main {
public static void main(String[] args) throws NoSuchAlgorithmException, IOException {
System.out.println("Are identical: " + isIdentical("c:\\myfile.txt", "c:\\myfile2.txt"));
}
public static boolean isIdentical(String leftFile, String rightFile) throws IOException, NoSuchAlgorithmException {
return md5(leftFile).equals(md5(rightFile));
}
private static String md5(String file) throws IOException, NoSuchAlgorithmException {
MessageDigest digest = MessageDigest.getInstance("MD5");
File f = new File(file);
InputStream is = new FileInputStream(f);
byte[] buffer = new byte[8192];
int read = 0;
try {
while ((read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
byte[] md5sum = digest.digest();
BigInteger bigInt = new BigInteger(1, md5sum);
String output = bigInt.toString(16);
return output;
} finally {
is.close();
}
}
}

You could include a checksum in your copy operation. Perform a checksum on the destination file and see that it matches a checksum on the source.

You could use commons io:
org.apache.commons.io.FileUtils.contentEquals(File file1, File file2)
or you could use checksum methods:
org.apache.commons.io.FileUtils:
static Checksum checksum(File file, Checksum checksum) //Computes the checksum of a file using the specified checksum object.
static long checksumCRC32(File file) //Computes the checksum of a file using the CRC32 checksum routine.

If you get no exception while copying streams, you should be OK. Make sure you don't ignore exceptions thrown by close method!
Update: If you use FileOutputStream, you can also make sure everything was written properly by calling fileOutputStream.getFD().sync() before closing your fileOutputStream.
Of course, if you want to absolutely make sure that files are the same, you can compare their checksums/digests, but that sounds bit paranoid to me.

If the sizes are different, perhaps you are not flushing the output stream before closing it.
Which file is bigger? What are the sizes of each file? Have you actually looked at the two files to see what is different?