I got a program which needs to compare a file that situated on the Local disk and one on a FTP server.
I've decided to go with md5 checksum. I am able to do it with the local file, but I am having problems with the ftp one. Also, I am using Apache FTPClient common.
MessageDigest digest = MessageDigest.getInstance("MD5");
FileInputStream is = new FileInputStream(FTP_listFiles[i]); //ERROR HERE
//FTP_files is a FTPFile from FTPClient apache commons.
byte[] buffer = new byte[8192];
int read = 0;
try {
while( (read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
byte[] md5sum = digest.digest();
BigInteger bigInt = new BigInteger(1, md5sum);
String output = bigInt.toString(16);
System.out.println("MD5: \n" + output);
}
catch(IOException e) {
throw new RuntimeException("Unable to process file for MD5", e);
} finally {
try {
is.close();
}
catch(IOException e) {
throw new RuntimeException("Unable to close input stream for MD5 calculation", e);
}
}
NB: If impossible, do you know any equivilant to md5 but can do the same?
What are trying to achieve? Are.you aware that to compute the md5 locally, you will need to download the full file ?
There is a special ftp extension that provide a server-side md5 to the client but it'd not generally supported. Some server also implement some similar proprietary functionality; you''ll have to check for your specific server.
You may want to have a look at the XCRC or XMD5 or XSHA1 commands, for example.
Related
I'm trying to develop a file updater for some files in a folder, to Sync an FTP server with a local folder, using Java on the client and PHP on the server side.
On the server side, I'm calculating the md5_file($filename) for the file and returning every of them on a JSON.
On Java, I'm checking first if the file exists in the local folder. If the file exists, then I check for the MD5 checksum to see if the file is exactly the same as the online one.
The MD5 is not matching when checking .txt or .lua files. It's ok when checking other file types, as .dds texture files.
The MD5 I'm using on Java is this:
private String md5(File f) throws FileNotFoundException, NoSuchAlgorithmException {
MessageDigest digest = MessageDigest.getInstance("MD5");
InputStream is = new FileInputStream(f);
byte[] buffer = new byte[8192];
int read = 0;
try {
while( (read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
byte[] md5sum = digest.digest();
BigInteger bigInt = new BigInteger(1, md5sum);
String output = bigInt.toString(16);
return output;
}
catch(IOException e) {
throw new RuntimeException("Unable to process file for MD5", e);
}
finally {
try {
is.close();
}
catch(IOException e) {
throw new RuntimeException("Unable to close input stream for MD5 calculation", e);
}
}
}
As an example, for a description.lua file, with the following contents:
livery = {
{"KC-130_fusel", 0 ,"KC-130_map_fus",false};
{"KC-130_wing", 0 ,"KC-130_map_wingS",false};
{"KC-130_wing_2", 0 ,"KC-130_map_wings_2",false};
{"KC-130_notes", 0 ,"KC-130_notes_empty",true};
{"KC-130_FPod", 0 ,"kc-130_map_drg",false};
}
name = "Spain ALA 31 TK.10-06"
countries = {"SPN"} -- and any others you want to add
PHP md5_file($filename) = d0c32f9e38cc6e1bb8b54a6aca4a0190
JAVA md5(File) = 08bf57441b904c69e9ce3ca02a9257c7
I've been trying to find a relation between those two codes to see what's making the difference, but have not find any. I have checked like 10 md5 scripts for Java and all of them give the same result.
Is there any way I can fix this?
EDIT: Solution given on first comment: Change the Transfer type on the FTP Client to Binary to avoid changing txt files to ASCII encoding, changing their length and md5.
I have a method which takes a file and upload it on given path.
Here is my service
public String fileUpload(MultipartFile file) throws IOException {
log.debug("uploading video");
File fileUpload = new File(file.getOriginalFilename());
if (!file.isEmpty()) {
InputStream inputStream = file.getInputStream();
byte[] buf = new byte[1024];
FileOutputStream fileOutputStream = new FileOutputStream(new File(
fileUploadPath + File.separator
+ file.getOriginalFilename()));
int numRead = 0;
while ((numRead = inputStream.read(buf)) >= 0) {
fileOutputStream.write(buf, 0, numRead);
}
inputStream.close();
fileOutputStream.close();
}
else {
return Constants.EMPTY_FILE;
}
}
After uploading the file i have to save it information in my database.File size could be 1GB or 2GB.My problem is how would i know the file is fully uploaded or not.So that i can save it status uploaded successfully in my db.
Anyone please help me looking into this ?
You can create a MD5 hash before uploading the file. Take a look at this on creating MD5 hash with JavaScript via How to calculate md5 hash of a file using javascript.
And after the file is completely uploaded, you can use MessageDigest to create another MD5 hash to compare it again the one before the upload. (See example: http://javarevisited.blogspot.com/2013/06/how-to-generate-md5-checksum-for-files.html)
I am trying to port the following java code to .net:
private final byte[] zipLicense(byte lic[])
{
byte buf[];
ByteArrayInputStream bis;
DeflaterInputStream dis;
ByteArrayOutputStream bos;
buf = new byte[64];
bis = new ByteArrayInputStream(lic);
dis = new DeflaterInputStream(bis, new Deflater());
bos = new ByteArrayOutputStream();
byte abyte0[];
int len;
while((len = dis.read(buf)) > 0)
bos.write(buf, 0, len);
abyte0 = bos.toByteArray();
try
{
bis.close();
dis.close();
bos.close();
}
catch(IOException ex)
{
ex.printStackTrace();
}
return abyte0;
}
My best shot was this code in C#:
private byte[] zipLicense(byte[] lic)
{
var outputMemStream = new MemoryStream();
ZipOutputStream zipStream;
using (zipStream = new ZipOutputStream(outputMemStream))
{
zipStream.Write(lic, 0, lic.Length);
Debug.WriteLine(string.Format("Compressed bytes: {0}", outputMemStream.Length));
}
return outputMemStream.ToArray();
}
ZipOutputStream is a class from SharpZipLib
When I try to run the C# code, I get error on first attempt to write to zipStream
zipStream.Write(lic, 0, lic.Length);
The error states that I haven't provided "No entry". I see in examples that one can and probably should seciffy an entry string to a zip stream, but what java code puts as an entry then? Please help in porting this java functionality to .Net. Thanks!
The Java DeflaterInputStream is more like .NET's DeflateStream. That is, it's simply a compressed stream, without the directory index that a full .zip file would contain.
Try this:
private byte[] zipLicense(byte[] lic)
{
var outputMemStream = new MemoryStream();
using (DeflateStream stream =
new DeflateStream(outputMemStream, CompressionMode.Compress, true))
{
stream.Write(lic, 0, lic.Length);
}
Debug.WriteLine(string.Format("Compressed bytes: {0}", outputMemStream.Length));
return outputMemStream.ToArray();
}
Note that I've added a call to Flush(). Without this, the outputMemStream.Length property may not be current (i.e. not quite the full length of the resulting stream).
For what it's worth, .NET now has reasonably good .zip file support built-in (e.g. ZipArchive class). So if you do find yourself actually needing that some day, I would try to use that first rather than adding a third-party library to your deployment.
I am trying to load a .swf file in my page, i would like to make this load faster by converting it to Base64, rather providing a src. This is working great with image formats by using the below code
Java code
BufferedImage buffImg = ImageIO.read(new File(imagePath));
ImageIO.write(buffImg, imgExtension, bos);
byte[] imageBytes = bos.toByteArray();
BASE64Encoder encoder = new BASE64Encoder();
imageString = encoder.encode(imageBytes);
but this is not working for swf file. is there any possible way to achieve this.
Html
<object width="10" height="10" data="data:application/x-shockwave-flash;base64, RldTCSEAAABIAZAAZAAADAEARBEIAAAAQwIAAP9AAAAA"></object>
thanks in advance.
Trying to get the file in base64 will not speed up the file transfer, it's just the opposite as it will convert the file which is stored in bytes (base256 if it can be said that way) to base64 (64 printable characters), so the final amount of data you will be transfering is more.
The only "win" is that you might be able to load it as part of the page instead of the browser making another call for the swf file, which should be no issue on http 1.1.
Unless you have some other good reason to do this, I would not suggest this kind of practice.
If you have your swf file(s) in a database as a blob, you could just make a servlet which sets the proper contenttype and write the whole file with the ServletOutputStream, without any tags. In your html code, you would have to reference to the servlet instead of a fixed file.
If you still want to convert the file to base64, you shouldn't use some image API, but get the file in a standard way for binary files, here's a sample that should do the job:
http://www.javapractices.com/topic/TopicAction.do?Id=245
You can still do the encoding as you did it once you have a byte array:
File file = new File(imagePath);
log("File size: " + file.length());
byte[] result = null;
try {
InputStream input = new BufferedInputStream(new FileInputStream(file));
result = readAndClose(input);
}
catch (FileNotFoundException ex){
log(ex);
}
BASE64Encoder encoder = new BASE64Encoder();
imageString = encoder.encode(result);
And the readAndClose method:
byte[] readAndClose(InputStream aInput){
byte[] bucket = new byte[32*1024];
ByteArrayOutputStream result = null;
try {
try {
result = new ByteArrayOutputStream(bucket.length);
int bytesRead = 0;
while(bytesRead != -1){
bytesRead = aInput.read(bucket);
if(bytesRead > 0){
result.write(bucket, 0, bytesRead);
}
}
}
finally {
aInput.close();
}
}
catch (IOException ex){
log(ex);
}
return result.toByteArray();
}
This should do the trick, maybe some fine tunings to adapt the code to your specific situation, optimize it and better error handling...
I need a very simple function that allows me to read the first 1k bytes of a file through FTP. I want to use it in MATLAB to read the first lines and, according to some parameters, to download only files I really need eventually. I found some examples online that unfortunately do not work. Here I'm proposing the sample code where I'm trying to download one single file (I'm using the Apache libraries).
FTPClient client = new FTPClient();
FileOutputStream fos = null;
try {
client.connect("data.site.org");
// filename to be downloaded.
String filename = "filename.Z";
fos = new FileOutputStream(filename);
// Download file from FTP server
InputStream stream = client.retrieveFileStream("/pub/obs/2008/021/ab120210.08d.Z");
byte[] b = new byte[1024];
stream.read(b);
fos.write(b);
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (fos != null) {
fos.close();
}
client.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}
the error is in stream which is returned empty. I know I'm passing the folder name in a wrong way, but I cannot understand how I have to do. I've tried in many way.
I've also tried with the URL's Java classes as:
URL url;
url = new URL("ftp://data.site.org/pub/obs/2008/021/ab120210.08d.Z");
URLConnection con = url.openConnection();
BufferedInputStream in =
new BufferedInputStream(con.getInputStream());
FileOutputStream out =
new FileOutputStream("C:\\filename.Z");
int i;
byte[] bytesIn = new byte[1024];
if ((i = in.read(bytesIn)) >= 0) {
out.write(bytesIn);
}
out.close();
in.close();
but it is giving an error when I'm closing the InputStream in!
I'm definitely stuck. Some comments about would be very useful!
Try this test
InputStream is = new URL("ftp://test:test#ftp.secureftp-test.com/bookstore.xml").openStream();
byte[] a = new byte[1000];
int n = is.read(a);
is.close();
System.out.println(new String(a, 0, n));
it definitely works
From my experience when you read bytes from a stream acquired from ftpClient.retrieveFileStream, for the first run it is not guarantied that you get your byte buffer filled up. However, either you should read the return value of stream.read(b); surrounded with a cycle based on it or use an advanced library to fill up the 1024 length byte[] buffer:
InputStream stream = null;
try {
// Download file from FTP server
stream = client.retrieveFileStream("/pub/obs/2008/021/ab120210.08d.Z");
byte[] b = new byte[1024];
IOUtils.read(stream, b); // will call periodically stream.read() until it fills up your buffer or reaches end-of-file
fos.write(b);
} catch (IOException e) {
e.printStackTrace();
} finally {
IOUtils.closeQuietly(inputStream);
}
I cannot understand why it doesn't work. I found this link where they used the Apache library to read 4096 bytes each time. I read the first 1024 bytes and it works eventually, the only thing is that if completePendingCommand() is used, the program is held for ever. Thus I've removed it and everything works fine.