I'm working on a project to Encrypt/Decrypt files. as this is my first time, I'm wondering if I'm doing it right or not. till now, my idea about encrypting is this :
Select a file -> Read all its bytes and add it to byte array -> Encrypt the byte array -> write encrypted bytes to same file.
note that in this project output file is same file as input. So I decided to clear file before writing encrypted bytes to it.
This might be stupid (and thats why I'm asking for help), so here is my way
public class Encryptor {
File file;
SecretKeySpec secretKeySpec;
public void setFile(String filePath) throws Exception {
this.file = new File(filePath);
if(!file.isFile()){
throw new Exception("The file you choosed is not valid");
}
}
public void setKey(String keyword){
try {
MessageDigest sha = MessageDigest.getInstance("SHA-256");
sha.update(keyword.getBytes("UTF-8"));
byte[] key = sha.digest();
secretKeySpec = new SecretKeySpec(key, "AES");
} catch (UnsupportedEncodingException | NoSuchAlgorithmException e) {
e.printStackTrace();
}
}
public void encrypt(){
byte[] bFile = new byte[(int) file.length()];
try {
//adding portocol bytes to the file bytes
//String portcol = "encryptor portocol";
//byte[] decPortocol = portcol.getBytes();
//convert file into array of bytes
BufferedInputStream bufferedInputStream = new BufferedInputStream(new FileInputStream(file));
bufferedInputStream.read(bFile);
bufferedInputStream.close();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
//outputStream.write(decPortocol);
outputStream.write(bFile);
byte[] cryptedFileBytes = outputStream.toByteArray();
//Cipher and encrypting
Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, secretKeySpec);
byte[] encryptedBytes = cipher.doFinal(cryptedFileBytes);
//Write Encrypted File
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(new FileOutputStream(file,false));
bufferedOutputStream.write(encryptedBytes);
bufferedOutputStream.flush();
bufferedOutputStream.close();
}catch (Exception e){
e.printStackTrace();
}
}
}
main question
are there other ways to read-encrypt-write on same file together at same time? like reading bytes part by part and at same time encrypting that part and overwrite it with encrypted bytes.
Can You Help me more ?
And also Any information about how to make my encrypted files more safe can also be helpful.
and does my program kill RAM ?!
(NOTE) I'm writing encrypted data on same file for some reasons. I'm not much familiar with how hard drive works. One of my reasons is to prevent file from being recovered later. is there anything I have to know about that ? does what I'm doing prevent unEncrypted file to be recovered later ?
EDIT
#erickson has pointed out something important in his answer. I got to know that this way of encrypting a file is not safe. What I was considering to prevent too, was preventing file from being recovered later. I mean there is no point to encrypt a file and keep it in your hard drive if you once had it unEncrypted there ! in my experience, everytime I recovered a file, I reached last edits of it and I could never get history of changes. I thought this must be the same if I was not wrong in first place. How can I help preventing data recovery then ?!
Writing to a file while reading can work, but it would be easy to introduce a bug that would corrupt the file. For safety's sake, it might be better to write to a temporary file, then delete the original file and replace it with the temporary file. That way, all of the file content is always safely in at least one file.
One caveat about this is that if you encrypt an existing file, there's no guarantee that the original file isn't still recorded on disk. Even if you write to the same file as you read, whether the same storage is overwritten with encrypted data will depend on the underlying file system.
It would be better if the original file was written in its encrypted form. Even if the writing application doesn't support encryption, most operating systems support the creation of an encrypted file system so that any application can keep files secret.
You need to close your reader after you have finished reading the file. You are currently doing it in this line:
bufferedInputStream.close();
So it's ok.
Then, instead of clearing file, you can just simply overwrite it using:
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(new FileOutputStream(filename, false);
Hope that helps :)
Related
I've got this function that encrypts a file, the encryption bit seems to be working, but i can't get it to overwrite the current file.
FileInputStream inputStream = new FileInputStream(input); // Selects file to encrypt
cipher.init(Cipher.ENCRYPT_MODE, secret, ivSpec); // Sets up the encryption
// Creates an the output stream, the encryption is performed here
CipherOutputStream cos = new CipherOutputStream(new FileOutputStream(input + ".secure"), cipher);
byte[] block = new byte[8];
int i;
while ((i = inputStream.read(block)) != -1) // Reads the file
{
cos.write(block, 0, i); // Writes the new file
}
cos.close();
This is working fine, i end up with an encrypted file with original_file_name.txt.secure, but i want it to overwrite the original file. If i remove the .secure bit it doesn't write the file properly.
How can I overwrite the file original file with the encrypted text?
If you remove the .secure part, you'll be trying to read from the file at the same time that you're writing to it. This is not a very good idea...
The best approach would be to do as you've done, and then if all has gone well, you can delete the original file and rename the old one to match its name, using Files.move().
In fact, if you pass the right options to Files.move(), you can get it to overwrite the existing file, meaning that you won't need to delete the original first.
This solves the simultaneous read/write problem you're having, but it's also a whole lot safer for an application like this. If your application crashes or there's a power cut in the middle of encrypting, and you're encrypting in place, then you're completely screwed. If you do it this way, then power failure in the middle still leaves you with your old file intact. You'll always have the complete old file around until the complete new file is ready.
By the way, you should make use of a BufferedInputStream too, rather than just using a raw FileInputStream. And I can't see an inputStream.close() anywhere.
I have a write method that will write a byte[] to disk. On very few devices I'm running into some strange problems where the written file.length() != byte[].length after a successful write operation.
Code and Problem
The code to write a file to disk
private static boolean writeByteFile(File file, byte[] byteData) throws IOException {
if (!file.exists()) {
boolean fileCreated = file.createNewFile();
if (!fileCreated) {
return false;
}
}
FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
bos.write(byteData);
bos.flush();
fos.getFD().sync(); // sync to disk as recommended: http://android-developers.blogspot.com/2010/12/saving-data-safely.html
fos.close();
if (file.length() != byteData.length) {
final byte[] originalMD5Hash = md.digest(byteData);
InputStream is = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(is);
byte[] buffer = new byte[4096];
while(bis.read(buffer) > -1) {
md.update(buffer);
}
is.close();
final byte[] writtenFileMD5Hash = md.digest();
if(!Arrays.equals(originalMD5Hash, writtenFileMD5Hash)) {
String message = String.format(
"After an fsync, the file's length is not equal to the number of bytes we wrote!\npath=%s, expected=%d, actual=%d. >> " +
"Original MD5 Hash: %s, written file MD5 hash: %s",
file.getAbsolutePath(), byteData.length, file.length(),
digestToHex(originalMD5Hash), digestToHex(writtenFileMD5Hash));
throw new GiantWtfException(message);
}
}
return true;
}
I'm running into the if-statement where I compare file length on a few devices.
One example output:
After an fsync, the file's length is not equal to the number of bytes we wrote! path=/mnt/sdcard/.folder/filename, expected=233510, actual=229376 >> Original MD5 Hash: f1d298c0484672c52d9c26d04a3a21dc, written file MD5 hash: ab30660bd2b476d9551c15b340207a8a
I currently see this problem on 5 devices as I'm slowly rolling out the code. Some device data:
Question
Is there anything else I can do or improve?
More stats and observations
Current system version
2.3.5
2.3.6
Model
N860 (LG)
GT-I9100G (Samsung)
GT-S5300 (Samsung)
GT-S7500 (Samsung)
LG-VS410PP (LG)
Other stats
In the general crash analytics (from Crittercism) there is always more then enough free disk space at the time the problem happens. Still some (not all) of the devices have thrown IOExceptions around no free disk space at a different point in time.
As always I've never been able to reproduce that problem on any test phone I have.
Assumptions / Observations:
Generally I would expect a IOException when the disk is full. Still all the exceptions that I catch have less bytes written then they should have.
Interestingly enough all the number of bytes that actually have been written to disk are a multiple of 2^15.
EDIT:
I added a MD5 check sum validation that also fails and simplified the example code a little for better readability. It still fails in the wild with different MD5 hashes.
philipp, file.length() is the file size as reported by the OS. It might be the space the file takes up on disk or the number of bytes in the file.
If the number returned is size on disk, it is related to the number of clusters that hold the file. For example NTFS generally uses 4KB clusters. If you save a text document with 3 ascii encoded characters in on an NTFS formatted volume, the size of the file is 3 bytes, the size of the file on disk is 4096 bytes. On NTFS with a 4KB cluster all files are a multiple of 4096 bytes on disk. See http://en.wikipedia.org/wiki/Data_cluster for more.
If the number returned is the length of the file in bytes (from the underlying file-system's meta-data) then you should have an exact match to how many bytes you wrote, though I wouldn't bet my life on it.
Android uses YAFFS or EXT4, if that helps at all.
I strongly agree with admdrew, use a hash. MD5 would work great. SHA or even CRC should work fine for this task. As you write bytes to the disk, feed the stream to your hash algorithm as well. Once the file is written, read it back and feed that to your hasher. Compare the results. If you want to be sure the data is clean, file size is not enough.
I need to create a zip file and am limited by the following conditions:
The entries come as byte[] (or ByteArrayOutputStream) and not as File.
The filenames for the entry can be non-ascii/UTF-8.
JDK 1.6 or earlier
Since java.util.zip only supports UTF-8 filenames from JDK 1.7 and onward, it seems better to use commons-compress ZipArchiveOutputStream. But how to create a ZipEntryArchive based on a byte array or ByteArrayOutputStream rather than a File?
Thank you!
The following method takes a byte[] as input, produces a zip and returns its content as another byte[]. All is done in memory. No IO operations on the disk. I stripped exception handling for a better overview.
byte[] zip(byte[] data, String filename) {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ZipArchiveOutputStream zos = new ZipArchiveOutputStream(bos);
ZipArchiveEntry entry = new ZipArchiveEntry(filename);
entry.setSize(data.length);
zos.putArchiveEntry(entry);
zos.write(data);
zos.closeArchiveEntry();
zos.close();
bos.close();
return bos.toByteArray();
}
Does this solve your problem?
I have an encryption algorithm (AES) that accepts a file converted to array byte and encrypt it.
Since I am going to process a very large files, the JVM may go out of memory.
I am planing to read the files in multiple byte arrays, each containing some part of the file. Then I iteratively feed the algorithm. Finally, I merge them to produce an encrypted file.
So my question is: Is there any way to read a file part by part to multiple byte arrays?
I thought I could use the following to read the file to a byte array:
IOUtils.toByteArray(InputStream input).
And then split the array into multiple bytes using:
Arrays.copyOfRange()
But I am afraid that the code that reads a file to ByteArray will make the JVM to go out of memory.
Look up cipher streams in Java. You can use them to encrypt/decrypt streams on the fly so you don't have to store the whole thing in memory. All you have to do is copy the regular FileInputStream for your source file to the CipherOutputStream that's wrapping your FileOutputStream for the encrypted sink file. IOUtils even conveniently contains a copy(InputStream, OutputStream) method to do this copy for you.
For example:
public static void main(String[] args) {
encryptFile("exampleInput.txt", "exampleOutput.txt");
}
public static void encryptFile(String source, String sink) {
FileInputStream fis = null;
try {
fis = new FileInputStream(source);
CipherOutputStream cos = null;
try {
cos = new CipherOutputStream(new FileOutputStream(sink), getEncryptionCipher());
IOUtils.copy(fis, cos);
} finally {
if (cos != null)
cos.close();
}
} finally {
if (fis != null)
fis.close();
}
}
private static Cipher getEncryptionCipher() {
// Create AES cipher with whatever padding and other properties you want
Cipher cipher = ... ;
// Create AES secret key
Key key = ... ;
cipher.init(Cipher.ENCRYPT_MODE, key);
}
If you need to know the number of bytes that were copied, you can use IOUtils.copyLarge instead of IOUtils.copy if the file sizes exceed Integer.MAX_VALUE bytes (2 GB).
To decrypt the file, do the same thing, but use CipherInputStream instead ofCipherOutputStream and initialize your Cipher using Cipher.DECRYPT_MODE.
Take a look here for more info on cipher streams in Java.
This will save you space because you won't need to store byte arrays of your own anymore. The only stored byte[] in this system is the internal byte[] of the Cipher, which will get cleared each time enough input is entered and an encrypted block is returned by Cipher.update, or on Cipher.doFinal when the CipherOutputStream is closed. However, you don't have to worry about any of this since it's all internal and everything is managed for you.
Edit: note that this can result in certain encryption exceptions being ignored, particularly BadPaddingException and IllegalBlockSizeException. This behavior can be found in the CipherOutputStream source code. (Granted, this source is from the OpenJDK, but it probably does the same thing in the Sun JDK.) Also, from the CipherOutputStream javadocs:
This class adheres strictly to the semantics, especially the failure semantics, of its ancestor classes java.io.OutputStream and java.io.FilterOutputStream. This class has exactly those methods specified in its ancestor classes, and overrides them all. Moreover, this class catches all exceptions that are not thrown by its ancestor classes.
The bolded line here implies that the cryptographic exceptions are ignored, which they are. This may cause some unexpected behavior while trying to read an encrypted file, especially for block and/or padding encryption algorithms like AES. Make a mental note of this that you will get zero or partial output for the encrypted (or decrypted for CipherInputStream) file.
If you're using IOUtils, perhaps you should consider IOUtils.copyLarge()
public static long copyLarge(InputStream input,
OutputStream output,
long inputOffset,
long length)
and specify a ByteArrayOutputStream as the output. You can then iterate through and load sections of your file using offset/length.
From the doc:
Copy some or all bytes from a large (over 2GB) InputStream to an
OutputStream, optionally skipping input bytes.
This problem seems to happen inconsistently. We are using a java applet to download a file from our site, which we store temporarily on the client's machine.
Here is the code that we are using to save the file:
URL targetUrl = new URL(urlForFile);
InputStream content = (InputStream)targetUrl.getContent();
BufferedInputStream buffered = new BufferedInputStream(content);
File savedFile = File.createTempFile("temp",".dat");
FileOutputStream fos = new FileOutputStream(savedFile);
int letter;
while((letter = buffered.read()) != -1)
fos.write(letter);
fos.close();
Later, I try to access that file by using:
ObjectInputStream keyInStream = new ObjectInputStream(new FileInputStream(savedFile));
Most of the time it works without a problem, but every once in a while we get the error:
java.io.StreamCorruptedException: invalid stream header: 0D0A0D0A
which makes me believe that it isn't saving the file correctly.
I'm guessing that the operations you've done with getContent and BufferedInputStream have treated the file like an ascii file which has converted newlines or carriage returns into carriage return + newline (0x0d0a), which has confused ObjectInputStream (which expects serialized data objects.
If you are using an FTP URL, the transfer may be occurring in ASCII mode.
Try appending ";type=I" to the end of your URL.
Why are you using ObjectInputStream to read it?
As per the javadoc:
An ObjectInputStream deserializes primitive data and objects previously written using an ObjectOutputStream.
Probably the error comes from the fact you didn't write it with ObjectOutputStream.
Try reading it wit FileInputStream only.
Here's a sample for binary ( although not the most efficient way )
Here's another used for text files.
There are 3 big problems in your sample code:
You're not just treating the input as bytes
You're needlessly pulling the entire object into memory at once
You're doing multiple method calls for every single byte read and written -- use the array based read/write!
Here's a redo:
URL targetUrl = new URL(urlForFile);
InputStream is = targetUrl.getInputStream();
File savedFile = File.createTempFile("temp",".dat");
FileOutputStream fos = new FileOutputStream(savedFile);
int count;
byte[] buff = new byte[16 * 1024];
while((count = is.read(buff)) != -1) {
fos.write(buff, 0, count);
}
fos.close();
content.close();
You could also step back from the code and check to see if the file on your client is the same as the file on the server. If you get both files on an XP machine, you should be able to use the FC utility to do a compare (check FC's help if you need to run this as a binary compare as there is a switch for that). If you're on Unix, I don't know the file compare program, but I'm sure there's something.
If the files are identical, then you're looking at a problem with the code that reads the file.
If the files are not identical, focus on the code that writes your file.
Good luck!