I have an android app which handles some large byte array but I am getting some OOM crash in my Firebase Crashlytics reports for devices with low memory while handling byte array whose size may go from 10 mb to 50mb. Below is my method that I have used. So could anyone help me to improve it to avoid OOM.
byte[] decrypt(File files) {
try {
FileInputStream fis = new FileInputStream(files);
SecretKeySpec sks = new SecretKeySpec(getResources().getString(R.string.encryptPassword).getBytes(),
"AES");
Cipher cipher = Cipher.getInstance("AES");
cipher.init(Cipher.DECRYPT_MODE, sks);
CipherInputStream cis = new CipherInputStream(fis, cipher);
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int b;
byte[] d = new byte[1024];
while ((b = cis.read(d)) != -1) {
buffer.write(d, 0, b); //this is one of the line which is being referred for the OOM in firebase
}
byte[] decryptedData = buffer.toByteArray();//this is the line which is being referred for the OOM in firebase
buffer.flush();
fis.close();
cis.close();
return decryptedData;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
EDIT
Actually I am using the above method for decrypting downloaded audio files which are encrypted during downloading.
The above methods return the content of the encrypted files to exoplayer to play its content and I am calling the above method in the following way
ByteArrayDataSource src= new ByteArrayDataSource(decrypt(some_file));
Uri uri = new UriByteDataHelper().getUri(decrypt(some_file));
DataSpec dataSpec = new DataSpec(uri);
src.open(dataSpec);
DataSource.Factory factory = new DataSource.Factory()
{
#Override public DataSource createDataSource()
{
return src;
}
};
audioSource = new ProgressiveMediaSource.Factory(factory).createMediaSource(uri);
First of all, I would make sure that the devices where this is running have enought heap memory to run this, it might be that simply the software has already been allocated with a lot of space and there might not be much more left on the heap to provide the software. This operation should not require much memory and I don't see anything obvious that would point towards trying to allocated and unexpectadely large amount of memory.
What I would recommend though, if you want to hav a quick test is actually simply lowering the byte array size, any particular reason why you are using 1024?
If possible perhaps try:
byte[] d = new byte[8];
Also, If that was me, I would store the read data temporarily, on an array perhaps and only once the read of the cypher has finished, I would call
buffer.write()
From my experience, trying to read and write at the same time tens to not be advised and can result in serveral issues, at the very least you should make sure you have the whole cypher and that it is a valid one (if you have some validation requirements) and only then send it.
Again, this should not be the core issue, the device seems that it is lacking enough availalbe memory to be allocated, perhaps too much reserved memory for other processes?
You should consider of writing the decrypted data to a tempfile and then reload the data for usage.
The main reasons for the Out of memory error are the ByteArrayOutputStream AND
byte[] decryptedData = buffer.toByteArray(), because both of them hold the complete (decrypted) data and that doubles the memory consumption of your decrypt method.
You could avoid this by decrypting the data to a tempfile in the first step and later load the data from the tempfile. I modified the decrypt method to handle the decrypted output stream and later there is a method for reloading the decrypted data (there is no propper exception handling and for my testing I setup a static encryptPassword-variable ...).
There is just one part left for you - you need to find a good place for the tempfile and I'm no Android specialist.
Just two notes: You are using the unsecure AES ECB mode and the String to byte[]conversion for your password should be
changed to
.getBytes(StandardCharsets.UTF_8)
on ecryption and decryption side to avoid errors caused by different encodings on different platforms.
public static void decryptNew(File files, File tempfiles) {
try (FileInputStream fis = new FileInputStream(files);
BufferedInputStream in = new BufferedInputStream(fis);
FileOutputStream out = new FileOutputStream(tempfiles);
BufferedOutputStream bos = new BufferedOutputStream(out)) {
byte[] ibuf = new byte[1024];
int len;
Cipher cipher = Cipher.getInstance("AES");
SecretKeySpec sks = new SecretKeySpec(encryptPassword.getBytes(),"AES"); // static password
// SecretKeySpec sks = new SecretKeySpec(getResources().getString(R.string.encryptPassword).getBytes(),"AES");
cipher.init(Cipher.DECRYPT_MODE, sks);
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
bos.write(obuf);
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
bos.write(obuf);
} catch (BadPaddingException | IllegalBlockSizeException | InvalidKeyException | IOException | NoSuchAlgorithmException | NoSuchPaddingException e) {
e.printStackTrace();
}
}
public static byte[] loadFile(File filename) throws IOException {
byte[] filecontent = new byte[0];
FileInputStream fileInputStream = null;
try {
fileInputStream = new FileInputStream(filename);
// int byteLength = fff.length();
// In android the result of file.length() is long
long byteLength = filename.length(); // byte count of the file-content
filecontent = new byte[(int) byteLength];
fileInputStream.read(filecontent, 0, (int) byteLength);
} catch (IOException e) {
e.printStackTrace();
fileInputStream.close();
return filecontent;
}
fileInputStream.close();
return filecontent;
}
After loading the tempfile content to the byte array you can delete the file with a one-liner (again no exception handling):
Files.deleteIfExists(tempFile.toPath());
I'm writing a second answer and do not edit my first answer as it is a total different approach to solve the problem.
As you post a part of your code I can see that you have a byte array with the complete and decrypted content that gets played by the exoplayer:
output:
byte[] decrypt(File files)
as input for
ByteArrayDataSource src= new ByteArrayDataSource(decrypt(some_file));
So to avoid double and moretime memory consumption when playing with large files (approx. 50mb) my approach it is to download the complete encrypted file and save it in a byte array.
On devices with a good memory equipment you can decrypt the encrypted byte array in one run to another byte array and play the music from this decrypted byte array (step 6 + 8 in my sample program).
Using a low memory device you decrypt the byte array in chunks (in my program in 16 byte long blocks) and save the decrypted chunks at the same place in the encrypted byte array. When all chunks got processed the (former) encrypted data are now decrypted and you used the memory of just one byte array length. Now you can play the music from this byte array (steps 7 + 8).
Just for explanation, steps 1-3 are on server side and in steps 3+4 the transmission takes place.
This example uses the AES CTR mode as it provides the same length for input and output data.
In the end I'm comparing the byte arrays to prove that the decryption was successful for steps 6 (direct decryption) and 7 (decryption in chunks):
output:
Decrypting 50mb data in chunks to avoid out of memory error
https://stackoverflow.com/questions/62412705/how-to-write-byte-array-without-oom
plaindata length: 52428810
cipherdata length: 52428810
decrypteddata length: 52428810
cipherdata parts in 16 byte long parts: 3276800 = rounds for decryption
cipherdata moduluo 16 byte long parts: 10 + 1 round for rest/modulus
cipherdata length: 52428810 (after decryption)
plaindata equals decrypteddata: true
plaindata equals cipherdata: true
code:
import javax.crypto.Cipher;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
import java.util.Arrays;
import java.util.Random;
public class EncryptionCtrSo4 {
public static void main(String[] args) throws GeneralSecurityException {
System.out.println("Decrypting 50mb data in chunks to avoid out of memory error");
System.out.println("https://stackoverflow.com/questions/62412705/how-to-write-byte-array-without-oom");
/*
* author michael fehr, http://javacrypto.bplaced.net
* no licence applies, no warranty
*/
// === server side ===
// 1 create a 50 mb byte array (unencrypted)
byte[] plaindata = new byte[(50 * 1024 * 1024 + 10)];
// fill array with (random) data
Random random = new Random();
random.nextBytes(plaindata);
// 2 encrypt the data with aes ctr mode, create random keys
SecureRandom secureRandom = new SecureRandom();
byte[] key = new byte[32]; // 32 byte = 256 bit aes key
secureRandom.nextBytes(key);
byte[] iv = new byte[16]; // 16 byte = 128 bit
secureRandom.nextBytes(iv);
SecretKeySpec keySpecEnc = new SecretKeySpec(key, "AES");
IvParameterSpec ivParameterSpecEnc = new IvParameterSpec(iv);
Cipher cipherEnc = Cipher.getInstance("AES/CTR/NoPadding");
cipherEnc.init(Cipher.ENCRYPT_MODE, keySpecEnc, ivParameterSpecEnc);
byte[] cipherdata = cipherEnc.doFinal(plaindata);
System.out.println("plaindata length: " + plaindata.length);
System.out.println("cipherdata length: " + cipherdata.length);
// 3 transfer the cipherdata to app
// ...
// === app side ===
// 4 receive encrypted data from server
// ...
// 5 decryption setup
SecretKeySpec keySpecDec = new SecretKeySpec(key, "AES");
IvParameterSpec ivParameterSpecDec = new IvParameterSpec(iv);
Cipher cipherDec = Cipher.getInstance("AES/CTR/NoPadding");
cipherDec.init(Cipher.DECRYPT_MODE, keySpecDec, ivParameterSpecDec);
// 6 decryption in one run on high memory devices
byte[] decrypteddata = cipherDec.doFinal(cipherdata);
System.out.println("decrypteddata length: " + decrypteddata.length);
// 7 decryption in chunks using the cipherdata byte array
int cipherdataLength = cipherdata.length;
int chunksize = 16; // should be a multiple of 16, minimum 16
byte[] decryptedPart = new byte[chunksize];
int parts16byteDiv = cipherdataLength / chunksize;
int parts16byteMod = cipherdataLength % chunksize;
System.out.println("cipherdata parts in " + chunksize + " byte long parts: " + parts16byteDiv + " = rounds for decryption");
System.out.println("cipherdata moduluo " + chunksize + " byte long parts: " + parts16byteMod + " + 1 round for rest/modulus");
for (int i = 0; i < parts16byteDiv; i++) {
cipherDec.update(cipherdata, (i * chunksize), chunksize, decryptedPart);
System.arraycopy(decryptedPart, 0, cipherdata, (i * chunksize), decryptedPart.length);
}
if (parts16byteMod > 0) {
decryptedPart = new byte[parts16byteMod];
cipherDec.update(cipherdata, (parts16byteDiv * chunksize), parts16byteMod, decryptedPart);
System.arraycopy(decryptedPart, 0, cipherdata, (parts16byteDiv * chunksize), decryptedPart.length);
}
System.out.println("cipherdata length: " + cipherdata.length + " (after decryption)");
// the cipherdata byte array is now decrypted !
// 8 use cipherdata (encrypted) or decrypteddata as input for exoplayer
// compare ciphertext with decrypteddata in step 6
System.out.println("plaindata equals decrypteddata: " + Arrays.equals(plaindata, decrypteddata));
// check that (decrypted) cipherdata equals plaindata of step 7
System.out.println("plaindata equals cipherdata: " + Arrays.equals(plaindata, cipherdata));
}
}
Related
I was trying to implement an encrypt/decrypt program using classes under javax.crypto and file streams for input/output.
To limit the memory usage, I run with -Xmx256m parameter.
It works fine with encryption and decryption with smaller files.
But when decrypt a huge file (1G in size), there is an out of memory exception:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at com.sun.crypto.provider.GaloisCounterMode.decrypt(GaloisCounterMode.java:505)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:782)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:667)
at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at javax.crypto.Cipher.update(Cipher.java:1831)
at javax.crypto.CipherOutputStream.write(CipherOutputStream.java:166)
Here is the decrypt code:
private final int _readSize = 0x10000;//64k
...
GCMParameterSpec gcmParameterSpec = new GCMParameterSpec(gcmTagSize, iv);
Key keySpec = new SecretKeySpec(key, keyParts[0]);
Cipher decCipher = Cipher.getInstance("AES/GCM/PKCS5Padding");
decCipher.init(Cipher.DECRYPT_MODE, keySpec, gcmParameterSpec);
try (InputStream fileInStream = Files.newInputStream(inputEncryptedFile);
OutputStream fileOutStream = Files.newOutputStream(outputDecryptedFile)) {
try (CipherOutputStream cipherOutputStream = new CipherOutputStream(fileOutStream, decCipher)) {
long count = 0L;
byte[] buffer = new byte[_readSize];
int n;
for (; (n = fileInStream.read(buffer)) != -1; count += (long) n) {
cipherOutputStream.write(buffer, 0, n);
}
}
}
The key parameters like gcmTagSize and iv are read from a key file, and it works fine with smaller files, like some one around size of 50M.
As I understand, every time there are only 64k data passed to decipher, why it runs out of heap memory?
How can I avoid this?
Edit:
Actually I have tried with 4k as buffer size, failed with same exception.
Edit 2:
With more testing, the max file size it can handle is around 1/4 of the heap size. Like, if you set -Xmx256m, files bigger than 64M will fail to decrypt.
This appears to be an issue with the implementation of the GCM mode. I'm not sure that you can work-around it.
If you look at your stack trace:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at com.sun.crypto.provider.GaloisCounterMode.decrypt(GaloisCounterMode.java:505)
The out-of-memory error is happening when writing to a ByteArrayOutputStream from within GaloisCounterMode. You use a FileOutputStream, so either you're not showing the right code or this ByteArrayStream is used internally.
If you look at the source for GaloisCounterMode you'll see that it defines an internal ByteArrayOutputStream (it actually defines two, but I think this is the one that's the problem):
// buffer for storing input in decryption, not used for encryption
private ByteArrayOutputStream ibuffer = null;
Then, later down, it writes bytes to this stream. Note the code comment.
int decrypt(byte[] in, int inOfs, int len, byte[] out, int outOfs) {
processAAD();
if (len > 0) {
// store internally until decryptFinal is called because
// spec mentioned that only return recovered data after tag
// is successfully verified
ibuffer.write(in, inOfs, len);
}
return 0;
}
That buffer isn't reset until decryptFinal().
Edit: looking at this CSx answer it looks like GCM needs to buffer the entire stream. That would make it a very bad choice if you have large files and not enough memory.
I think your best solution is to switch to CBC mode.
The bad news are: IMHO the error is caused by an bad implementation of AES GCM-mode in native Java. Even if you could get it to work you will find that the decryption of a large file (1 GB or so) will take a lot of time (maybe hours ?).
But there are good news: you could/should use BouncyCastle as service provider for your decryption task - that way the decryption will work and it's much faster.
The following full example will create a sample file of 1 gb size, encrypts it with BouncyCastle and later decrypts it. In the end there is a file compare to show that plain and decrypted file contents are equal and the files will be deleted. You need temporary a total of more than 3 GB free space on your device to run this example.
Using a buffer of 64 KB I'm running this example with this data:
Milliseconds for Encryption: 14295 | Decryption: 16249
A buffer of 1 KB is a little bit slower on encryption side but much slower on decryption task:
Milliseconds for Encryption: 15250 | Decryption: 21952
A last word regarding your cipher - "AES/GCM/PKCS5Padding" is not existing and "available" in some implementations but the real used algorithm is "AES/GCM/NoPadding" (see Can PKCS5Padding be in AES/GCM mode? for more details).
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.nio.file.Files;
import java.security.*;
import java.util.Arrays;
public class GcmTestBouncyCastle {
public static void main(String[] args) throws IOException, NoSuchPaddingException, InvalidAlgorithmParameterException,
NoSuchAlgorithmException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException, InvalidKeyException {
System.out.println("Encryption & Decryption with BouncyCastle AES-GCM-Mode");
System.out.println("https://stackoverflow.com/questions/61792534/out-of-memory-exception-when-decrypt-large-file-using-cipher");
// you need bouncy castle, get version 1.65 here:
// https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on/1.65
Security.addProvider(new BouncyCastleProvider());
// setup files
// filenames
String filenamePlain = "plain.dat";
String filenameEncrypt = "encrypt.dat";
String filenameDecrypt = "decrypt.dat";
// generate a testfile of 1024 byte | 1 gb
//createFileWithDefinedLength(filenamePlain, 1024);
createFileWithDefinedLength(filenamePlain, 1024 * 1024 * 1024); // 1 gb
// time measurement
long startMilli = 0;
long encryptionMilli = 0;
long decryptionMilli = 0;
// generate nonce/iv
int GCM_NONCE_LENGTH = 12; // for a nonce of 96 bit length
int GCM_TAG_LENGTH = 16;
int GCM_KEY_LENGTH = 32; // 32 = 256 bit keylength, 16 = 128 bit keylength
SecureRandom r = new SecureRandom();
byte[] nonce = new byte[GCM_NONCE_LENGTH];
r.nextBytes(nonce);
// key should be generated as random byte[]
byte[] key = new byte[GCM_KEY_LENGTH];
r.nextBytes(key);
// encrypt file
startMilli = System.currentTimeMillis();
encryptWithGcmBc(filenamePlain, filenameEncrypt, key, nonce, GCM_TAG_LENGTH);
encryptionMilli = System.currentTimeMillis() - startMilli;
startMilli = System.currentTimeMillis();
decryptWithGcmBc(filenameEncrypt, filenameDecrypt, key, nonce, GCM_TAG_LENGTH);
decryptionMilli = System.currentTimeMillis() - startMilli;
// check that plain and decrypted files are equal
System.out.println("SHA256-file compare " + filenamePlain + " | " + filenameDecrypt + " : "
+ Arrays.equals(sha256File(filenamePlain), sha256File(filenameDecrypt)));
System.out.println("Milliseconds for Encryption: " + encryptionMilli + " | Decryption: " + decryptionMilli);
// clean up with files
Files.deleteIfExists(new File(filenamePlain).toPath());
Files.deleteIfExists(new File(filenameEncrypt).toPath());
Files.deleteIfExists(new File(filenameDecrypt).toPath());
}
public static void encryptWithGcmBc(String filenamePlain, String filenameEnc, byte[] key, byte[] nonce, int gcm_tag_length)
throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException,
InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException {
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
SecretKeySpec keySpec = new SecretKeySpec(key, "AES");
GCMParameterSpec gcmSpec = new GCMParameterSpec(gcm_tag_length * 8, nonce);
cipher.init(Cipher.ENCRYPT_MODE, keySpec, gcmSpec);
try (FileInputStream fis = new FileInputStream(filenamePlain);
BufferedInputStream in = new BufferedInputStream(fis);
FileOutputStream out = new FileOutputStream(filenameEnc);
BufferedOutputStream bos = new BufferedOutputStream(out)) {
//byte[] ibuf = new byte[1024];
byte[] ibuf = new byte[0x10000]; // = 65536
int len;
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
bos.write(obuf);
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
bos.write(obuf);
}
}
public static void decryptWithGcmBc(String filenameEnc, String filenameDec, byte[] key, byte[] nonce, int gcm_tag_length)
throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException,
InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException {
try (FileInputStream in = new FileInputStream(filenameEnc);
FileOutputStream out = new FileOutputStream(filenameDec)) {
//byte[] ibuf = new byte[1024];
byte[] ibuf = new byte[0x10000]; // = 65536
int len;
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
SecretKeySpec keySpec = new SecretKeySpec(key, "AES");
GCMParameterSpec gcmSpec = new GCMParameterSpec(gcm_tag_length * 8, nonce);
cipher.init(Cipher.DECRYPT_MODE, keySpec, gcmSpec);
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
out.write(obuf);
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
out.write(obuf);
}
}
// just for creating a large file within seconds
private static void createFileWithDefinedLength(String filenameString, long sizeLong) throws IOException {
RandomAccessFile raf = new RandomAccessFile(filenameString, "rw");
try {
raf.setLength(sizeLong);
} finally {
raf.close();
}
}
// just for file comparing
public static byte[] sha256File(String filenameString) throws IOException, NoSuchAlgorithmException {
byte[] buffer = new byte[8192];
int count;
MessageDigest md = MessageDigest.getInstance("SHA-256");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(filenameString));
while ((count = bis.read(buffer)) > 0) {
md.update(buffer, 0, count);
}
bis.close();
return md.digest();
}
}
We implemented chunked file decryption in a way that the initialization vector is added at the beginning of the file and followed by the encrypted data. The following decryption method decrypts the file and write:
private void decrypt_AES_CBC_PKCS7(final byte[] symKeyBytes, final FileInputStream inputStream, final FileOutputStream outputStream) throws Exception {
Security.addProvider(new BouncyCastleProvider());
// Read init vector
final byte[] iv = new byte[16];
inputStream.read(iv, 0, 16);
// Prepare for decryption
final SecretKeySpec secretKeySpec = new SecretKeySpec(symKeyBytes, "AES");
final Cipher cipher = Cipher.getInstance("AES/CBC/PKCS7Padding", new BouncyCastleProvider());
cipher.init(Cipher.DECRYPT_MODE, secretKeySpec, new IvParameterSpec(iv));
// Decrypt chunk by chunk
int chunkLen = 0;
final byte[] buffer = new byte[CHUNK_SIZE_DECRTPY]; // CHUNK_SIZE_DECRTPY = 20 * 1024 * 1024;
while ((chunkLen = inputStream.read(buffer)) > 0) {
byte[] decrypted = cipher.doFinal(buffer, 0, chunkLen);
outputStream.write(decrypted, 0, decrypted.length);
}
// close streams
inputStream.close();
outputStream.close();
}
The code worked fine in earlier Android versions (L & M), but when we tried it on Nexus 5X with Android N, 16 "junk" bytes were inserted at the begginning of the resulting file. This happens only in files that consist of one chunk only, i.e., files with size larger than one chunk would be decrypted correctly and no extra bytes would be prepended. Interestingly, when the code is run with Android studio debugger attached, with a breakpoint between reading IV and reading input stream, the decryption works fine and no extra bytes appear in the output file.
Example encrypted file (IV is visible as first 16 bytes, i.e., first row):
Example encrypted file
Example decrypted file, first 16 bytes are added only in Android N:
Example decrypted file
All suggestions are welcome!
As suggested by #martijno, I checked the result of inputStream.read(iv, 0, 16) and it turned out that the first invocation of it returns -1 in Android N. So, I modified the part of the code that reads the IV:
final byte[] iv = new byte[16];
int result;
int numTries = 0;
int MAX_NUM_TRIES = 2;
do {
result = inputStream.read(iv, 0, 16);
++numTries;
} while (result == -1 && numTries < MAX_NUM_TRIES);
if (result == -1) {
throw new Exception("Failed to read IV from file!");
}
It turns out that two tries for reading IV are enough, we'll test this further when more Android N devices become available.
I have done Encryption and Decryption in android when file downloading
but I want to improve time performance when file decrypted.
My problem is when I am downloading any file so I have add encryption over there but at this stage I am showing Progress loader so it looks good but but when file completely download and try to open that file then it is decrypted that file this time it's taking too much time .
which is look very bad. How can I reduce decryption time? Here is my code
Encryption Code
byte data[] = new byte[1024];
String seed = "password";
byte[] rawKey = getRawKey(seed.getBytes());
SecretKeySpec skeySpec = new SecretKeySpec(rawKey, "AES");
Cipher cipher = Cipher.getInstance("AES");
cipher.init(Cipher.ENCRYPT_MODE, skeySpec);
output = new CipherOutputStream(output, cipher);
long total = 0;
while ((count = input.read(data)) != -1) {
total += count;
publishProgress("" + (int) ((total * 100) / lenghtOfFile));
output.write(data, 0, count);
}
Decryption Code Here:
String newPath = sdCardPath + "/" + dPdfName;
File f1 = new File(newPath);
if (!f1.exists())
try {
f1.createNewFile();
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
try {
InputStream fis = new FileInputStream(f);
OutputStream fos = new FileOutputStream(f1);
String seed = "password";
byte[] rawKey = getRawKey(seed.getBytes());
SecretKeySpec skeySpec = new SecretKeySpec(rawKey,
"AES");
Cipher cipher = Cipher.getInstance("AES");
cipher.init(Cipher.DECRYPT_MODE, skeySpec);
fis = new CipherInputStream(fis, cipher);
int b;
byte[] data = new byte[4096];
while ((b = fis.read(data)) != -1) {
// fos.write(cipher.doFinal(data), 0, b);
fos.write(data, 0, b);
}
fos.flush();
fos.close();
fis.close();
} catch (Exception e) {
// TODO: handle exceptionpri
e.printStackTrace();
}
Get Row Key Method:
private static byte[] getRawKey(byte[] seed) throws Exception {
KeyGenerator kgen = KeyGenerator.getInstance("AES");
SecureRandom sr = SecureRandom.getInstance("SHA1PRNG");
sr.setSeed(seed);
kgen.init(128, sr);
SecretKey skey = kgen.generateKey();
byte[] raw = skey.getEncoded();
return raw;
}
Suggestions:
1) move the decryption to another asynctask and add another progress indicator.
2) reserve last, say, 10% of the progress indicator to decryption. This is what I actually did once, but I was doing an integrity check (against an MD5 hash IIRC), not decryption.
3) move the decryption to the downloading asynctask, decrypt each received portion of data immediately and so hide the decryption time behind the download time.
4) not sure this will be any faster, but you may have two service threads: one downloading file and another decrypting it. It's better no to use AsyncTask here, because they may behave differently on different versions of Android (including sequential execution on a single thread, see Is AsyncTask really conceptually flawed or am I just missing something? for the discussion, and my note https://stackoverflow.com/a/14602486/755804 )
Note also that the thread responsible for downloading and decryption belongs to Model (in MVC sense) and must not be owned by an Activity which is a Controller that cannot outlive a screen turn: https://stackoverflow.com/a/14603375/755804
If your download takes a long time, you may be interested in resuming interrupted downloads, and it's better to think about it from the very beginning. It's always easier to modify simple solutions, and multi-threaded solutions are rather complex. If you transfer a number of files, it may happen that one of them gets broken during transfer, and you may want an option to retransfer only the file(s) that are broken. You may also want an integrity check.
Make sure you align the data buffer to the block size of the encryption.
For a Example see: https://stackoverflow.com/a/33171612/475496
Using this method has speedup our encryption enormous.
I develop an android application for encrypt files on the phone. By searching, i found this topic : How to encrypt file from SD card using AES in Android?
The method works fine but it is very slow to encrypt files...
At this line : byte[] d = new byte[8]; why only 8 bytes ? can't we set an higher value ?
Also, do you know a way to encrypt files fastly ? I heard of crypto++ for native code implementation but how can I implement JNI on my application ?
Thank you,
EDIT : Encryption function
public void encrypt (String RSAPrivateKey, String Key, byte[] iv, String zipname, ZipEncryptAsyncTask task) throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException, InvalidAlgorithmParameterException
{
FileInputStream fis = new FileInputStream (zipname + ".temp");
FileOutputStream fos = new FileOutputStream (zipname);
SecretKeySpec sks = new SecretKeySpec (Base64.decode(Key, Base64.DEFAULT), "AES");
IvParameterSpec ivspec = new IvParameterSpec (iv);
Cipher cipher = Cipher.getInstance ("AES/CBC/PKCS5Padding");
fos.write(String.valueOf(RSAPrivateKey.getBytes().length).getBytes());
fos.write(RSAPrivateKey.getBytes());
cipher.init(Cipher.ENCRYPT_MODE, sks, ivspec);
CipherOutputStream cos = new CipherOutputStream(fos, cipher);
long size = 0;
byte[] d = new byte[8];
for(int b; (b = fis.read(d)) != -1; )
{
cos.write(d, 0, b);
task.doProgress((size += 8));
}
cos.flush();
cos.close();
fis.close();
new File(zipname + ".temp").delete();
}
As an alternative, you could consider changing the cypher mode you are using. CBC mode must be used serially, block by block. Counter mode (CTR) can be run in parallel with different blocks being encrypted simultaneously. Of course there are overheads to parallel processing, and you will need to rework your code, but if buffer resizing does not give you enough speed gains, then it might be the next option to try.
as #CodesInChaos said, a way to do this, is to increase the size of the buffer.
Now I do a benchmark which determines the best size for the buffer and I use the optimal value.
So I have these large files (6GB+) that I need to decrypt on a 32 bit computer. The general procedure that I used previously was to read the entire file in memory, then pass it on to the decrypt function and then write it all back to a file. This doesn't really work due to memory limitations. I did try passing the file in parts to the decrypt function but it seems to mess up around the boundaries of where I break up the file before sending it to the decrypt function.
I've tried breaking up the file in parts relative to key size but that doesnt seem to matter. I tried a byte array of size 2048 as well as a byte aray of size 294 thinking that might be the special boundary but, no luck. I can see parts of the file correctly decrypted but parts which are total gibberish.
Is it just NOT POSSIBLE to decrypt the file in chunks? If there is a way, then how?
Here is my decryption function / my attempt to decrypt in parts.
private Path outFile;
private void decryptFile(FileInputStream fis, byte[] initVector, byte[] aesKey, long used) {
//Assume used = 0 for this function.
byte[] chunks = new byte[2048]; //If this number is greater than or equal to the size of the file then we are good.
try {
if (outFile.toFile().exists())
outFile.toFile().delete();
outFile.toFile().createNewFile();
FileOutputStream fos = new FileOutputStream(outFile.toFile());
OutputStreamWriter out = new OutputStreamWriter(fos);
IvParameterSpec spec = new IvParameterSpec(Arrays.copyOfRange(initVector, 0, 16));
SecretKeySpec key = new SecretKeySpec(aesKey, "AES");
Cipher cipher = Cipher.getInstance("AES/CFB/NoPadding");
cipher.init(Cipher.DECRYPT_MODE, key, spec);
int x;
while ((x = fis.read(chunks, 0, chunks.length)) != -1) {
byte[] dec = cipher.doFinal(Arrays.copyOfRange(chunks, 0, x));
out.append(new String(dec));
}
out.close();
fos.close();
} catch (Exception e) {
e.printStackTrace();
LOG.error(ExceptionUtils.getStackTrace(e));
}
}
Consider using Cipher#update(byte[], int, int, byte[], int) instead of doFinal() for multipart operations. This will take care of part boundaries for you.
The last part of the deciphered data can be obtained by calling the doFinal(byte[] output, int outputOffset) method.