Develop Reference

How to write byte array without OOM?

I have an android app which handles some large byte array but I am getting some OOM crash in my Firebase Crashlytics reports for devices with low memory while handling byte array whose size may go from 10 mb to 50mb. Below is my method that I have used. So could anyone help me to improve it to avoid OOM.
byte[] decrypt(File files) {
try {
FileInputStream fis = new FileInputStream(files);
SecretKeySpec sks = new SecretKeySpec(getResources().getString(R.string.encryptPassword).getBytes(),
"AES");
Cipher cipher = Cipher.getInstance("AES");
cipher.init(Cipher.DECRYPT_MODE, sks);
CipherInputStream cis = new CipherInputStream(fis, cipher);
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int b;
byte[] d = new byte[1024];
while ((b = cis.read(d)) != -1) {
buffer.write(d, 0, b); //this is one of the line which is being referred for the OOM in firebase
}
byte[] decryptedData = buffer.toByteArray();//this is the line which is being referred for the OOM in firebase
buffer.flush();
fis.close();
cis.close();
return decryptedData;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
EDIT
Actually I am using the above method for decrypting downloaded audio files which are encrypted during downloading.
The above methods return the content of the encrypted files to exoplayer to play its content and I am calling the above method in the following way
ByteArrayDataSource src= new ByteArrayDataSource(decrypt(some_file));
Uri uri = new UriByteDataHelper().getUri(decrypt(some_file));
DataSpec dataSpec = new DataSpec(uri);
src.open(dataSpec);
DataSource.Factory factory = new DataSource.Factory()
{
#Override public DataSource createDataSource()
{
return src;
}
};
audioSource = new ProgressiveMediaSource.Factory(factory).createMediaSource(uri);

First of all, I would make sure that the devices where this is running have enought heap memory to run this, it might be that simply the software has already been allocated with a lot of space and there might not be much more left on the heap to provide the software. This operation should not require much memory and I don't see anything obvious that would point towards trying to allocated and unexpectadely large amount of memory.
What I would recommend though, if you want to hav a quick test is actually simply lowering the byte array size, any particular reason why you are using 1024?
If possible perhaps try:
byte[] d = new byte[8];
Also, If that was me, I would store the read data temporarily, on an array perhaps and only once the read of the cypher has finished, I would call
buffer.write()
From my experience, trying to read and write at the same time tens to not be advised and can result in serveral issues, at the very least you should make sure you have the whole cypher and that it is a valid one (if you have some validation requirements) and only then send it.
Again, this should not be the core issue, the device seems that it is lacking enough availalbe memory to be allocated, perhaps too much reserved memory for other processes?

You should consider of writing the decrypted data to a tempfile and then reload the data for usage.
The main reasons for the Out of memory error are the ByteArrayOutputStream AND
byte[] decryptedData = buffer.toByteArray(), because both of them hold the complete (decrypted) data and that doubles the memory consumption of your decrypt method.
You could avoid this by decrypting the data to a tempfile in the first step and later load the data from the tempfile. I modified the decrypt method to handle the decrypted output stream and later there is a method for reloading the decrypted data (there is no propper exception handling and for my testing I setup a static encryptPassword-variable ...).
There is just one part left for you - you need to find a good place for the tempfile and I'm no Android specialist.
Just two notes: You are using the unsecure AES ECB mode and the String to byte[]conversion for your password should be
changed to
.getBytes(StandardCharsets.UTF_8)
on ecryption and decryption side to avoid errors caused by different encodings on different platforms.
public static void decryptNew(File files, File tempfiles) {
try (FileInputStream fis = new FileInputStream(files);
BufferedInputStream in = new BufferedInputStream(fis);
FileOutputStream out = new FileOutputStream(tempfiles);
BufferedOutputStream bos = new BufferedOutputStream(out)) {
byte[] ibuf = new byte[1024];
int len;
Cipher cipher = Cipher.getInstance("AES");
SecretKeySpec sks = new SecretKeySpec(encryptPassword.getBytes(),"AES"); // static password
// SecretKeySpec sks = new SecretKeySpec(getResources().getString(R.string.encryptPassword).getBytes(),"AES");
cipher.init(Cipher.DECRYPT_MODE, sks);
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
bos.write(obuf);
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
bos.write(obuf);
} catch (BadPaddingException | IllegalBlockSizeException | InvalidKeyException | IOException | NoSuchAlgorithmException | NoSuchPaddingException e) {
e.printStackTrace();
}
}
public static byte[] loadFile(File filename) throws IOException {
byte[] filecontent = new byte[0];
FileInputStream fileInputStream = null;
try {
fileInputStream = new FileInputStream(filename);
// int byteLength = fff.length();
// In android the result of file.length() is long
long byteLength = filename.length(); // byte count of the file-content
filecontent = new byte[(int) byteLength];
fileInputStream.read(filecontent, 0, (int) byteLength);
} catch (IOException e) {
e.printStackTrace();
fileInputStream.close();
return filecontent;
}
fileInputStream.close();
return filecontent;
}
After loading the tempfile content to the byte array you can delete the file with a one-liner (again no exception handling):
Files.deleteIfExists(tempFile.toPath());

I'm writing a second answer and do not edit my first answer as it is a total different approach to solve the problem.
As you post a part of your code I can see that you have a byte array with the complete and decrypted content that gets played by the exoplayer:
output:
byte[] decrypt(File files)
as input for
ByteArrayDataSource src= new ByteArrayDataSource(decrypt(some_file));
So to avoid double and moretime memory consumption when playing with large files (approx. 50mb) my approach it is to download the complete encrypted file and save it in a byte array.
On devices with a good memory equipment you can decrypt the encrypted byte array in one run to another byte array and play the music from this decrypted byte array (step 6 + 8 in my sample program).
Using a low memory device you decrypt the byte array in chunks (in my program in 16 byte long blocks) and save the decrypted chunks at the same place in the encrypted byte array. When all chunks got processed the (former) encrypted data are now decrypted and you used the memory of just one byte array length. Now you can play the music from this byte array (steps 7 + 8).
Just for explanation, steps 1-3 are on server side and in steps 3+4 the transmission takes place.
This example uses the AES CTR mode as it provides the same length for input and output data.
In the end I'm comparing the byte arrays to prove that the decryption was successful for steps 6 (direct decryption) and 7 (decryption in chunks):
output:
Decrypting 50mb data in chunks to avoid out of memory error
https://stackoverflow.com/questions/62412705/how-to-write-byte-array-without-oom
plaindata length: 52428810
cipherdata length: 52428810
decrypteddata length: 52428810
cipherdata parts in 16 byte long parts: 3276800 = rounds for decryption
cipherdata moduluo 16 byte long parts: 10 + 1 round for rest/modulus
cipherdata length: 52428810 (after decryption)
plaindata equals decrypteddata: true
plaindata equals cipherdata: true
code:
import javax.crypto.Cipher;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
import java.util.Arrays;
import java.util.Random;
public class EncryptionCtrSo4 {
public static void main(String[] args) throws GeneralSecurityException {
System.out.println("Decrypting 50mb data in chunks to avoid out of memory error");
System.out.println("https://stackoverflow.com/questions/62412705/how-to-write-byte-array-without-oom");
/*
* author michael fehr, http://javacrypto.bplaced.net
* no licence applies, no warranty
*/
// === server side ===
// 1 create a 50 mb byte array (unencrypted)
byte[] plaindata = new byte[(50 * 1024 * 1024 + 10)];
// fill array with (random) data
Random random = new Random();
random.nextBytes(plaindata);
// 2 encrypt the data with aes ctr mode, create random keys
SecureRandom secureRandom = new SecureRandom();
byte[] key = new byte[32]; // 32 byte = 256 bit aes key
secureRandom.nextBytes(key);
byte[] iv = new byte[16]; // 16 byte = 128 bit
secureRandom.nextBytes(iv);
SecretKeySpec keySpecEnc = new SecretKeySpec(key, "AES");
IvParameterSpec ivParameterSpecEnc = new IvParameterSpec(iv);
Cipher cipherEnc = Cipher.getInstance("AES/CTR/NoPadding");
cipherEnc.init(Cipher.ENCRYPT_MODE, keySpecEnc, ivParameterSpecEnc);
byte[] cipherdata = cipherEnc.doFinal(plaindata);
System.out.println("plaindata length: " + plaindata.length);
System.out.println("cipherdata length: " + cipherdata.length);
// 3 transfer the cipherdata to app
// ...
// === app side ===
// 4 receive encrypted data from server
// ...
// 5 decryption setup
SecretKeySpec keySpecDec = new SecretKeySpec(key, "AES");
IvParameterSpec ivParameterSpecDec = new IvParameterSpec(iv);
Cipher cipherDec = Cipher.getInstance("AES/CTR/NoPadding");
cipherDec.init(Cipher.DECRYPT_MODE, keySpecDec, ivParameterSpecDec);
// 6 decryption in one run on high memory devices
byte[] decrypteddata = cipherDec.doFinal(cipherdata);
System.out.println("decrypteddata length: " + decrypteddata.length);
// 7 decryption in chunks using the cipherdata byte array
int cipherdataLength = cipherdata.length;
int chunksize = 16; // should be a multiple of 16, minimum 16
byte[] decryptedPart = new byte[chunksize];
int parts16byteDiv = cipherdataLength / chunksize;
int parts16byteMod = cipherdataLength % chunksize;
System.out.println("cipherdata parts in " + chunksize + " byte long parts: " + parts16byteDiv + " = rounds for decryption");
System.out.println("cipherdata moduluo " + chunksize + " byte long parts: " + parts16byteMod + " + 1 round for rest/modulus");
for (int i = 0; i < parts16byteDiv; i++) {
cipherDec.update(cipherdata, (i * chunksize), chunksize, decryptedPart);
System.arraycopy(decryptedPart, 0, cipherdata, (i * chunksize), decryptedPart.length);
}
if (parts16byteMod > 0) {
decryptedPart = new byte[parts16byteMod];
cipherDec.update(cipherdata, (parts16byteDiv * chunksize), parts16byteMod, decryptedPart);
System.arraycopy(decryptedPart, 0, cipherdata, (parts16byteDiv * chunksize), decryptedPart.length);
}
System.out.println("cipherdata length: " + cipherdata.length + " (after decryption)");
// the cipherdata byte array is now decrypted !
// 8 use cipherdata (encrypted) or decrypteddata as input for exoplayer
// compare ciphertext with decrypteddata in step 6
System.out.println("plaindata equals decrypteddata: " + Arrays.equals(plaindata, decrypteddata));
// check that (decrypted) cipherdata equals plaindata of step 7
System.out.println("plaindata equals cipherdata: " + Arrays.equals(plaindata, cipherdata));
}
}

android cipher doesn't decrypt first 16 bytes / characters of encrypted data

I am working on a file encryption/decryption app. I am using a simple .txt file for testing. When I select the file from within the app and choose to encrypt, the entire file data is encrypted. However, when I decrypt only part of the file data gets decrypted. For some reason the first 16 bytes/characters doesn't get decrypted.
test_file.txt contents: "This sentence is used to check file encryption/decryption results."
encryption result: "¾mÁSTÐÿT:Y­„"O¤]ÞPÕµß~ëqrÈb×ßq²¨†ldµJ,O|56\e^-’#þûÝû"
decryption result: "£ÿÒÜÑàh]VÄþ„- used to check file encryption/decryption results."
There aren't any errors in the logcat.
What am I doing wrong?
Method to encrypt file:
public void encryptFile(String password, String filePath) {
byte[] encryptedFileData = null;
byte[] fileData = null;
try {
fileData = readFile(filePath);//method provided below
// 64 bit salt for testing only
byte[] salt = "goodsalt".getBytes("UTF-8");
SecretKey key = generateKey(password.toCharArray(), salt);//method provided below
byte[] keyData = key.getEncoded();
SecretKeySpec sKeySpec = new SecretKeySpec(keyData, "AES");
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, sKeySpec);
encryptedFileData = cipher.doFinal(fileData);
saveData(encryptedFileData, filePath);//method provided below
}
catch (Exception e) {
e.printStackTrace();
}
}
Method to read file content:
public byte[] readFile(String filePath) {
byte[] fileData;
File file = new File(filePath);
int size = (int) file.length();
fileData = new byte[size];
try {
BufferedInputStream inputStream = new BufferedInputStream(new FileInputStream(file));
inputStream.read(fileData);
inputStream.close();
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
return fileData;
}
Method to generate secret key:
private SecretKey generateKey(char[] password, byte[] salt) throws NoSuchAlgorithmException, InvalidKeySpecException {
// Number of PBKDF2 hardening rounds to use. Larger values increase computation time. You
// should select a value that causes computation to take >100ms.
final int iterations = 1000;
// Generate a 256-bit key
final int outputKeyLength = 256;
SecretKeyFactory secretKeyFactory;
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
// Use compatibility key factory -- only uses lower 8-bits of passphrase chars
secretKeyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1And8bit");
}
else {
// Traditional key factory. Will use lower 8-bits of passphrase chars on
// older Android versions (API level 18 and lower) and all available bits
// on KitKat and newer (API level 19 and higher).
secretKeyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
}
KeySpec keySpec = new PBEKeySpec(password, salt, iterations, outputKeyLength);
return secretKeyFactory.generateSecret(keySpec);
}
Method to save encrypted/decrypted data to the file:
private void saveData(byte[] newFileData, String filePath) {
File file = new File(filePath);
try {
BufferedOutputStream outputStream = new BufferedOutputStream(new FileOutputStream(file));
outputStream.write(newFileData);
outputStream.flush();
outputStream.close();
}
catch (IOException e) {
e.printStackTrace();
}
}
Method to decrypt file:
public void decryptFile(String password, String filePath) {
byte[] decryptedFileData = null;
byte[] fileData = null;
try {
fileData = readFile(filePath);
byte[] salt = "goodsalt".getBytes("UTF-8");//generateSalt();
SecretKey key = generateKey(password.toCharArray(), salt);
byte[] keyData = key.getEncoded();
SecretKeySpec sKeySpec = new SecretKeySpec(keyData, "AES");
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, sKeySpec);
decryptedFileData = cipher.doFinal(fileData);
saveData(decryptedFileData, filePath);
}
catch (Exception e) {
e.printStackTrace();
}
}
This line of code encrypts the file:
//simple password for testing only
encryptor.encryptFile("password", "storage/emulated/0/Download/test_file.txt");
This line decrypts the file:
encryptor.decryptFile("password", "storage/emulated/0/Download/test_file.txt");
Edit: Thanks to DarkSquirrel42 and Oncaphillis. You guys are awesome!
Adding this line of code to both encrypt and decrypt functions solved my problem.
//note: the initialization vector (IV) must be 16 bytes in this case
//so, if a user password is being used to create it, measures must
//be taken to ensure proper IV length; random iv is best and should be
//stored, possibly alongside the encrypted data
IvParameterSpec ivSpec = new IvParameterSpec(password.getBytes("UTF-8"));
and then,
cipher.init(Cipher.XXXXXXX_MODE, sKeySpec, ivSpec);

your problem has something to do with the cipher's mode of operation ... cbc, or cipher block chaining mode
in general CBC is simple ... take whatever the output of your previous encryiption block was, and xor that onto the current input before encrypting it
for the first block we obviously have a problem... there is no previous block ... therefore we introduce something called IV ... an initialisation vector ... a block ength of random bytes ...
now ... as you can imagine, you will need the same IV when you want to decrypt ...
since you don't save that, the AES implementation will give you a random IV every time ...
therefore you don't have all information to decrypt block 1 ... which is the first 16 bytes in case of AES ...
when handling CBC mode data, it's allways a good choice to simply prepend the used IV in your cypertext output ... the IV shall just be random ... it is no secret ...

Like #ÐarkSquirrel42 already points out the en/decrytion routine for CBC seems to interpret the first 16 bytes as an initialisation vector. This worked for me:
// got to be random
byte[] iv = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };
IvParameterSpec ivspec = new IvParameterSpec(iv);
cipher.init(Cipher.XXXXX_MODE, sKeySpec,ivspec);

Increase speed of encryption AES android

I develop an android application for encrypt files on the phone. By searching, i found this topic : How to encrypt file from SD card using AES in Android?
The method works fine but it is very slow to encrypt files...
At this line : byte[] d = new byte[8]; why only 8 bytes ? can't we set an higher value ?
Also, do you know a way to encrypt files fastly ? I heard of crypto++ for native code implementation but how can I implement JNI on my application ?
Thank you,
EDIT : Encryption function
public void encrypt (String RSAPrivateKey, String Key, byte[] iv, String zipname, ZipEncryptAsyncTask task) throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException, InvalidAlgorithmParameterException
{
FileInputStream fis = new FileInputStream (zipname + ".temp");
FileOutputStream fos = new FileOutputStream (zipname);
SecretKeySpec sks = new SecretKeySpec (Base64.decode(Key, Base64.DEFAULT), "AES");
IvParameterSpec ivspec = new IvParameterSpec (iv);
Cipher cipher = Cipher.getInstance ("AES/CBC/PKCS5Padding");
fos.write(String.valueOf(RSAPrivateKey.getBytes().length).getBytes());
fos.write(RSAPrivateKey.getBytes());
cipher.init(Cipher.ENCRYPT_MODE, sks, ivspec);
CipherOutputStream cos = new CipherOutputStream(fos, cipher);
long size = 0;
byte[] d = new byte[8];
for(int b; (b = fis.read(d)) != -1; )
{
cos.write(d, 0, b);
task.doProgress((size += 8));
}
cos.flush();
cos.close();
fis.close();
new File(zipname + ".temp").delete();
}

As an alternative, you could consider changing the cypher mode you are using. CBC mode must be used serially, block by block. Counter mode (CTR) can be run in parallel with different blocks being encrypted simultaneously. Of course there are overheads to parallel processing, and you will need to rework your code, but if buffer resizing does not give you enough speed gains, then it might be the next option to try.

as #CodesInChaos said, a way to do this, is to increase the size of the buffer.
Now I do a benchmark which determines the best size for the buffer and I use the optimal value.

Decrypt file in parts

So I have these large files (6GB+) that I need to decrypt on a 32 bit computer. The general procedure that I used previously was to read the entire file in memory, then pass it on to the decrypt function and then write it all back to a file. This doesn't really work due to memory limitations. I did try passing the file in parts to the decrypt function but it seems to mess up around the boundaries of where I break up the file before sending it to the decrypt function.
I've tried breaking up the file in parts relative to key size but that doesnt seem to matter. I tried a byte array of size 2048 as well as a byte aray of size 294 thinking that might be the special boundary but, no luck. I can see parts of the file correctly decrypted but parts which are total gibberish.
Is it just NOT POSSIBLE to decrypt the file in chunks? If there is a way, then how?
Here is my decryption function / my attempt to decrypt in parts.
private Path outFile;
private void decryptFile(FileInputStream fis, byte[] initVector, byte[] aesKey, long used) {
//Assume used = 0 for this function.
byte[] chunks = new byte[2048]; //If this number is greater than or equal to the size of the file then we are good.
try {
if (outFile.toFile().exists())
outFile.toFile().delete();
outFile.toFile().createNewFile();
FileOutputStream fos = new FileOutputStream(outFile.toFile());
OutputStreamWriter out = new OutputStreamWriter(fos);
IvParameterSpec spec = new IvParameterSpec(Arrays.copyOfRange(initVector, 0, 16));
SecretKeySpec key = new SecretKeySpec(aesKey, "AES");
Cipher cipher = Cipher.getInstance("AES/CFB/NoPadding");
cipher.init(Cipher.DECRYPT_MODE, key, spec);
int x;
while ((x = fis.read(chunks, 0, chunks.length)) != -1) {
byte[] dec = cipher.doFinal(Arrays.copyOfRange(chunks, 0, x));
out.append(new String(dec));
}
out.close();
fos.close();
} catch (Exception e) {
e.printStackTrace();
LOG.error(ExceptionUtils.getStackTrace(e));
}
}

Consider using Cipher#update(byte[], int, int, byte[], int) instead of doFinal() for multipart operations. This will take care of part boundaries for you.
The last part of the deciphered data can be obtained by calling the doFinal(byte[] output, int outputOffset) method.

Encryption of image files on Android -- Cipher(Output|Input)Stream problems

I'm trying to encrypt image files on Android with password based encryption. To save the encrypted image I just do this:
FileOutputStream fos = new FileOutputStream(thumbnailFile);
CipherOutputStream cos = new CipherOutputStream(fos, encryptCipher);
Bitmap thumbnail = Bitmap.createScaledBitmap(bm2, 140, 140, true);
thumbnail.compress(Bitmap.CompressFormat.JPEG, 80, cos);
and to read it, this:
FileInputStream fis = new FileInputStream(f);
CipherInputStream cis = new CipherInputStream(fis, decryptCipher);
Bitmap b = BitmapFactory.decodeStream(cis);
but the Bitmap ends up as null. The code works when I bypass the encryption; that is when I use the File(Input|Output)Streams rather than the Cipher(Input|Output)streams.
My Ciphers are created as follows:
public void initCiphers(char password[]) {
PBEKeySpec pbeKeySpec;
PBEParameterSpec pbeParamSpec;
SecretKeyFactory keyFac;
byte[] salt = {
(byte)0xc7, (byte)0x73, (byte)0x21, (byte)0x8c,
(byte)0x7e, (byte)0xc8, (byte)0xee, (byte)0x99
};
int count = 20;
pbeParamSpec = new PBEParameterSpec(salt, count);
pbeKeySpec = new PBEKeySpec(password);
try {
keyFac = SecretKeyFactory.getInstance("PBEWithMD5AndDES");
SecretKey pbeKey = keyFac.generateSecret(pbeKeySpec);
encryptCipher = Cipher.getInstance("PBEWithMD5AndDES");
decryptCipher = Cipher.getInstance("PBEWithMD5AndDES");
encryptCipher.init(Cipher.ENCRYPT_MODE, pbeKey, pbeParamSpec);
decryptCipher.init(Cipher.DECRYPT_MODE, pbeKey, pbeParamSpec);
} catch (Exception e) {
Log.v("tag", e.toString());
}
I don't get any exceptions.
There is obviously some problem with using Cipher(Output|Input)Streams with the android functions for encoding and/or decoding images, but since those functions are opaque and there are no exceptions, its hard to know what it is. I suspect it has to do with padding or flushing. Any assistance would be gratefully appreciated.

When writing to a CipherOutputStream, make sure you close() the stream after writing the data (and not closing the underlying stream before it). Closing makes sure the right padding is added. A flush() alone is not enough here.
Also, I would advise to not use DES for new protocols - preferred nowadays is AES.

You could subclass CipherOutputStream or even just OutputStream, and just override the flush() method to do nothing.