Develop Reference

How to write byte array without OOM?

I have an android app which handles some large byte array but I am getting some OOM crash in my Firebase Crashlytics reports for devices with low memory while handling byte array whose size may go from 10 mb to 50mb. Below is my method that I have used. So could anyone help me to improve it to avoid OOM.
byte[] decrypt(File files) {
try {
FileInputStream fis = new FileInputStream(files);
SecretKeySpec sks = new SecretKeySpec(getResources().getString(R.string.encryptPassword).getBytes(),
"AES");
Cipher cipher = Cipher.getInstance("AES");
cipher.init(Cipher.DECRYPT_MODE, sks);
CipherInputStream cis = new CipherInputStream(fis, cipher);
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int b;
byte[] d = new byte[1024];
while ((b = cis.read(d)) != -1) {
buffer.write(d, 0, b); //this is one of the line which is being referred for the OOM in firebase
}
byte[] decryptedData = buffer.toByteArray();//this is the line which is being referred for the OOM in firebase
buffer.flush();
fis.close();
cis.close();
return decryptedData;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
EDIT
Actually I am using the above method for decrypting downloaded audio files which are encrypted during downloading.
The above methods return the content of the encrypted files to exoplayer to play its content and I am calling the above method in the following way
ByteArrayDataSource src= new ByteArrayDataSource(decrypt(some_file));
Uri uri = new UriByteDataHelper().getUri(decrypt(some_file));
DataSpec dataSpec = new DataSpec(uri);
src.open(dataSpec);
DataSource.Factory factory = new DataSource.Factory()
{
#Override public DataSource createDataSource()
{
return src;
}
};
audioSource = new ProgressiveMediaSource.Factory(factory).createMediaSource(uri);

First of all, I would make sure that the devices where this is running have enought heap memory to run this, it might be that simply the software has already been allocated with a lot of space and there might not be much more left on the heap to provide the software. This operation should not require much memory and I don't see anything obvious that would point towards trying to allocated and unexpectadely large amount of memory.
What I would recommend though, if you want to hav a quick test is actually simply lowering the byte array size, any particular reason why you are using 1024?
If possible perhaps try:
byte[] d = new byte[8];
Also, If that was me, I would store the read data temporarily, on an array perhaps and only once the read of the cypher has finished, I would call
buffer.write()
From my experience, trying to read and write at the same time tens to not be advised and can result in serveral issues, at the very least you should make sure you have the whole cypher and that it is a valid one (if you have some validation requirements) and only then send it.
Again, this should not be the core issue, the device seems that it is lacking enough availalbe memory to be allocated, perhaps too much reserved memory for other processes?

You should consider of writing the decrypted data to a tempfile and then reload the data for usage.
The main reasons for the Out of memory error are the ByteArrayOutputStream AND
byte[] decryptedData = buffer.toByteArray(), because both of them hold the complete (decrypted) data and that doubles the memory consumption of your decrypt method.
You could avoid this by decrypting the data to a tempfile in the first step and later load the data from the tempfile. I modified the decrypt method to handle the decrypted output stream and later there is a method for reloading the decrypted data (there is no propper exception handling and for my testing I setup a static encryptPassword-variable ...).
There is just one part left for you - you need to find a good place for the tempfile and I'm no Android specialist.
Just two notes: You are using the unsecure AES ECB mode and the String to byte[]conversion for your password should be
changed to
.getBytes(StandardCharsets.UTF_8)
on ecryption and decryption side to avoid errors caused by different encodings on different platforms.
public static void decryptNew(File files, File tempfiles) {
try (FileInputStream fis = new FileInputStream(files);
BufferedInputStream in = new BufferedInputStream(fis);
FileOutputStream out = new FileOutputStream(tempfiles);
BufferedOutputStream bos = new BufferedOutputStream(out)) {
byte[] ibuf = new byte[1024];
int len;
Cipher cipher = Cipher.getInstance("AES");
SecretKeySpec sks = new SecretKeySpec(encryptPassword.getBytes(),"AES"); // static password
// SecretKeySpec sks = new SecretKeySpec(getResources().getString(R.string.encryptPassword).getBytes(),"AES");
cipher.init(Cipher.DECRYPT_MODE, sks);
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
bos.write(obuf);
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
bos.write(obuf);
} catch (BadPaddingException | IllegalBlockSizeException | InvalidKeyException | IOException | NoSuchAlgorithmException | NoSuchPaddingException e) {
e.printStackTrace();
}
}
public static byte[] loadFile(File filename) throws IOException {
byte[] filecontent = new byte[0];
FileInputStream fileInputStream = null;
try {
fileInputStream = new FileInputStream(filename);
// int byteLength = fff.length();
// In android the result of file.length() is long
long byteLength = filename.length(); // byte count of the file-content
filecontent = new byte[(int) byteLength];
fileInputStream.read(filecontent, 0, (int) byteLength);
} catch (IOException e) {
e.printStackTrace();
fileInputStream.close();
return filecontent;
}
fileInputStream.close();
return filecontent;
}
After loading the tempfile content to the byte array you can delete the file with a one-liner (again no exception handling):
Files.deleteIfExists(tempFile.toPath());

I'm writing a second answer and do not edit my first answer as it is a total different approach to solve the problem.
As you post a part of your code I can see that you have a byte array with the complete and decrypted content that gets played by the exoplayer:
output:
byte[] decrypt(File files)
as input for
ByteArrayDataSource src= new ByteArrayDataSource(decrypt(some_file));
So to avoid double and moretime memory consumption when playing with large files (approx. 50mb) my approach it is to download the complete encrypted file and save it in a byte array.
On devices with a good memory equipment you can decrypt the encrypted byte array in one run to another byte array and play the music from this decrypted byte array (step 6 + 8 in my sample program).
Using a low memory device you decrypt the byte array in chunks (in my program in 16 byte long blocks) and save the decrypted chunks at the same place in the encrypted byte array. When all chunks got processed the (former) encrypted data are now decrypted and you used the memory of just one byte array length. Now you can play the music from this byte array (steps 7 + 8).
Just for explanation, steps 1-3 are on server side and in steps 3+4 the transmission takes place.
This example uses the AES CTR mode as it provides the same length for input and output data.
In the end I'm comparing the byte arrays to prove that the decryption was successful for steps 6 (direct decryption) and 7 (decryption in chunks):
output:
Decrypting 50mb data in chunks to avoid out of memory error
https://stackoverflow.com/questions/62412705/how-to-write-byte-array-without-oom
plaindata length: 52428810
cipherdata length: 52428810
decrypteddata length: 52428810
cipherdata parts in 16 byte long parts: 3276800 = rounds for decryption
cipherdata moduluo 16 byte long parts: 10 + 1 round for rest/modulus
cipherdata length: 52428810 (after decryption)
plaindata equals decrypteddata: true
plaindata equals cipherdata: true
code:
import javax.crypto.Cipher;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
import java.util.Arrays;
import java.util.Random;
public class EncryptionCtrSo4 {
public static void main(String[] args) throws GeneralSecurityException {
System.out.println("Decrypting 50mb data in chunks to avoid out of memory error");
System.out.println("https://stackoverflow.com/questions/62412705/how-to-write-byte-array-without-oom");
/*
* author michael fehr, http://javacrypto.bplaced.net
* no licence applies, no warranty
*/
// === server side ===
// 1 create a 50 mb byte array (unencrypted)
byte[] plaindata = new byte[(50 * 1024 * 1024 + 10)];
// fill array with (random) data
Random random = new Random();
random.nextBytes(plaindata);
// 2 encrypt the data with aes ctr mode, create random keys
SecureRandom secureRandom = new SecureRandom();
byte[] key = new byte[32]; // 32 byte = 256 bit aes key
secureRandom.nextBytes(key);
byte[] iv = new byte[16]; // 16 byte = 128 bit
secureRandom.nextBytes(iv);
SecretKeySpec keySpecEnc = new SecretKeySpec(key, "AES");
IvParameterSpec ivParameterSpecEnc = new IvParameterSpec(iv);
Cipher cipherEnc = Cipher.getInstance("AES/CTR/NoPadding");
cipherEnc.init(Cipher.ENCRYPT_MODE, keySpecEnc, ivParameterSpecEnc);
byte[] cipherdata = cipherEnc.doFinal(plaindata);
System.out.println("plaindata length: " + plaindata.length);
System.out.println("cipherdata length: " + cipherdata.length);
// 3 transfer the cipherdata to app
// ...
// === app side ===
// 4 receive encrypted data from server
// ...
// 5 decryption setup
SecretKeySpec keySpecDec = new SecretKeySpec(key, "AES");
IvParameterSpec ivParameterSpecDec = new IvParameterSpec(iv);
Cipher cipherDec = Cipher.getInstance("AES/CTR/NoPadding");
cipherDec.init(Cipher.DECRYPT_MODE, keySpecDec, ivParameterSpecDec);
// 6 decryption in one run on high memory devices
byte[] decrypteddata = cipherDec.doFinal(cipherdata);
System.out.println("decrypteddata length: " + decrypteddata.length);
// 7 decryption in chunks using the cipherdata byte array
int cipherdataLength = cipherdata.length;
int chunksize = 16; // should be a multiple of 16, minimum 16
byte[] decryptedPart = new byte[chunksize];
int parts16byteDiv = cipherdataLength / chunksize;
int parts16byteMod = cipherdataLength % chunksize;
System.out.println("cipherdata parts in " + chunksize + " byte long parts: " + parts16byteDiv + " = rounds for decryption");
System.out.println("cipherdata moduluo " + chunksize + " byte long parts: " + parts16byteMod + " + 1 round for rest/modulus");
for (int i = 0; i < parts16byteDiv; i++) {
cipherDec.update(cipherdata, (i * chunksize), chunksize, decryptedPart);
System.arraycopy(decryptedPart, 0, cipherdata, (i * chunksize), decryptedPart.length);
}
if (parts16byteMod > 0) {
decryptedPart = new byte[parts16byteMod];
cipherDec.update(cipherdata, (parts16byteDiv * chunksize), parts16byteMod, decryptedPart);
System.arraycopy(decryptedPart, 0, cipherdata, (parts16byteDiv * chunksize), decryptedPart.length);
}
System.out.println("cipherdata length: " + cipherdata.length + " (after decryption)");
// the cipherdata byte array is now decrypted !
// 8 use cipherdata (encrypted) or decrypteddata as input for exoplayer
// compare ciphertext with decrypteddata in step 6
System.out.println("plaindata equals decrypteddata: " + Arrays.equals(plaindata, decrypteddata));
// check that (decrypted) cipherdata equals plaindata of step 7
System.out.println("plaindata equals cipherdata: " + Arrays.equals(plaindata, cipherdata));
}
}

AES encryption in java and decryption in javascript using CryptoJS

I have below code to encrypt some file content in java by using AES/CTR/NOPADDING mode. I am using crypto package of javax. Also I am using same secret key to generate key and iv.
Cipher cipher = Cipher.getInstance("AES/CTR/NoPadding");
byte[] secretKey = Base64.decodeBase64("uQsaW+WMUrjcsq1HMf+2JQ==");
SecretKeySpec key = new SecretKeySpec(secretKey, "AES");
IvParameterSpec iv = new IvParameterSpec(secretKey);
cipher.init(mode, key , iv);
FileInputStream fileInputStream = new FileInputStream(sourceFilePath);
FileOutputStream fileOutputStream = new FileOutputStream(destFilePath);
int read = 0;
while ((fileInputStream.available()) > 0) {
byte[] block = new byte[4096];
read = fileInputStream.read(block);
byte[] writeBuffer = cipher.update(block);
fileOutputStream.write(writeBuffer, 0, read);
}
byte[] writeBuffer = cipher.doFinal();
fileOutputStream.write(writeBuffer, 0, writeBuffer.length);
fileInputStream.close();
fileOutputStream.close();
I am not able to decrypt encrypted content in javascript by using cryptojs.
Here is something I have tried.
var key = CryptoJS.enc.Hex.parse(atob('uQsaW+WMUrjcsq1HMf+2JQ=='));
var decrypted = CryptoJS.AES.decrypt(encryptedContent, key, {
mode: CryptoJS.mode.CTR,
iv: key,
padding: CryptoJS.pad.NoPadding
});
var decryptedText = CryptoJS.enc.Utf8.stringify(decrypted);
Can somebody tell me what I am doing wrong? Or tell me how to do it.
I am able to encrypt and decrypt in java and javascript independently.

In the CryptoJS-documentation is explained which data types and parameters the CryptoJS.decrypt()-method expects and which encoders are available:
The key has to be passed to the CryptoJS.decrypt()-method as a WordArray. Since the key-data are Base64-encoded, they can be converted into a WordArray with the CryptoJS.enc.Base64.parse()-method.
The ciphertext can be passed to the CryptoJS.decrypt()-method as a WordArray inside a CipherParams-object. The Java-code stores the encrypted data in a file. Assuming that the string encryptedContent contains those data as hex-string (unfortunately, this does not emerge from the posted code, so that an assumption must be made here), they can be converted into a WordArray with the CryptoJS.enc.Hex.parse()-method and wrapped in a CipherParams-object.
The CryptoJS.decrypt()-method returns a WordArray which can be converted with the CryptoJS.enc.Utf8.stringify()-method into a string.
If the following plain text is contained in the input-file:
This is the plain text which needs to be encrypted!
the Java-code stores the following byte-sequence (= encrypted data) in the output-file:
52F415AB673427C42278E8D6F34C16134D7E3FE7986500980ED4063F3CF51162592CE0F5412CCA0BC2DBAE3F2AEC2D585EE8D7
The JavaScript-Code for the decryption is:
var key = CryptoJS.enc.Base64.parse('uQsaW+WMUrjcsq1HMf+2JQ==');
var encryptedContent = '52F415AB673427C42278E8D6F34C16134D7E3FE7986500980ED4063F3CF51162592CE0F5412CCA0BC2DBAE3F2AEC2D585EE8D7';
var cipherParams = CryptoJS.lib.CipherParams.create({
ciphertext: CryptoJS.enc.Hex.parse(encryptedContent)
});
var decrypted = CryptoJS.AES.decrypt(cipherParams, key, {
mode: CryptoJS.mode.CTR,
iv: key,
padding: CryptoJS.pad.NoPadding
});
var decryptedText = CryptoJS.enc.Utf8.stringify(decrypted);
console.log(decryptedText);
which displays the original plain text in the console. To run the code above at least CryptoJS-version 3.1.4 is needed (see versions, cdnjs).

Your encryption loop is wrong. I am not sure if it's the cause of your issues, but I'd start with it
read = fileInputStream.read(block);
byte[] writeBuffer = cipher.update(block);
even if you read only partial size of the block, you execute encryption operation over the whole block, you may try
byte[] writeBuffer = cipher.update(block, 0, read);
About using the key as IV, I have to stress that with CTR mode the security will be completely broken.

How to Produce an Actual Encrypted Files with AES? [JAVA]

Hi i have explore many good site about AES Encryption, Most Site will be nicely detail about how to encrypt files and really help me understand the AES Encryption.
but i am still unclear about how to produce files that is encrypted. this tutorial example explain how AES encryption was done but i still cannot see the physical encrypted file. Most example show only how to encrypt and decrypt but did not explain about how to produce encrypted physical file.
My Question here is how do we actually produce an actual encrypted files, i believe this question is relevance to SO citizen as this might help other in future.
Answer
The code below will encrypt a text file with physical encrypted file.
final Path origFile = Paths.get("C:\\3.txt");
final byte[] contents = Files.readAllBytes(origFile);
// Get the KeyGenerator
KeyGenerator kgen = KeyGenerator.getInstance("AES");
kgen.init(128); // 192 and 256 bits may not be available
// Generate the secret key specs.
SecretKey skey = kgen.generateKey();
byte[] raw = skey.getEncoded();
SecretKeySpec skeySpec = new SecretKeySpec(raw, "AES");
// Instantiate the cipher
Cipher cipher = Cipher.getInstance("AES");
cipher.init(Cipher.ENCRYPT_MODE, skeySpec);
byte[] encrypted = cipher.doFinal(contents.toString().getBytes());
System.out.println("encrypted string: " + encrypted.toString());
cipher.init(Cipher.DECRYPT_MODE, skeySpec);
byte[] original =cipher.doFinal(encrypted);
String originalString = new String(original);
System.out.println("Original string: " +originalString);
final Path newFile = Paths.get("C:\\3encrypted.aes");
Files.write(newFile, encrypted, StandardOpenOption.CREATE);
}
As fge suggest, this is not suite for encrypting large file. ill provide new answer when i done with my research.

Your code is not correct; you try and read bytes from a file and then put it into a StringBuffer which is a character sequence. Don't do that!
Read the bytes directly:
final Path origFile = Paths.get("C:\\3.txt");
final byte[] contents = Files.readAllBytes(origFile);
Then encrypt like you do, and write your encrypted byte array into a new file:
final Path newFile = Paths.get("C:\\3encrypted.aes"); // or another name
Files.write(newFile, encrypted, StandardOpenOption.CREATE);
It is very important to understand that String is not suitable for binary data. Please see this link for more details.

Decrypt file in parts

So I have these large files (6GB+) that I need to decrypt on a 32 bit computer. The general procedure that I used previously was to read the entire file in memory, then pass it on to the decrypt function and then write it all back to a file. This doesn't really work due to memory limitations. I did try passing the file in parts to the decrypt function but it seems to mess up around the boundaries of where I break up the file before sending it to the decrypt function.
I've tried breaking up the file in parts relative to key size but that doesnt seem to matter. I tried a byte array of size 2048 as well as a byte aray of size 294 thinking that might be the special boundary but, no luck. I can see parts of the file correctly decrypted but parts which are total gibberish.
Is it just NOT POSSIBLE to decrypt the file in chunks? If there is a way, then how?
Here is my decryption function / my attempt to decrypt in parts.
private Path outFile;
private void decryptFile(FileInputStream fis, byte[] initVector, byte[] aesKey, long used) {
//Assume used = 0 for this function.
byte[] chunks = new byte[2048]; //If this number is greater than or equal to the size of the file then we are good.
try {
if (outFile.toFile().exists())
outFile.toFile().delete();
outFile.toFile().createNewFile();
FileOutputStream fos = new FileOutputStream(outFile.toFile());
OutputStreamWriter out = new OutputStreamWriter(fos);
IvParameterSpec spec = new IvParameterSpec(Arrays.copyOfRange(initVector, 0, 16));
SecretKeySpec key = new SecretKeySpec(aesKey, "AES");
Cipher cipher = Cipher.getInstance("AES/CFB/NoPadding");
cipher.init(Cipher.DECRYPT_MODE, key, spec);
int x;
while ((x = fis.read(chunks, 0, chunks.length)) != -1) {
byte[] dec = cipher.doFinal(Arrays.copyOfRange(chunks, 0, x));
out.append(new String(dec));
}
out.close();
fos.close();
} catch (Exception e) {
e.printStackTrace();
LOG.error(ExceptionUtils.getStackTrace(e));
}
}

Consider using Cipher#update(byte[], int, int, byte[], int) instead of doFinal() for multipart operations. This will take care of part boundaries for you.
The last part of the deciphered data can be obtained by calling the doFinal(byte[] output, int outputOffset) method.

Byte Array -> String -> Byte Array for encrypted text: not the same size

I am trying to encrypt a message, which works and returns it as a byte array. I then convert this byte array to a string, in order to send via a tcp network message. On the other end, I convert the string back to a byte array, however the resulting array is larger and I can't figure out why. I think it may be something to do with the encoding as if I use "MacRoman", I do not have this problem, however the program needs to be able to run on systems which do not support this encoding, so I decided to use UTF-8.
String message="222233332221";
//Encrypt message
byte[] encryptedMsg = encryptString(message, temp.loadCASPublicKey());
System.out.println("ENCRYPTED MESSAGE byte Length: "+encryptedMsg.length);
//Convert to String in order to send
String stringMessage = new String(encryptedMsg);
System.out.println("ENCRYPTED MESSAGE String Length: "+stringMessage.length());
//Convert String back to Byte[] and decrpt
byte[] byteMessage = stringMessage.getBytes("UTF-8");
System.out.println("ENCRYPTED MESSAGE byte Length: "+byteMessage.length);
Outputs:
ENCRYPTED MESSAGE byte Length: 256
ENCRYPTED MESSAGE String Length: 235
ENCRYPTED MESSAGE byte Length: 446
Can any one please point me in the right direction as to why the resulting byte array is 446 bytes not 256 bytes.
The encryptString part is as follows. I believe this returns a byte array in UTF-8?
private static byte[] encryptString(String message, Key publicKey) throws NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException, UnsupportedEncodingException {
Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.ENCRYPT_MODE, publicKey);
byte[] cipherData = cipher.doFinal(message.getBytes("UTF-8"));
return cipherData;
}

Managed to fix it using Base64.
byte[] encryptedMsg = Base64.encodeBase64(encryptString(message, temp.loadCASPublicKey()));
System.out.println("ENCRYPTED MESSAGE byte Length: "+encryptedMsg.length);
//Convert to String in order to send
String stringMessage = new String(encryptedMsg, "UTF-8");
System.out.println("ENCRYPTED MESSAGE String Length: "+stringMessage.length());
//Convert String back to Byte[] and decrpt
byte[] byteMessage = Base64.decodeBase64(stringMessage.getBytes("UTF-8"));
System.out.println("ENCRYPTED MESSAGE byte Length: "+byteMessage.length);

It's an encoding issue.
1) You have a byte array. It contains bytes
2) You convert it to a string. As soon as you do this, you have a UTF16 encoded String. So you have taken the bytes and changed them to characters.
3) You now convert those characters back to bytes. But if the original bytes were not UTF8 or UTF16, you might not have the same number of bytes. If the default encoding of the platform is MacRoman, then in step 3 you are translating your UTF16 String into bytes, but treating the characters as MacRoman.

I guess there is a good reason of doing encryption manually, however just in case... Do you consider using SSLSocket instead?