I'm new to encryption/compression in Java and I'm working on a test project where the goal is to compress and then encrypt files via a buffered input in Java. At no point should the file be stored on disk in a non-encrypted format, therefore I want to do the compression and encryption solely on a buffer until the file is fully written.
So the progression would be: read part of file into memory (buffer, 1024 bytes) -> compress (~32 bytes)-> encrypt -> output to disk -> repeat until entire file is written
The issue I'm facing is that once I perform the reverse operations to read the compressed/encrypted file back, only part of the data is there.
To accomplish my goal, I've been combining the Inflater/Deflater classes and a block cipher with AES 256 encryption.
Encryption setup:
byte[] randomSalt = new byte[8];
SecureRandom secRand = new SecureRandom();
secRand.nextBytes(randomSalt);
String randomPassword = new BigInteger(130, secRand).toString(32);
SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
KeySpec spec = new PBEKeySpec(randomPassword.toCharArray(), randomSalt, 65536, 256);
SecretKey tmp = factory.generateSecret(spec);
SecretKey secret = new SecretKeySpec(tmp.getEncoded(), "AES");
Cipher cipher = Cipher.getInstance("AES/CTR/NoPadding");
cipher.init(Cipher.ENCRYPT_MODE, secret);
AlgorithmParameters params = cipher.getParameters();
byte[] iv = params.getParameterSpec(IvParameterSpec.class).getIV();
Getting input / writing output:
BufferedInputStream bufferedInput = new BufferedInputStream(
new FileInputStream("file.txt"));
BufferedOutputStream bufferedOutput = new BufferedOutputStream(
new FileOutputStream("encrypted file"));
byte[] buffer = new byte[1024];
try {
while (bufferedInput.read(buffer) != -1) {
byte[] encryptedBuffer = cipher.doFinal(compress(buffer));
bufferedOutput.write(encryptedBuffer);
bufferedOutput.flush();
}
} catch (Exception e) {
//snip
} finally {
bufferedInput.close();
bufferedOutput.close();
}
Compress method:
public static byte[] compress(byte[] data) throws IOException{
Deflater deflater = new Deflater();
deflater.setInput(data);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(data.length);
deflater.finish();
byte[] buffer = new byte[1024];
while(!deflater.finished()){
int count = deflater.deflate(buffer);
outputStream.write(buffer, 0, count);
}
outputStream.close();
return outputStream.toByteArray();
}
What can I do to be able to compress and encrypt a file 1KB at a time and get the file back in its entirety when I perform the reverse operations?
Related
I'm currently doing an assignment for a college course using Java's JCA.
The application takes in a file and encrypts it (or decrypts it) using DES-ECB. I am fully aware that it's not a secure encryption algorithm.
It encrypts fine, I believe, however when decrypting it blows up with a "Input length must be multiple of 8" even though the original message is being padded with PKCS5.
I have read all literature and quetions regarding this problem here on StackOverflow, but none of the answers seem to resolve this issue, which leads me to believe I am somehow corrupting the message/file...
For the encryption:
Cipher cipher = Cipher.getInstance("DES/ECB/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, symmetricKey);
File file = new File(filePath);
FileOutputStream outputStream = new FileOutputStream("encrypted_"+file.getName());
CipherInputStream cipherStream = new CipherInputStream( new FileInputStream(file), cipher);
byte[] buffer = new byte[MAX_BUFFER]; //buffer para leitura
int bytes; //bytes a ler
//Encoder base64 - Apache Commons Codec
Base64 encoder = new Base64();
while ( (bytes = cipherStream.read(buffer)) != -1 ) {
byte[] encodedBuffer = encoder.encode(buffer);
outputStream.write(encodedBuffer, 0, bytes);
}
cipherStream.close();
outputStream.flush();
return outputStream;
For the decryption:
Cipher cipher = Cipher.getInstance("DES/ECB/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, symmetricKey);
File file = new File(filePath);
FileInputStream cipheredStream = new FileInputStream(file);
FileOutputStream outputStream = new FileOutputStream("decrypted_"+file.getName());
CipherOutputStream cipherOutStream = new CipherOutputStream(outputStream, cipher);
byte[] buffer = new byte[MAX_BUFFER];
int bytes;
//Decoder base 64 - Apache Commons Codec
Base64 decoder = new Base64();
cipheredStream.read(buffer);
byte[] decodedBuffer = decoder.decode(buffer);
byte[] output = cipher.doFinal(decodedBuffer);
cipherOutStream.write(output);
//TODO bug here -> use this for big files
/*while ( (bytes = cipheredStream.read(buffer)) != -1 ) {
byte[] decodedBuffer = decoder.decode(buffer);
cipherOutStream.write(decodedBuffer, 0, bytes);
}*/
cipherOutStream.close();
cipheredStream.close();
return outputStream;
I've tried using AES to no avail; I've tried no padding, obviously it didn't work.
I'm just lost and would appreciate knowing what I'm doing wrong.
Thanks to #Topaco, the solution was found using Base64InputStream.
Because the deciphering was being done BEFORE decoding, it was generating that error. It was fixed by doing this encryption side:
Base64OutputStream encoder = new Base64OutputStream(outputStream);
while ( (nBytes = cipherStream.read(buffer, 0, MAX_BUFFER)) != -1 )
encoder.write(buffer, 0, nBytes);
And decryption side the exact opposite:
Base64InputStream decoder = new Base64InputStream(fileInputStream);
while ( (nBytes = decoder.read(buffer, 0, MAX_BUFFER)) != -1 )
cipherOutStream.write(buffer, 0, nBytes);
I export a database with mysqldump a database in Ubuntu with java, then I encrypt and decrypt it with Java. I doing that with the following classes Encrypt and Decrypt with Java. But after the decryption some characters at the start of the file is wrong. Here is the problem:
At the first image is the file which programmatically have mysqldump, encrypt and decrypt. At the second one is just the mysqldump from the same command line. Can you point me the direction what to do? Thanks
EDIT
I have create a salt and stored it in a file like this:
Encryption:
FileInputStream saltFis = new FileInputStream("salt.enc");
byte[] salt = new byte[8];
saltFis.read(salt);
saltFis.close();
// reading the iv
FileInputStream ivFis = new FileInputStream("iv.enc");
byte[] iv = new byte[16];
ivFis.read(iv);
ivFis.close();
SecretKeyFactory factory = SecretKeyFactory.getInstance(secretAlgorithm1);
KeySpec keySpec = new PBEKeySpec(rsaSecret.toCharArray(), salt, 65536, 256);
SecretKey secretKey = factory.generateSecret(keySpec);
SecretKey secret = new SecretKeySpec(secretKey.getEncoded(), secretAlgorithm2);
//
Cipher cipher = Cipher.getInstance(algorithmEncryption);
cipher.init(Cipher.ENCRYPT_MODE, secret);
// file encryption
byte[] input = new byte[64];
int bytesRead;
while ((bytesRead = inFile.read(input)) != -1) {
byte[] output = cipher.update(input, 0, bytesRead);
if (output != null)
outFile.write(output);
}
byte[] output = cipher.doFinal();
if (output != null)
outFile.write(output);
inFile.close();
outFile.flush();
outFile.close();
Decryption:
FileInputStream saltFis = new FileInputStream("salt.enc");
byte[] salt = new byte[8];
saltFis.read(salt);
saltFis.close();
// reading the iv
FileInputStream ivFis = new FileInputStream("iv.enc");
byte[] iv = new byte[16];
ivFis.read(iv);
ivFis.close();
SecretKeyFactory factory = SecretKeyFactory.getInstance(secretAlgorithm1);
KeySpec keySpec = new PBEKeySpec(rsaSecret.toCharArray(), salt, 65536, 256);
SecretKey secretKey = factory.generateSecret(keySpec);
SecretKey secret = new SecretKeySpec(secretKey.getEncoded(), secretAlgorithm2);
// file decryption
Cipher cipher = Cipher.getInstance(algorithmEncryption);
cipher.init(Cipher.DECRYPT_MODE, secret, new IvParameterSpec(iv));
FileInputStream fis = new FileInputStream(decodedB64);
FileOutputStream fos = new FileOutputStream(outputFile);
byte[] in = new byte[64];
int read;
while ((read = fis.read(in)) != -1) {
byte[] output = cipher.update(in, 0, read);
if (output != null)
fos.write(output);
}
byte[] output = cipher.doFinal();
if (output != null)
fos.write(output);
fis.close();
fos.flush();
fos.close();
System.out.println("File Decrypted.");
Oh, that one is simple. That idiotic (but funny enough, seeming largely correct otherwise) method of file encryption using CBC stores the IV in a separate file, overwriting any old one. So if you overwrite or take the wrong IV file then you'll get 16 random bytes at the start after decryption. So unless you can find the IV file that hopefully makes sense, your first 16 bytes (/characters) are now lost forever.
Of course, any sane encryption program stores the salt (a password & PBKDF2 is used for key derivation) and IV in the same file as the ciphertext.
Still, if you managed to lose the salt file or password then all the data would have been lost, so there's that...
With the added code the issue becomes even more clear. In the encryption mode you are forgetting to create & use an IvParameterSpec entirely during initialization:
cipher.init(Cipher.ENCRYPT_MODE, secret);
however, because of the way the IV data is read, you don't get any warning that the variable isn't used:
ivFis.read(iv);
If you would have created a nice method such as IvParameterSpec iv = readIvFromFile() then you would have caught this error.
Note that Java (by default in the included provider for Cipher) uses an all zero IV, so you're lucky and your data isn't partially gone.
I'm writing a code that send encrypted file from client to server
but first the client send the encrypted message digest of the file to the server and then send the name of the file and at the end it will send the bytes of encrypted file,
but in the server side it read all these variables as one variable which is the digest ,
and when the server trying to decrypt the digest it throws Illegal Block Size Exception
My question here is how can the server read and save them in different variables ??
Client
// set mode to encrypt
AesCipher.init(Cipher.ENCRYPT_MODE, key);
DataOutputStream toServer = new DataOutputStream(socket.getOutputStream());
// get the digest of the file
MessageDigest md = MessageDigest.getInstance("MD5");
byte[] hash = md.digest(bytes);
// encrypt digest and write it to file
byte [] encryptedHash = AesCipher.doFinal(hash);
toServer.write(encryptedHash);
// write file name to server
toServer.writeUTF(fileName);
//encrypt file
byte[] encryptedByte = AesCipher.doFinal(bytes);
// write file to server
toServer.write(encryptedByte);
toServer.flush();
socket.close();
Server
// read digest of the file
byte [] digest =IOUtils.toByteArray(fromClient);
// decrypt it
AesCipher.init(Cipher.DECRYPT_MODE, key);
byte[] decryptedDigest = AesCipher.doFinal(digest);
// read file name to be received
String fileName = fromClient.readUTF();
File file = new File(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS), fileName);
file.createNewFile();
FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
// read file bytes from client
byte[] fileBytes = IOUtils.toByteArray(fromClient);
AesCipher.init(Cipher.DECRYPT_MODE, key);
byte[] decryptedByte = AesCipher.doFinal(fileBytes);
bos.write(decryptedByte, 0, decryptedByte.length);
bos.close();
also I tried this code but it didn't works too
// read digest of the file
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[1024];
while ((nRead = fromClient.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
byte[] digest = buffer.toByteArray();
// decrypt it
AesCipher.init(Cipher.DECRYPT_MODE, key);
byte[] decryptedDigest = AesCipher.doFinal(digest);
// read file name to be received
String fileName = fromClient.readUTF();
File file = new File(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS), fileName);
file.createNewFile();
FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
// read file bytes from client
byte[] fileBytes = IOUtils.toByteArray(fromClient);
AesCipher.init(Cipher.DECRYPT_MODE, key);
byte[] decryptedByte = AesCipher.doFinal(fileBytes);
bos.write(decryptedByte, 0, decryptedByte.length);
bos.close();
IOUtils.toByteArray(InputStream) reads the entire stream. So instead of just getting the hash bytes, you got the whole stream, and there was nothing left for the filename or the ciphertext, and the hash didn't check.
You don't need external libraries for this. You can do it all with DataInputStream and DataOutputStream. But you do need to send the length of the hash ahead of the hash.
Client:
// set mode to encrypt
AesCipher.init(Cipher.ENCRYPT_MODE, key);
DataOutputStream toServer = new DataOutputStream(socket.getOutputStream());
// get the digest of the file
MessageDigest md = MessageDigest.getInstance("MD5");
byte[] hash = md.digest(bytes);
// encrypt digest and write it to file
byte [] encryptedHash = AesCipher.doFinal(hash);
toServer.writeInt(encryptedHash.length);
toServer.write(encryptedHash);
// write file name to server
toServer.writeUTF(fileName);
//encrypt file
byte[] encryptedByte = AesCipher.doFinal(bytes);
// write file to server
toServer.writeInt(encryptedByte.length);
toServer.write(encryptedByte);
socket.close();
Server:
// read digest of the file
int digestLength = fromClient.readInt();
byte[] digest = new byte[digestLength];
fromClient.readFully(digest);
// decrypt it
AesCipher.init(Cipher.DECRYPT_MODE, key);
byte[] decryptedDigest = AesCipher.doFinal(digest);
// read file name to be received
String fileName = fromClient.readUTF();
File file = new File(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS), fileName);
FileOutputStream fos = new FileOutputStream(file);
BufferedOutputStream bos = new BufferedOutputStream(fos);
// read file bytes from client
int fileLength = fromClient.readInt();
byte[] fileBytes = new byte[fileLength];
fromClient.readFully(fileBytes);
AesCipher.init(Cipher.DECRYPT_MODE, key);
byte[] decryptedByte = AesCipher.doFinal(fileBytes);
bos.write(decryptedByte, 0, decryptedByte.length);
bos.close();
However the encryption and decryption parts of this would be much better done with CipherInputStream and CipherOutputStream. You shouldn't load entire files into memory.
Note that the file.createNewFile() call was redundant before new FileOutputStream(...).
Why you're encrypting a message digest is another mystery. You should be using it as a final step to compare with a locally-generated digest after decryption.
I'm getting Given final block not properly padded error while decrypting AES/CBC/PKCS5Padding cipher on large encrypted file.
I think this issue is caused by adding wrong initialization vector in cipher.init() method.
I can't read whole file at runtime, so i need to encrypt fixed-size blocks. At this point I'm creating IV and storing it to .txt file. But in decrypting method I'm using the same IV every decryption cycle. How should I change this?
Encryption:
void encrypt() throws Exception{
char[] password = passwordText.getText().toCharArray();
byte[] salt = new byte[8];
/* Creating and saving salt */
salt = saveSalt(salt);
/* Securing password */
SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
KeySpec spec = new PBEKeySpec(password, salt, 65536, 128);
SecretKey tmp = factory.generateSecret(spec);
SecretKey secret = new SecretKeySpec(tmp.getEncoded(), "AES");
if (choosedFile != null) {
/* Choosing algorithm for decryption */
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
/* Getting plain file */
CipherInputStream fis = new CipherInputStream(new FileInputStream(choosedFile), cipher);
CipherOutputStream fos = new CipherOutputStream(new FileOutputStream(choosedFile+".encrypted"), cipher);
/* Encrypting and Measuring */
long startTime = System.currentTimeMillis();
cipher.init(Cipher.ENCRYPT_MODE, secret);
byte[] rawText = new byte[128];
int count;
while((count = fis.read(rawText)) > 0) {
System.out.println(count);
byte[] encryptedText = cipher.doFinal(rawText);
fos.write(encryptedText, 0, count);
}
long stopTime = System.currentTimeMillis();
fis.close();
fos.close();
/* Creating initialization vector and storing*/
byte[] iVector = cipher.getIV();
saveIVector(iVector);
text.setText(text.getText() + "File was encrypted in " + (stopTime - startTime) + "ms.\n");
}
}
Decryption:
void decrypt() throws Exception {
/* Getting salt */
byte[] salt = getSalt();
/* Getting initialization vector */
byte[] iVector = getIVector();
/* Getting user password */
char[] password = passwordText.getText().toCharArray();
/* Securing password */
SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
KeySpec spec = new PBEKeySpec(password, salt, 65536, 128);
SecretKey tmp = factory.generateSecret(spec);
SecretKey secret = new SecretKeySpec(tmp.getEncoded(), "AES");
if (choosedFile != null) {
/* Choosing algorithm for decryption */
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
/* Getting ciphered file */
CipherInputStream fis = new CipherInputStream(new FileInputStream(choosedFile), cipher);
CipherOutputStream fos = new CipherOutputStream(new FileOutputStream(choosedFile+".decrypted"), cipher);
/* Decrypting and Measuring */
long startTime = System.currentTimeMillis();
cipher.init(Cipher.DECRYPT_MODE, secret, new IvParameterSpec(iVector));
byte[] rawText = new byte[128];
int count;
while((count = fis.read(rawText)) > 0) {
byte[] encryptedText = cipher.doFinal(rawText);
fos.write(encryptedText, 0, count);
}
long stopTime = System.currentTimeMillis();
fis.close();
fos.close();
When using CipherInputStream and CipherOutputStream, the streams handle all the calls to the cipher (that's why you pass the cipher to it on initialization). You just need to initialize it correctly, and stream the data through the stream, and the cipher stream will do the needed calls to update() and doFinal(). Remember to close the steams to trigger the doFinal().
Currently your code passes the data through the cipher several times in an uncontrolled way, and the data is messed up.
Also, you only need a CipherInputStream for decrypt, and a CipherOutputStream for encrypt. In your current code you use both for both encrypt and decrypt.
Encrypt could be something like this (this don't handle the iv ..):
...
cipher.init(Cipher.ENCRYPT_MODE, secret);
InputStream is = new FileInputStream(choosedFile);
OutputStream os = new CipherOutputStream(new FileOutputStream(choosedFile+".encrypted"), cipher);
byte[] buffer = new byte[1024];
int len;
while ((len = is.read(buffer)) != -1) {
os.write(buffer, 0, len);
}
is.close();
os.close();
...
Worried about "until I had to cut large file into fixed-size blocks".
Using "chunk" in place of "block" above because "block"has a specific meaning in block ciphers such as AES.
What are toy doing with the chunks, concatenating them?
With CBC mode, after the first block the the previous encrypted block value is effectively used as the IV for the next block. So when splitting and then concatenating the chunks the value of the last block of the previous chunk is the IV for the next chunk.
See CBC mode.
Or are you doing something completely different?
In a larger application doing other things - I need to encrypt and decrypt a file. So I have been looking around and have implemented these two core functions that basically use RSA keys to wrap a random AES key that encrypts a file. The symmetric key and iv are written to the start of the file.
I'm getting an exception ("javax.crypto.BadPaddingException: Decryption error") in the decrypt function portion of below. On the unpackKeyandIV line -- the doFinal. Specifically this line is the Exception point:
Object[] keyIv = unpackKeyAndIV(xCipher.doFinal(keyBlock));
I've checked and remade the RSA key pairs. I've also checked the save/load of the keyBlock.
My gut is the problem has something to do with how I write/read the keyBlock --- or encoding perhaps?
One goal is to keep the RSA/AES instance as generic as possible so as not to need Bouncy Castle or extra Java security unlimited strength extensions.
Any thoughts on where I might be going wrong.
Thanks in advance.
[Final update: This code below is working. Error was passing in a corrupted privKey]
// RSA_INSTANCE = "RSA";
// ASSYM_CRYPTO_STR = 1024;
// SYM_CRYPTO_STR = 128;
// SYM_CRYPTO = "AES";
// AES_INSTANCE = "AES/CTR/NoPadding";
//
// File in = plain input file
// File out = encrypted output file
// Key pubKey = public Key (that wraps a random AES key)
public static void encryptFile(File in, File out, Key pubKey) throws Exception {
FileInputStream fin;
FileOutputStream fout;
int nread = 0;
byte[] inbuf = new byte[1024];
fout = new FileOutputStream(out);
fin = new FileInputStream(in);
SecureRandom random = new SecureRandom();
// symmetric wrapping
Key sKey = createKeyForAES(Config.SYM_CRYPTO_STR, random);
IvParameterSpec sIvSpec = createCtrIvForAES(0, random);
// encrypt symmetric key with RSA/pub key
Cipher xCipher = Cipher.getInstance(Config.RSA_INSTANCE);
xCipher.init(Cipher.ENCRYPT_MODE, pubKey, random);
byte[] keyBlock = xCipher.doFinal(packKeyAndIv(sKey, sIvSpec));
fout.write(keyBlock);
// encrypt data with symmetric key
Cipher sCipher = Cipher.getInstance(Config.AES_INSTANCE);
sCipher.init(Cipher.ENCRYPT_MODE, sKey, sIvSpec);
// Now read our file and encrypt it.
while((nread = fin.read(inbuf)) > 0) {
fout.write(sCipher.update(inbuf, 0, nread)); // cannot be null, by construction
}
// NB doFinal() cannot return null, but can return a zero-length array, which is benign below.
fout.write(sCipher.doFinal());
fout.flush();
fin.close();
fout.close();
}
// Decrypt File
public static void decryptFile(File in, File out, Key privKey) throws Exception {
FileInputStream fin;
FileOutputStream fout;
int nread = 0;
byte[] inbuf = new byte[1024];
fout = new FileOutputStream(out);
fin = new FileInputStream(in);
byte[] keyBlock = new byte[128];
nread = fin.read(keyBlock);
Cipher xCipher = Cipher.getInstance(Config.RSA_INSTANCE);
Cipher sCipher = Cipher.getInstance(Config.AES_INSTANCE);
// symmetric key/iv unwrapping step
xCipher.init(Cipher.DECRYPT_MODE, privKey);
Object[] keyIv = unpackKeyAndIV(xCipher.doFinal(keyBlock));
// decryption step
sCipher.init(Cipher.DECRYPT_MODE, (Key)keyIv[0], (IvParameterSpec)keyIv[1]);
while((nread = fin.read(inbuf)) >0) {
fout.write(sCipher.update(inbuf,0,nread));
}
fout.write(sCipher.doFinal());
fout.flush();
fin.close();
fout.close();
}
public static byte[] packKeyAndIv(Key key, IvParameterSpec ivSpec) throws IOException {
ByteArrayOutputStream bOut = new ByteArrayOutputStream();
bOut.write(ivSpec.getIV());
bOut.write(key.getEncoded());
return bOut.toByteArray();
}
public static Object[] unpackKeyAndIV(byte[] data) {
byte[] keyD = new byte[16];
byte[] iv = new byte[data.length - 16];
return new Object[] {
new SecretKeySpec(data, 16, data.length - 16, "AES"),
new IvParameterSpec(data, 0, 16)
};
}
Per edits and comments. Error was a corrupted privKey being passed into the decrypt function. Above code works fine.
try adding the following under your constructor -
Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider());