I have an encryption algorithm (AES) that accepts a file converted to array byte and encrypt it.
Since I am going to process a very large files, the JVM may go out of memory.
I am planing to read the files in multiple byte arrays, each containing some part of the file. Then I iteratively feed the algorithm. Finally, I merge them to produce an encrypted file.
So my question is: Is there any way to read a file part by part to multiple byte arrays?
I thought I could use the following to read the file to a byte array:
IOUtils.toByteArray(InputStream input).
And then split the array into multiple bytes using:
Arrays.copyOfRange()
But I am afraid that the code that reads a file to ByteArray will make the JVM to go out of memory.
Look up cipher streams in Java. You can use them to encrypt/decrypt streams on the fly so you don't have to store the whole thing in memory. All you have to do is copy the regular FileInputStream for your source file to the CipherOutputStream that's wrapping your FileOutputStream for the encrypted sink file. IOUtils even conveniently contains a copy(InputStream, OutputStream) method to do this copy for you.
For example:
public static void main(String[] args) {
encryptFile("exampleInput.txt", "exampleOutput.txt");
}
public static void encryptFile(String source, String sink) {
FileInputStream fis = null;
try {
fis = new FileInputStream(source);
CipherOutputStream cos = null;
try {
cos = new CipherOutputStream(new FileOutputStream(sink), getEncryptionCipher());
IOUtils.copy(fis, cos);
} finally {
if (cos != null)
cos.close();
}
} finally {
if (fis != null)
fis.close();
}
}
private static Cipher getEncryptionCipher() {
// Create AES cipher with whatever padding and other properties you want
Cipher cipher = ... ;
// Create AES secret key
Key key = ... ;
cipher.init(Cipher.ENCRYPT_MODE, key);
}
If you need to know the number of bytes that were copied, you can use IOUtils.copyLarge instead of IOUtils.copy if the file sizes exceed Integer.MAX_VALUE bytes (2 GB).
To decrypt the file, do the same thing, but use CipherInputStream instead ofCipherOutputStream and initialize your Cipher using Cipher.DECRYPT_MODE.
Take a look here for more info on cipher streams in Java.
This will save you space because you won't need to store byte arrays of your own anymore. The only stored byte[] in this system is the internal byte[] of the Cipher, which will get cleared each time enough input is entered and an encrypted block is returned by Cipher.update, or on Cipher.doFinal when the CipherOutputStream is closed. However, you don't have to worry about any of this since it's all internal and everything is managed for you.
Edit: note that this can result in certain encryption exceptions being ignored, particularly BadPaddingException and IllegalBlockSizeException. This behavior can be found in the CipherOutputStream source code. (Granted, this source is from the OpenJDK, but it probably does the same thing in the Sun JDK.) Also, from the CipherOutputStream javadocs:
This class adheres strictly to the semantics, especially the failure semantics, of its ancestor classes java.io.OutputStream and java.io.FilterOutputStream. This class has exactly those methods specified in its ancestor classes, and overrides them all. Moreover, this class catches all exceptions that are not thrown by its ancestor classes.
The bolded line here implies that the cryptographic exceptions are ignored, which they are. This may cause some unexpected behavior while trying to read an encrypted file, especially for block and/or padding encryption algorithms like AES. Make a mental note of this that you will get zero or partial output for the encrypted (or decrypted for CipherInputStream) file.
If you're using IOUtils, perhaps you should consider IOUtils.copyLarge()
public static long copyLarge(InputStream input,
OutputStream output,
long inputOffset,
long length)
and specify a ByteArrayOutputStream as the output. You can then iterate through and load sections of your file using offset/length.
From the doc:
Copy some or all bytes from a large (over 2GB) InputStream to an
OutputStream, optionally skipping input bytes.
Related
I'm not well aware with Java KeyStore. What I want to do is to have an encrypted structure to store my keys.
I've multiple clusters and there exists a key associated with every cluster & now I want to store those keys securely such that they are all encrypted using single main key (for an instance, 'loginid')
I wandered alot in search of this issue and somewhere on stackoverflow itself someone suggested about Java keyStore to store SecretKey (Symmetric Encryption). I read its documentation & found it perfect as per my requirements but couldn't understand its implementation properly.
Here is a code snippet I'm working on -
public class Prac {
public static void main(String[] args) throws KeyStoreException, FileNotFoundException, IOException, NoSuchAlgorithmException, CertificateException, UnrecoverableKeyException, UnrecoverableEntryException {
KeyStore ks = KeyStore.getInstance("JCEKS");
char[] ksPwd = "yashkaranje98".toCharArray();
ks.load(null, ksPwd);
KeyStore.ProtectionParameter protParam = new KeyStore.PasswordProtection(ksPwd);
javax.crypto.SecretKey mySecretKey = new SecretKeySpec("_anky!#ubn#$0e41".getBytes(),"AES");
KeyStore.SecretKeyEntry skEntry = new KeyStore.SecretKeyEntry(mySecretKey);
ks.setEntry("cluster1", skEntry, protParam);
java.io.FileOutputStream fos = null;
try {
fos = new java.io.FileOutputStream("keystore.ks");
ks.store(fos, ksPwd);
} finally {
if (fos != null) {
fos.close();
}
}
java.io.FileInputStream fis = null;
try {
ks.load(new FileInputStream("keystore.ks"), ksPwd);
} finally {
if (fis != null) {
fis.close();
}
}
SecretKey key = (SecretKey)ks.getKey("cluster1", ksPwd);
String encodedKey = Base64.getEncoder().encodeToString(key.getEncoded());
System.out.println(encodedKey);
}
}
Alias: "cluster1"
Key to store: _anky!#ubn#$0e41
Protection Parameter: yashkaranje98
It prints: X2Fua3khQHVibiMkMGU0MQ==
What I expect is key itsef: _anky!#ubn#$0e41
Kindly please let me know what I'm missing...but before please tell me what I'm expecting is it even legit? or does it make sense?
(I am still learning about this KeyStore concept so there might be some silly mistakes.)
A secret AES key consists of random bytes. Such a key should not be printed directly, because the bytes may not represent valid characters or they may present control characters that don't print on screen. If you'd copy them then you might miss data. If you print them in the wrong terminal you may send terminal control codes.
Because of this you need to print out key values as hexadecimals or base 64. Normally for symmetric keys hex is preferred as it is easy to see the contents and size from the hex (the size in bytes is half that of the hex size, the size in bits is 4 times the hex size as each hex digit represents a 4 bit nibble). However, as Java still lacks a good Hex encoder, base 64 is also a good option.
Of course, in that case, to compare, you should also decode it from base 64 before you insert it into the key store.
Also beware that you don't specify the character encoding when you call getBytes on the string. If you would use higher valued characters then you may get different results on various systems, as getBytes without argument assumes the platform encoding. Specifying StandardCharsets.UTF_8 usually makes more sense.
Of course, as keys should contain random bytes, the getBytes method needs to go entirely, but you should keep this in mind anyway.
When I look at the code it seems you've missed the last 10 years of Java progress. No var, no null avoidance, missing imports, and no try-with-resources. That's a shame, because those would make your code a lot more readable. It's valid, mind you, but yeah...
I was given a public key id, and being asked to use this public key id to encrypt a .txt file. I can find a reference on how to perform this, but in C# language using Bouncycastle and nothing with Java.
The specific public key id is provided by the encrypted-file recipient. Decryption is done by the encrypted-file recipient, therefore I have no concern on any decryption or private key id, if these info are essential which is beyond my knowledge for time being.
I am using Java and very new with encryption, please direct me to any LATEST java application examples or tutorial that encrypt a text file using specific public key id given instead of generated ones. Thanks!
You can use „raw“ Bouncy Castle for Java or one of several wrappers around the API. For using Bouncy Castle you need to understand the OpenPGP RFC (rfc4880).
Alternatively you can use existing wrappers like e.g. Bouncy GPG:
final String original_message = "I love deadlines. I like the whooshing sound they make as they fly by. Douglas Adams";
// Most likely you will use one of the KeyringConfigs.... methods.
// These are wrappers for the test.
KeyringConfig keyringConfigOfSender = Configs
.keyringConfigFromResourceForSender();
ByteArrayOutputStream result = new ByteArrayOutputStream();
try (
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(result, 16384 * 1024);
final OutputStream outputStream = BouncyGPG
.encryptToStream()
.withConfig(keyringConfigOfSender)
.withStrongAlgorithms()
.toRecipients("recipient#example.com", "sender#example.com")
.andSignWith("sender#example.com")
.binaryOutput()
.andWriteTo(bufferedOutputStream);
// Maybe read a file or a webservice?
final ByteArrayInputStream is = new ByteArrayInputStream(original_message.getBytes())
) {
Streams.pipeAll(is, outputStream);
// It is very important that outputStream is closed before the result stream is read.
// The reason is that GPG writes the signature at the end of the stream.
// This is triggered by closing the stream.
// In this example outputStream is closed via the try-with-resources mechanism of Java
}
result.close();
byte[] chipertext = result.toByteArray();
I'm working on a project to Encrypt/Decrypt files. as this is my first time, I'm wondering if I'm doing it right or not. till now, my idea about encrypting is this :
Select a file -> Read all its bytes and add it to byte array -> Encrypt the byte array -> write encrypted bytes to same file.
note that in this project output file is same file as input. So I decided to clear file before writing encrypted bytes to it.
This might be stupid (and thats why I'm asking for help), so here is my way
public class Encryptor {
File file;
SecretKeySpec secretKeySpec;
public void setFile(String filePath) throws Exception {
this.file = new File(filePath);
if(!file.isFile()){
throw new Exception("The file you choosed is not valid");
}
}
public void setKey(String keyword){
try {
MessageDigest sha = MessageDigest.getInstance("SHA-256");
sha.update(keyword.getBytes("UTF-8"));
byte[] key = sha.digest();
secretKeySpec = new SecretKeySpec(key, "AES");
} catch (UnsupportedEncodingException | NoSuchAlgorithmException e) {
e.printStackTrace();
}
}
public void encrypt(){
byte[] bFile = new byte[(int) file.length()];
try {
//adding portocol bytes to the file bytes
//String portcol = "encryptor portocol";
//byte[] decPortocol = portcol.getBytes();
//convert file into array of bytes
BufferedInputStream bufferedInputStream = new BufferedInputStream(new FileInputStream(file));
bufferedInputStream.read(bFile);
bufferedInputStream.close();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
//outputStream.write(decPortocol);
outputStream.write(bFile);
byte[] cryptedFileBytes = outputStream.toByteArray();
//Cipher and encrypting
Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, secretKeySpec);
byte[] encryptedBytes = cipher.doFinal(cryptedFileBytes);
//Write Encrypted File
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(new FileOutputStream(file,false));
bufferedOutputStream.write(encryptedBytes);
bufferedOutputStream.flush();
bufferedOutputStream.close();
}catch (Exception e){
e.printStackTrace();
}
}
}
main question
are there other ways to read-encrypt-write on same file together at same time? like reading bytes part by part and at same time encrypting that part and overwrite it with encrypted bytes.
Can You Help me more ?
And also Any information about how to make my encrypted files more safe can also be helpful.
and does my program kill RAM ?!
(NOTE) I'm writing encrypted data on same file for some reasons. I'm not much familiar with how hard drive works. One of my reasons is to prevent file from being recovered later. is there anything I have to know about that ? does what I'm doing prevent unEncrypted file to be recovered later ?
EDIT
#erickson has pointed out something important in his answer. I got to know that this way of encrypting a file is not safe. What I was considering to prevent too, was preventing file from being recovered later. I mean there is no point to encrypt a file and keep it in your hard drive if you once had it unEncrypted there ! in my experience, everytime I recovered a file, I reached last edits of it and I could never get history of changes. I thought this must be the same if I was not wrong in first place. How can I help preventing data recovery then ?!
Writing to a file while reading can work, but it would be easy to introduce a bug that would corrupt the file. For safety's sake, it might be better to write to a temporary file, then delete the original file and replace it with the temporary file. That way, all of the file content is always safely in at least one file.
One caveat about this is that if you encrypt an existing file, there's no guarantee that the original file isn't still recorded on disk. Even if you write to the same file as you read, whether the same storage is overwritten with encrypted data will depend on the underlying file system.
It would be better if the original file was written in its encrypted form. Even if the writing application doesn't support encryption, most operating systems support the creation of an encrypted file system so that any application can keep files secret.
You need to close your reader after you have finished reading the file. You are currently doing it in this line:
bufferedInputStream.close();
So it's ok.
Then, instead of clearing file, you can just simply overwrite it using:
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(new FileOutputStream(filename, false);
Hope that helps :)
I've got this function that encrypts a file, the encryption bit seems to be working, but i can't get it to overwrite the current file.
FileInputStream inputStream = new FileInputStream(input); // Selects file to encrypt
cipher.init(Cipher.ENCRYPT_MODE, secret, ivSpec); // Sets up the encryption
// Creates an the output stream, the encryption is performed here
CipherOutputStream cos = new CipherOutputStream(new FileOutputStream(input + ".secure"), cipher);
byte[] block = new byte[8];
int i;
while ((i = inputStream.read(block)) != -1) // Reads the file
{
cos.write(block, 0, i); // Writes the new file
}
cos.close();
This is working fine, i end up with an encrypted file with original_file_name.txt.secure, but i want it to overwrite the original file. If i remove the .secure bit it doesn't write the file properly.
How can I overwrite the file original file with the encrypted text?
If you remove the .secure part, you'll be trying to read from the file at the same time that you're writing to it. This is not a very good idea...
The best approach would be to do as you've done, and then if all has gone well, you can delete the original file and rename the old one to match its name, using Files.move().
In fact, if you pass the right options to Files.move(), you can get it to overwrite the existing file, meaning that you won't need to delete the original first.
This solves the simultaneous read/write problem you're having, but it's also a whole lot safer for an application like this. If your application crashes or there's a power cut in the middle of encrypting, and you're encrypting in place, then you're completely screwed. If you do it this way, then power failure in the middle still leaves you with your old file intact. You'll always have the complete old file around until the complete new file is ready.
By the way, you should make use of a BufferedInputStream too, rather than just using a raw FileInputStream. And I can't see an inputStream.close() anywhere.
Is there a way to load a crypted file (that is a jar file), decrypted it (so obtains the real byte array) and use it without create a file on filesystem and launch it?
byte[] jarBytes=Decrypter.decrypt("my\\encrypted\\jar\\file");
//use jarBytes to execute an application without create a real File with this bytes...
Sure.
The input stream can come from a resource (i.e., on the classpath). The bytes can be manipulated as required.
See this post for an example of loading the jar from a directory. Your version would be almost identical, except for where the bytes come from originally, and with an additional transformation step.
If anybody is that interested in your code they'll either just a) reverse-engineer your class loader, or b) just use your class loader as-is and not worry about reverse-engineering the encrypted code... but good luck anyway :)
Like this?
Cipher c = Cipher.getInstance(ALGORITHM);
c.init(Cipher.DECRYPT_MODE, key);
byte[] decValue = c.doFinal(data);