I have a existing c++ code which will encrypt a string. Now I did the same encryption in .
Some of the encrypted strings are matching . Some are mismatching in one or two characters.
I am unable to figure out why it is happening. I ran both the codes in debug mode until they call their libraries both have the same key, salt, iv string to be encrypted.
I know that even if a single byte padding change will modify encrypted string drastically. But here I am just seeing a one or two characters change. Here is a sample (Bold characters in between stars is the part mis matching)
java:
U2FsdGVkX18xMjM0NTY3OGEL9nxFlHrWvodMqar82NT53krNkqat0rrgeV5FAJFs1vBsZIJPZ08DJVrQ*Pw*yV15HEoyECBeAZ6MTeN+ZYHRitKanY5jiRU2J0KP0Fzola
C++:
U2FsdGVkX18xMjM0NTY3OGEL9nxFlHrWvodMqar82NT53krNkqat0rrgeV5FAJFs1vBsZIJPZ08DJVrQ*jQ*yV15HEoyECBeAZ6MTeN+ZYHRitKanY5jiRU2J0KP0Fzola
I am using AES encryption. provider is SunJCE version 1.6. I tried changing provider to Bouncy Castle. Even then result is same.
Added One More sample:
C++:
U2FsdGVkX18xMjM0NTY3O*I*/BMu11HkHgnkx+dLPDU1lbfRwb+aCRrwkk7e9dy++MK+/94dKLPXaZDDlWlA3gdUNyh/Fxv*oF*STgl3QgpS0XU=
java:
U2FsdGVkX18xMjM0NTY3O*D*/BMu11HkHgnkx+dLPDU1lbfRwb+aCRrwkk7e9dy++MK+/94dKLPXaZDDlWlA3gdUNyh/Fxv*j9*STgl3QgpS0XU=
UPDATE:
As per the comments I feel base 64 encryption is the culprit. I am using Latin-1 char set in both places. Anything else that I can check
Sigh!!
The problem almost certainly is that after you encrypt the data and receive the encrypted data as a byte string, you are doing some sort of character conversion on the data before sending it through Base-64 conversion.
Note that if you encrypt the strings "ABC_D_EFG" and "ABC_G_EFG", the encrypted output will be completely different starting with the 4th character, and continuing to the end. In other words, the Base-64 outputs would be something like (using made-up values):
U2FsdGVkX18xMj
and
U2FsdGXt91mJpz
The fact that, in the above examples, only two isolated Base-64 characters (one byte) are messed up in each case pretty much proves that the corruption occurs AFTER encryption.
The output of an encryption process is a byte sequence, not a character sequence. The corruption observed is consistent with erroneously interpreting the bytes as characters and attempting to perform a code page conversion on them, prior to feeding them into the Base-64 converter. The output from the encryptor should be fed directly into the Base-64 converter without any conversions.
You say you are using the "Latin-1 char set in both places", a clear sign that you are doing some conversion you should not be doing -- there should be no need to muck with char sets.
First a bit of code:
import javax.xml.bind.DatatypeConverter;
...
public static void main(String[] args) {
String s1j = "U2FsdGVkX18xMjM0NTY3OGEL9nxFlHrWvodMqar82NT53krNkqat0rrgeV5FAJFs1vBsZIJPZ08DJVrQPwyV15HEoyECBeAZ6MTeN+ZYHRitKanY5jiRU2J0KP0Fzola";
String s1c = "U2FsdGVkX18xMjM0NTY3OGEL9nxFlHrWvodMqar82NT53krNkqat0rrgeV5FAJFs1vBsZIJPZ08DJVrQjQyV15HEoyECBeAZ6MTeN+ZYHRitKanY5jiRU2J0KP0Fzola";
byte[] bytesj = DatatypeConverter.parseBase64Binary(s1j);
byte[] bytesc = DatatypeConverter.parseBase64Binary(s1c);
int nmax = Math.max(bytesj.length, bytesc.length);
int nmin = Math.min(bytesj.length, bytesc.length);
for (int i = 0; i < nmax; ++i) {
if (i >= nmin) {
boolean isj = i < bytesj.length;
byte b = isj? bytesj[i] : bytesc[i];
System.out.printf("%s [%d] %x%n", (isj? "J" : "C++"), i, (int)b & 0xFF);
} else {
byte bj = bytesj[i];
byte bc = bytesc[i];
if (bj != bc) {
System.out.printf("[%d] J %x != C++ %x%n", i, (int)bj & 0xFF, (int)bc & 0xFF);
}
}
}
}
This delivers
[60] J 3f != C++ 8d
Now 0x3f is the code of the question mark.
The error is, that 0x80 - 0xBF are in Latin-1, officially ISO-8859-1, control characters.
Windows Latin-1, officially Windows-1252, uses these codes for other characters.
Hence you should use "Windows-1252" or "Cp1252" (Code-Page) in Java.
Blundly
In the encryption the original bytes in the range 0x80 .. 0xBF were replaced with a question mark because of some translation to ISO-8859-1 instead of Windows-1252 to byte[].
Related
I have a bunch of old AES-encrypted Strings encrypted roughly like this:
String is converted to bytes with ISO-8859-1 encoding
Bytes are encrypted with AES
Result is converted to BASE64 encoded char array
Now I would like to change the encoding to UTF8 for new values (eg. '€' does not work with ISO-8859-1). This will of
course cause problems if I try to decrypt the old ISO-8859-1 encoded values with UTF-8 encoding:
org.junit.ComparisonFailure: expected:<!#[¤%&/()=?^*ÄÖÖÅ_:;>½§#${[]}<|'äöå-.,+´¨]'-Lorem ipsum dolor ...> but was:<!#[�%&/()=?^*����_:;>��#${[]}<|'���-.,+��]'-Lorem ipsum dolor ...>
I'm thinking of creating some automatic encoding fallback for this.
So the main question would be that is it enough to inspect the decrypted char array for '�' characters to figure out encoding mismatch? And what is the 'correct' way to declare that '�' symbol when comparing?
if (new String(utf8decryptedCharArray).contains("�")) {
// Revert to doing the decrypting with ISO-8859-1
decryptAsISO...
}
When decrypting, you get back the original byte sequence (result of your step 1), and then you can only guess whether these bytes denote characters according to the ISO-8859-1 or the UTF-8 encoding.
From a byte sequence, there's no way to clearly tell how it is to be interpreted.
A few ideas:
You could migrate all the old encrypted strings (decrypt, decode to string using ISO-8859-1, encode to byte array using UTF-8, encrypt). Then the problem is solved once and forever.
You could try to decode the byte array in both versions, see if one version is illegal, or if both versions are equal, and if it still is ambiguous, take the one with higher probability according to expected characters. I wouldn't recommend to go that way, as it needs a lot of work and still there's some probability of error.
For the new entries, you could prepend the string / byte sequence by some marker that doesn't appear in ISO-8859-1 text. E.g. some people follow the convention to prepend a Byte Order Marker at the beginning of UTF-8 encoded files. Although the resulting bytes (EF BB BF) aren't strictly illegal in ISO-8859-1 (being read as ), they are highly unlikely. Then, when your decrypted bytes start with EF BB BF, decode to string using UTF-8, otherwise using ISO-8859-1. Still, there's a non-zero probability of error.
If ever possible, I'd go for migrating the existing entries. Otherwise, you'll have to carry on with "old-format compatibility stuff" in your code base forever, and still can't absolutely guarantee correct behaviour.
When decoding bytes to text, don't rely on the � character to detect malformed input. Use a strict decoder. Here is a helper method for that:
static String decodeStrict(byte[] bytes, Charset charset) throws CharacterCodingException {
return charset.newDecoder()
.onMalformedInput(CodingErrorAction.REPORT)
.onUnmappableCharacter(CodingErrorAction.REPORT)
.decode(ByteBuffer.wrap(bytes))
.toString();
}
Here is the corresponding strict encoder helper method, in case you need it:
static byte[] encodeStrict(String str, Charset charset) throws CharacterCodingException {
ByteBuffer buf = charset.newEncoder()
.onMalformedInput(CodingErrorAction.REPORT)
.onUnmappableCharacter(CodingErrorAction.REPORT)
.encode(CharBuffer.wrap(str));
byte[] bytes = buf.array();
if (bytes.length == buf.limit())
return bytes;
return Arrays.copyOfRange(bytes, 0, buf.limit());
}
Since ISO-8859-1 allows all bytes, you can't use it to detect malformed input. UTF-8 is however validating, so it is very likely to detect malformed input. It is however not 100% guaranteed, but it's the best we get do.
So, try decoding using strict UTF-8, and then fall back to ISO-8859-1 if it fails:
static String decode(byte[] bytes) {
try {
return decodeStrict(bytes, StandardCharsets.UTF_8);
} catch (CharacterCodingException e) {
return new String(bytes, StandardCharsets.ISO_8859_1);
}
}
Test
System.out.println(decode("señor".getBytes(StandardCharsets.ISO_8859_1))); // prints: señor
System.out.println(decode("señor".getBytes(StandardCharsets.UTF_8))); // prints: señor
System.out.println(decode("€100".getBytes(StandardCharsets.UTF_8))); // prints: €100
Good evening!
In my android app the smartphones load a AES encrypted String from my server and store it in a variable. After that process the variable and a key are pass to a method which decrypt the string. My mistake is that german umlauts (ä, ü, ö) aren't correct decoded. All umlauts displayed as question marks with black background...
My Code:
public static String decrypt(String input, String key) {
byte[] output = null;
String newString = "";
try {
SecretKeySpec skey = new SecretKeySpec(key.getBytes(), "AES");
Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, skey);
output = cipher.doFinal(Base64.decode(input, Base64.DEFAULT));
newString = new String(output);
} catch(Exception e) {}
return newString;
}
The code works perfectly - only umlauts displayed not correctly, an example is that (should be "ö-ä-ü"):
How can I set the encoding of the decrypted String? In my iOS app I use ASCII to encoding the decoded downloaded String. That works perfectly! Android and iOS get the String from the same Server on the same way - so I think the problem is the local Code above.
I hope you can help me with my problem... Thanks!
There is no text but encoded text.
It seems like you are guessing at the character set and encoding—That's no way to communicate.
To recover the text, you need to reverse the original process applied to it with the parameters associated with each step.
For explanation, assume that the server is taking text from a Java String and sending it to you securely.
String uses the Unicode character set (specifically, Unicode's UTF-16 encoding).
Get the bytes for the String, using some specific encoding, say ISO8859-1. (UTF-8 could be better because it is also an encoding for the Unicode character set, whereas ISO8859-1 has a lot fewer characters.) As #Andy points out, exceptions are your friends here.
Encrypt the bytes with a specific key. The key is a sequence of bytes, so, if you are generating this from a string, you have to use a specific encoding.
Encode the encrypted bytes with Base64, producing a Java String (again, UTF-16) with a subset of characters so reduced that it can be re-encoded in just about any character encoding and placed in just about any context such as SMTP, XML, or HTML without being misinterpreted or making it invalid.
Transmit the string using a specific encoding. An HTTP header and/or HTML charset value is usually used to communicate which encoding.
To receive the text, you have to get:
the bytes,
the encoding from step 5,
the key from step 3,
the encoding from step 3 and
the encoding from step 2.
Then you can reverse all of the steps. Per your comments, you discovered you weren't using the encoding from step 2. You also need to use the encoding from step 3.
I essentially have the exact opposite problem as
new-line-appending-on-my-encrypted-string
It seems like the old Java Base64 utility would always add new lines every 76 characters when returning a string, but using the following code, I don't get those breaks I need.
Path path = Paths.get(file);
byte[] data = Files.readAllBytes(path);
String txt= Base64.getEncoder().encodeToString(data);
Is there an easy way to tell the encoder to add the newlines?
I've tried implementing a stringbuilder to insert the newlines, But it ends up changing the entire output (I copy the text from java console to HxD editor, and compare against my known working 'BLOB' with newlines).
String txt= Base64.getEncoder().encodeToString(data);
//Byte code for newline
byte b1 = 0x0D;
byte b2 = 0x0A;
StringBuilder sb = new StringBuilder();
for (int i = 0; i < txt.length(); i++) {
if (i > 0 && (i % 76 == 0)) {
sb.append((char)b1);
sb.append((char)b2);
}
sb.append(txt.charAt(i));
}
EDIT (in response to question):
It's not the easiest thing to explain, but when I don't use string builder, the output of the encode will start like this:
AAAAPAog4lBVgGJrT2b+mQVicHN3d////////3hhcDJiLWVtMjUwLWVtMjUwLWRldjA0NTUAAAAAAA
But I want it to look like this:
AAAAPAog4lBVgGJrT2b+mQVicHN3d////////3hhcDJiLWVtMjUwLWVtMjUwLWRldjA0NTUAAAAA..AA
As you can see, the ".." represents 0x0D and 0X0A or a newline, which is insterted at the 76th character (this is what the old base64 would output).
However, when I append the bytes b1 and b2 (newline) after the 76th character, the output becomes:
BPwAFHwA0CUFoG8AgDRCAAIlQgAAJUIAAhUfNEIAAiUkmw/0fADQFSInART/ADUlfADQFQE0fADQ..
So it looks like the ".." is in the right spot, but everything before it is different.
Thanks!
You want getMimeEncoder instead:
MIME
Uses the "The Base64 Alphabet" as specified in Table 1 of RFC 2045 for encoding and decoding operation. The encoded output must be represented in lines of no more than 76 characters each and uses a carriage return '\r' followed immediately by a linefeed '\n' as the line separator. No line separator is added to the end of the encoded output. All line separators or other characters not found in the base64 alphabet table are ignored in decoding operation.
(emphasis mine)
Note that the encoding scheme is otherwise the same as the basic encoder from getEncoder - they are both derived from RFC 2045.
Today I splitted Base64 representation of X509Certificate with the following code:
StringBuilder sb = new StringBuilder();
int chunksCount = str.length()/76;
for(int i=0;i<chunksCount;i++){
sb.append(str.substring(76*i,76*(i+1))).append("\r\n");
}
if(str.length() % 76 != 0) sb.append(str.substring(76*chunksCount)).append("\r\n");
I think, adding big parts better than iterating over each letter. Also, some libraries provide Base64 encoder with special parameter allowing to split with required part size but I had to use some library without such feature.
I have an API spec that reads:
Encryption Algorithm
The API utilizes the AES-128 (also known as Rijndael-128) algorithm with a 192-bit key in
CBC mode for encryption and decryption of sensitive pieces of information – the password parameter in the user/signin and user/signup methods, the authentication token, etc. The steps of the algorithm are listed below:
Encryption
Pad the input data so that its size is a multiple of the encryption algorithm block size – 16 bytes. In case the length of input data is a multiple of 16, a block of additional 16 bytes needs to be appended. The value of each pad byte is the number of pad bytes as an 'unsigned char'. That is, the last byte of the padded data should always be between 0x01 and 0x10.
Generate a 16-byte long initialization vector (IV) for the encryption algorithm.
Encrypt the padded data using AES-128 with the EK and the generated IV.
Combine the IV with the encrypted data.
Encode the result with urlsafe Base64. The urlsafe Base46 alphabet uses '–' instead of '+' and '_' instead of '/'.
Decryption
Base64-decode the input data.
Extract the first 16 bytes – these are the IV for the AES algorithm.
Decrypt the data using AES-128 with the EK and IV.
Read the value of the last byte of the decrypted data and remove that many bytes off its tail.
The only example provided by the supplier of this API is in PHP, using mcrypt. I know absolutely nothing about PHP, and am not an encryption expert. I need to be able to represent the above algorithm using ColdFusion 10.
I started by trying to take the example PHP files and find equivalents in either the ColdFusion tag or function library, and then by looking for a Java library with the same interface. I just don't know enough to make this happen.
Is there someone here that can point me in the right direction, or work with me offline to assist?
EDIT:
Here's the example given, for the basic task of doing a "check" on the keys (partner key and encryption key) provided to me for use with the API.
Object Client.php, has this constructor:
public function __construct($hostApiUrl, $partnerKey, $encryptionKey, $token = null)
{
$this->_pk = $partnerKey;
$this->_ek = $encryptionKey;
$this->_crypt = new Crypt($encryptionKey);
$this->_url = rtrim($hostApiUrl, '/') . self::BASE_URL;
if ($token) {
$this->setUserSession($token);
}
}
and this is the function I'm attempting to use:
public function checkKeys()
{
$secret = $this->_encodeParam($this->_ek);
$result = $this->call('partner/checkkeys', array(
'secret' => $secret
));
if (!$result || !$this->_isCodeOk($result->code)) {
return false;
}
return true;
}
So the client object already has the partner key and encryption key when this method is called, obviously.
so the "secret" is created by "encoding" the encryption key provided, using _encodeParam() method. that looks like this:
protected function _encodeParam($secret)
{
$secret = "{$secret}:{$this->_pk}";
return $this->_crypt->encrypt($secret);
}
so the secret is appended with the partner key. and then encrypted using this method in the crypt object (AES_BLOCK_SIZE is set as 16):
public function encrypt($data)
{
$pad = self::AES_BLOCK_SIZE - strlen($data) % self::AES_BLOCK_SIZE;
$data .= str_repeat(chr($pad), $pad);
if (stristr(PHP_OS, 'win') !== false) {
$random_source = MCRYPT_RAND;
} else {
$random_source = MCRYPT_DEV_URANDOM;
}
$iv = mcrypt_create_iv(self::AES_BLOCK_SIZE, $random_source);
mcrypt_generic_init($this->_td, $this->_key, $iv);
$data = $iv . mcrypt_generic($this->_td, $data);
mcrypt_generic_deinit($this->_td);
return self::urlsafe_b64encode($data);
}
this is returned back to the above checkKeys() function which sends the request to the API, which then returns a response. That actual API call is a POST which is easy enough to generate of course, but all those encryption hoops, including the MCRYPT library calls, are where I get stuck trying to determine the equivalent in CF10 or Java.
If I were to get an example thus far, I think I'd stand a chance of replicating the other functions in the crypt object (the ones that are even necessary, which may not be, since some may be built right into the CF encrypt() and decrypt() functions). This seems like a reasonable starting point, however.
I have been tearing my hair out all day trying to solve this...
I have an objective-c client running on the iPhone, connecting to a Java server. The iPhone is encrypting data using AES but I cannot decrypt it on the server. I am using a known passphrase and message (single string) and am generating the byte array on the iPhone, generating a comparison byte array on the Java server using the same key and message but the byte arrays are completely different (and hence can't be decoded on the Java side).
The client is using the CommonCrypto library with the following settings...
Data is an NSData holding the word "message" using dataUsingEncoding:NSASCIIStringEncoding
Key is an NSData holding the phrase "1234567891123456" again using the encoding as above.
Algorithm is kCCAlgorithmAES128
Options is kCCOptionsPKCS7Padding (which I believe equates to ECB on the server?!)
The server is using the following code...
byte[] key = "1234567891123456".getBytes();
Cipher c = Cipher.getInstance("AES/ECB/PKCS5Padding");
SecretKeySpec k = new SecretKeySpec(key, "AES");
c.init(Cipher.ENCRYPT_MODE, k);
byte[] encryptedData = c.doFinal("message".getBytes());
BUT the data in encryptedData does not match that which is being generated in the objective-c code, the byte arrays are completely different.
Can anyone see anything obvious I am doing wrong? I think the settings are all the same... :(
UPDATE - As requested....
Ok so here goes....
iPhone client is encrypting the following string "message"
It uses the key "1234567891123456"
It uses an initialisation vector of "1010101010101010"
It is using AES128, with CBC mode (as far as I can tell) and options of kCCOptionsPKCS7Padding.
The result of the encryption (with base64 encoding) is UHIYllDFAXl81ZM7OZPAuA==
The server is encrypting the same string, with the same key and initialisation vector.
It is using the following Cipher.getInstance("AES/CBC/PKCS5Padding");
The result of the encryption (with base64 encoding) is ALBnFIHysLbvAxjvtNo9vQ==
Thanks.
UPDATE 2 - As requested...
Here is the iPhone code....
NSData *toencrypt = [#"message" dataUsingEncoding:NSASCIIStringEncoding];
NSData *pass = [#"1234567891123456" dataUsingEncoding:NSASCIIStringEncoding];
NSData *iv = [#"1010101010101010" dataUsingEncoding:NSASCIIStringEncoding];
CCCryptorStatus status = kCCSuccess;
NSData *encrypted = [toencrypt dataEncryptedUsingAlgorithm:kCCAlgorithmAES128 key:pass initializationVector:iv options:kCCOptionPKCS7Padding error:&status];
NSString *text = [NSString base64StringFromData:encrypted length:[encrypted length]];
The NSData category for encrypting comes from here...
http://github.com/AlanQuatermain/aqtoolkit/tree/master/CommonCrypto/
Incidentally, I have checked the byte arrays that are in toencrypt, pass and iv and they match those that are on the server.
This is not my area but it looks like on the client you have PKCS7 but on the server you have PKCS5.
I just ran across the exact same problem. I am using CommonCrypto on the iOS client, using settings:
NSData * encrypted = [data dataEncryptedUsingAlgorithm:kCCAlgorithmAES128 key:pass initializationVector:iv options:kCCOptionPKCS7Padding error:&status];
The server uses Cipher.getInstance("AES/CBC/PKCS5Padding"); with the same key and initialization vector as the client.
After banging my head against the wall for the last few hours, I finally followed Jason's advice and checked the dataEncryptedUsingAlgorithm routine and printed out keyData right after FixKeyLengths. It turns out my 128bit key was extended to 192bit with 0s added to the end. After fixing this, everything works properly. :)
Update: My answer was posted almost 2 years ago, and this issue seems to be fixed in the latest NSData+CommonCrypto code. Specifically, this was the part that caused the issue:
static void FixKeyLengths( CCAlgorithm algorithm, NSMutableData * keyData, NSMutableData * ivData )
{
NSUInteger keyLength = [keyData length];
switch ( algorithm )
{
case kCCAlgorithmAES128:
{
if ( keyLength <= 16 )
{
[keyData setLength: 16];
}
else if ( keyLength <= 24 )
{
[keyData setLength: 24];
}
else
{
[keyData setLength: 32];
}
break;
}
The first check keyLength <= 16 wasn't there before.
If you are still experiencing problems now, it's probably something else.
What mode is the iPhone using with AES? You don't list anything, so perhaps that means it's using no chaining (ECB).
However, on the Java side, you are using CBC, but not specifying an initialization vector. That is definitely wrong. If you really are using CBC, you have to have the IV that was used during encryption. The IV is not secret; it can be sent along with the ciphertext.
If you are really using ECB, there is no IV, but your Java specifies the wrong mode.
Based on your samples, the server is doing it right, and the client is not.
Looking at the data, I would guess that the key is wrong. Please show us the iPhone code, especially the code to go from "1234567891123456" to your key.
I recently ran across this in another project. The problem was that the key was one byte too long to fit into the char buffer inside of the dataEncryptedUsingAlgorithm method.
The problem was that the getBytes method on the NSString was soft-failing. It would copy most of the string into the buffer, but then since the key was one byte too long, it would "mark" the operation as failed by setting the first char to NUL (char 0).
Step into that method in Xcode and see what your key char[16] buffer looks like. It may have this same problem, and have the contents { 0, '2', '3', '4', ... }.