Develop Reference

Replicating Java password hashing code in Python (PBKDF2WithHmacSHA1)

I have been trying to replicate the java password authenticate to python, however the resulted hash is different.
password: abcd1234
password token (java): $31$16$sWy1dDEx52vwQUCswXDYMQMzTJC39g1_nmrK384T4-w
generated password token (python): $pbkdf2$16$c1d5MWRERXg1MnZ3UVVDcw$qPQvE4QbrnYJTmRXk0M7wlfhH5U
From the Java code, the Iteration is 16, SALT should the first 16 char in sWy1dDEx52vwQUCswXDYMQMzTJC39g1_nmrK384T4-w, which is sWy1dDEx52vwQUCs and the hash should be wXDYMQMzTJC39g1_nmrK384T4-w
however, applying the variables to python gave me a different hash result which, qPQvE4QbrnYJTmRXk0M7wlfhH5U which is different from Java's hash.
Where did i missed out?
Java:
private static final String ALGORITHM = "PBKDF2WithHmacSHA1";
private static final int SIZE = 128;
private static final Pattern layout = Pattern.compile("\\$31\\$(\\d\\d?)\\$(.{43})");
public boolean authenticate(char[] password, String token)
{
Matcher m = layout.matcher(token);
if (!m.matches())
throw new IllegalArgumentException("Invalid token format");
int iterations = iterations(Integer.parseInt(m.group(1)));
byte[] hash = Base64.getUrlDecoder().decode(m.group(2));
byte[] salt = Arrays.copyOfRange(hash, 0, SIZE / 8);
byte[] check = pbkdf2(password, salt, iterations);
int zero = 0;
for (int idx = 0; idx < check.length; ++idx)
zero |= hash[salt.length + idx] ^ check[idx];
return zero == 0;
}
Python:
from passlib.hash import pbkdf2_sha1
def hasher(password):
size = 128
key0 = "abcd1234"
iter = int(password.split("$")[2])
salt0 = password.split("$")[3][0: 16]
hash = pbkdf2_sha1.using(rounds=iter, salt = salt0.encode()).hash(key0)
print(hash.split('$')[4])
return hash
Original Link for Java code: How can I hash a password in Java?

There's a bunch of things different between how that java code does things, and how passlib's pbkdf2_sha1 hasher does things.
The java hash string contains a log cost parameter, which needs passing through 1<<cost to get the number of rounds / iterations.
The salt+digest needs to be base64 decoded, then take the first 16 bytes as the salt (which actually corresponds to first 21 1/3 characters of base64 data).
Similarly, since the digest's bits start in the middle of a base64 character, when the salt+digest is decoded, and digest is then encoded separately, the base64 string would be
AzNMkLf2DX-easrfzhPj7A (noticably different from the original encoded string).
Based on that, the following bit of code converts a java hash into the format used by pbkdf1_sha1.verify:
from passlib.utils.binary import b64s_decode, ab64_encode
def adapt_java_hash(jhash):
_, ident, cost, data = jhash.split("$")
assert ident == "31"
data = b64s_decode(data.replace("_", ".").replace("-", "+"))
return "$pbkdf2$%d$%s$%s" % (1<<int(cost), ab64_encode(data[:16]),
ab64_encode(data[16:]))
>>> adapt_java_hash("$31$16$sWy1dDEx52vwQUCswXDYMQMzTJC39g1_nmrK384T4-w")
'$pbkdf2$65536$sWy1dDEx52vwQUCswXDYMQ$AzNMkLf2DX.easrfzhPj7A'
The resulting string should be suitable for passing into pbkdf2_sha1.verify("abcd1234", hash); except for one issue:
The java code truncates the sha1 digest to 16 bytes, rather than the full 20 bytes; and way passlib's hasher is coded, the digest must be the full 20 bytes.
If you alter the java code to use SIZE=160 instead of SIZE=128, running the hash through the above adapt() function should then work in passlib.

Java Mac HMAC vs C++ OpenSSL hmac

This is going to be a long question but I have a really weird bug. I use OpenSSL in C++ to compute a HMAC and compare them to a simular implementation using javax.crypto.Mac. For some keys the HMAC calculation is correct and for others there is a difference in HMAC. I believe the problem occurs when the keys get to big. Here are the details.
Here is the most important code for C++:
void computeHMAC(std::string message, std::string key){
unsigned int digestLength = 20;
HMAC_CTX hmac_ctx_;
BIGNUM* key_ = BN_new();;
BN_hex2bn(&key_, key);
unsigned char convertedKey[BN_num_bytes(key_)];
BIGNUM* bn = BN_new();
HMAC_CTX_init(&hmac_ctx_);
BN_bn2bin(bn, convertedKey);
int length = BN_bn2bin(key_, convertedKey);
HMAC_Init_ex(&hmac_ctx_, convertedKey, length, EVP_sha1(), NULL);
/*Calc HMAC */
std::transform( message.begin(), message.end(), message.begin(), ::tolower);
unsigned char digest[digestLength];
HMAC_Update(&hmac_ctx_, reinterpret_cast<const unsigned char*>(message.c_str()),
message.length());
HMAC_Final(&hmac_ctx_, digest, &digestLength);
char mdString[40];
for(unsigned int i = 0; i < 20; ++i){
sprintf(&mdString[i*2], "%02x", (unsigned int)digest[i]);
}
std::cout << "\n\nMSG:\n" << message << "\nKEY:\n" + std::string(BN_bn2hex(key_)) + "\nHMAC\n" + std::string(mdString) + "\n\n";
}
The java test looks like this:
public String calculateKey(String msg, String key) throws Exception{
HMAC = Mac.getInstance("HmacSHA1");
BigInteger k = new BigInteger(key, 16);
HMAC.init(new SecretKeySpec(k.toByteArray(), "HmacSHA1"));
msg = msg.toLowerCase();
HMAC.update(msg.getBytes());
byte[] digest = HMAC.doFinal();
System.out.println("Key:\n" + k.toString(16) + "\n");
System.out.println("HMAC:\n" + DatatypeConverter.printHexBinary(digest).toLowerCase() + "\n");
return DatatypeConverter.printHexBinary(digest).toLowerCase();
}
Some test runs with different keys (all strings are interpreted as hex):
Key1:
736A66B29072C49AB6DC93BB2BA41A53E169D14621872B0345F01EBBF117FCE48EEEA2409CFC1BD92B0428BA0A34092E3117BEB4A8A14F03391C661994863DAC1A75ED437C1394DA0741B16740D018CA243A800DA25311FDFB9CA4361743E8511E220B79C2A3483FCC29C7A54F1EB804481B2DC87E54A3A7D8A94253A60AC77FA4584A525EDC42BF82AE2A1FD6E3746F626E0AFB211F6984367B34C954B0E08E3F612590EFB8396ECD9AE77F15D5222A6DB106E8325C3ABEA54BB59E060F9EA0
Msg:
test
Hmac OpenSSL:
b37f79df52afdbbc4282d3146f9fe7a254dd23b3
Hmac Java Mac:
b37f79df52afdbbc4282d3146f9fe7a254dd23b3
Key 2: 636A66B29072C49AB6DC93BB2BA41A53E169D14621872B0345F01EBBF117FCE48EEEA2409CFC1BD92B0428BA0A34092E3117BEB4A8A14F03391C661994863DAC1A75ED437C1394DA0741B16740D018CA243A800DA25311FDFB9CA4361743E8511E220B79C2A3483FCC29C7A54F1EB804481B2DC87E54A3A7D8A94253A60AC77FA4584A525EDC42BF82AE2A1FD6E3746F626E0AFB211F6984367B34C954B0E08E3F612590EFB8396ECD9AE77F15D5222A6DB106E8325C3ABEA54BB59E060F9EA0
Msg:
test
Hmac OpenSSL:
bac64a905fa6ae3f7bf5131be06ca037b3b498d7
Hmac Java Mac:
bac64a905fa6ae3f7bf5131be06ca037b3b498d7
Key 3: 836A66B29072C49AB6DC93BB2BA41A53E169D14621872B0345F01EBBF117FCE48EEEA2409CFC1BD92B0428BA0A34092E3117BEB4A8A14F03391C661994863DAC1A75ED437C1394DA0741B16740D018CA243A800DA25311FDFB9CA4361743E8511E220B79C2A3483FCC29C7A54F1EB804481B2DC87E54A3A7D8A94253A60AC77FA4584A525EDC42BF82AE2A1FD6E3746F626E0AFB211F6984367B34C954B0E08E3F612590EFB8396ECD9AE77F15D5222A6DB106E8325C3ABEA54BB59E060F9EA0
Msg:
test
Hmac OpenSSL:
c189c637317b67cee04361e78c3ef576c3530aa7
Hmac Java Mac:
472d734762c264bea19b043094ad0416d1b2cd9c
As the data shows, when the key gets to big, an error occurs. If have no idea which implementation is faulty. I have also tried with bigger keys and smaller keys. I haven't determined the exact threshold. Can anyone spot the problem? Is there anyone capable of telling me which HMAC is incorrect in the last case by doing a simulation using different software or can anyone tell me which 3rd implementation I could use to check mine?
Kind regards,
Roel Storms

When you convert a hexadecimal string to a BigInt in Java, it assumes the number is positive (unless the string includes a - sign).
But the internal representation of it is twos-complement. Meaning that one bit is used for the sign.
If you are converting a value that starts with a hex between 00 and 7F inclusive, then that's not a problem. It can convert the byte directly, because the leftmost bit is zero, which means that the number is considered positive.
But if you are converting a value that starts with 80 through FF, then the leftmost bit is 1, which will be considered negative. To avoid this, and keep the BigInteger value exactly as it is supplied, it adds another zero byte at the beginning.
So, internally, the conversion of a number such as 7ABCDE is the byte array
0x7a 0xbc 0xde
But the conversion of a number such as FABCDE (only the first byte is different!), is:
0x00 0xfa 0xbc 0xde
This means that for keys that begin with a byte in the range 80-FF, the BigInteger.toByteArray() is not producing the same array that your C++ program produced, but an array one byte longer.
There are several ways to work around this - like using your own hex-to-byte-array parser or finding an existing one in some library. If you want to use the one produced by BigInteger, you could do something like this:
BigInteger k = new BigInteger(key, 16);
byte[] kByteArr = k.toByteArray();
if ( kByteArr.length > (key.length() + 1) / 2 ) {
kByteArr = Arrays.copyOfRange(kByteArr,1,kByteArr.length);
}
Now you can use the kByteArr to perform the operation properly.
Another issue you should watch out for is keys whose length is odd. In general, you shouldn't have a hex octet string that has an odd length. A string like F8ACB is actually 0F8ACB (which is not going to cause an extra byte in BigInteger) and should be interpreted as such. This is why I wrote (key.length() + 1) in my formula - if key is odd-length, it should be interpreted as a one octet longer. This is also important to watch out for if you write your own hex-to-byte-array converter - if the length is odd, you should add a zero at the beginning before you start converting.

Negative numbers in APDU command

I am writing Java Applet. I need to upload pre-generated RSAPrivateCrtKey to applet with APDU Command.
In my computer I am generating key. Next I serialize this key in privateKeyBuffer:
keyPair = new KeyPair(KeyPair.ALG_RSA_CRT, KeyBuilder.LENGTH_RSA_512);
keyPair.genKeyPair();
rsaPrivateKey = (RSAPrivateCrtKey) keyPair.getPrivate();
byte[] privateKeyBuffer = new byte[165];
short lengthDP1 = rsaPrivateKey.getDP1(privateKeyBuffer, 0);
offset += lengthDP1;
short lengthDQ1 = rsaPrivateKey.getDQ1(privateKeyBuffer, offset);
offset += lengthDQ1;
short lengthP = rsaPrivateKey.getP(privateKeyBuffer, offset);
offset += lengthP;
short lengthQ = rsaPrivateKey.getQ(privateKeyBuffer, offset);
offset += lengthQ;
short lengthPQ = rsaPrivateKey.getPQ(privateKeyBuffer, offset);
After generating a buffer I should convert each array element to hex and then send with APDU and last in applet restore my private key, but in privateKeyBuffer we have negative numbers :
37,65,-96,-110,38,6,-2,73,-37,28,120,-90... (etc)
How I should convert them to hex and keep APDU correct (because as I know only positive numbers allowed) or may be there another way to push key to JavaCard?
Any ideas?

You can simply copy your byte array to the APDU buffer and send it. It will work.
final byte[] apduBuffer = apdu.getBuffer();
Util.arrayCopyNonAtomic(privateKeyBuffer, (short) 0, apduBuffer, (short) 0, (short) privateKeyBuffer.length);
apdu.setOutgoingAndSend((short) 0, (short) privateKeyBuffer.length);
You do not have to think about hex values and positive/negative numbers at all! When you send a byte = 8 bits, you do not deal with its number representation or meaning at all.
An extra note:
byte[] privateKeyBuffer = new byte[165]; is generally a bad idea. You do not need this buffer at all. You can copy key values directly into the APDU buffer, save some persistent memory and make your applet faster.
Another note based on dear #Abraham's comment:
The only problem with positive/negative numbers you could face in Java Card is the fact that Java Card byte is signed, unfortunately. That means its values are always in range [-128,127], although people often forget about it:
final byte b = getSomeByteValue(); //from APDU buffer, for example
if (b == 0x80) { //always false!
//this never happens!
}
If you want to understand your byte as unsigned (range [0, 255]), you have to cast it to short this way:
final byte b = getSomeByteValue(); //from APDU buffer, for example
final short unsignedValue = (short) (b & 0xFF);
if (unsignedValue == 0x80) { //correct!
//can happen
}

Converting a BigInteger into a Key

how do i cast a Big Integer into a Key for java cryptography library?
I am trying to use a shared diffie hellman key that i generated myself for the key value for AES encryption.
Below is the code that i used
BigInteger bi; long value = 1000000000;
bi = BigInteger.valueOf(value);
Key key = new Key (bi);
however it did not work.
May i know how do i convert a BigInteger value into a Key value?
Thanks in advance!

First, you cannot cast it. There is no relationship between the BigInteger class and the Key interface.
Second, Key is an interface not a class, so you can't create instances of it. What you need to create is an instance of some class that implements Key. And it most likely needs to be a specific implementation class, not (say) an anonymous class.
The final thing is that the Java crypto APIs are designed to hide the representation of the key. To create a key from bytes, you need to create a KeySpec object; e.g. SecretKeySpec(byte[] key, String algorithm)) and then use a KeyFactory to "generate" a key from it. Typical KeySpec constructors take a byte[] as a parameter, so you first need to get the byte array from your BigInteger instance.

You need to convert your BigInteger to a byte array of a specific size, then use the first (leftmost) bytes to create a key. For this you need to know the size of the prime p used in DH, as the value needs to be left-padded to represent a key. I would suggest to use standardized DH parameters (or at least make sure that the size of the prime is dividable by 8).
Note that there may be a zero valued byte in front of the byte array retrieved using BigInteger.toByteArray() because the value returned is encoded as a signed (two-complement) big-endian byte array. You need to remove this byte if the result is bigger than the prime (in bytes) because of it.
public static byte[] encodeSharedSecret(final BigInteger sharedSecret, final int primeSizeBits) {
// TODO assignment add additional tests on input
final int sharedSecretSize = (primeSizeBits + Byte.SIZE - 1) / Byte.SIZE;
final byte[] signedSharedSecretEncoding = sharedSecret.toByteArray();
final int signedSharedSecretEncodingLength = signedSharedSecretEncoding.length;
if (signedSharedSecretEncodingLength == sharedSecretSize) {
return signedSharedSecretEncoding;
}
if (signedSharedSecretEncodingLength == sharedSecretSize + 1) {
final byte[] sharedSecretEncoding = new byte[sharedSecretSize];
System.arraycopy(signedSharedSecretEncoding, 1, sharedSecretEncoding, 0, sharedSecretSize);
return sharedSecretEncoding;
}
if (signedSharedSecretEncodingLength < sharedSecretSize) {
final byte[] sharedSecretEncoding = new byte[sharedSecretSize];
System.arraycopy(signedSharedSecretEncoding, 0,
sharedSecretEncoding, sharedSecretSize - signedSharedSecretEncodingLength, signedSharedSecretEncodingLength);
return sharedSecretEncoding;
}
throw new IllegalArgumentException("Shared secret is too big");
}
After that you need to derive the key bytes using some kind of key derivation scheme. The one you should use depends on the standard you are implementing:
As stated in RFC 2631
X9.42 provides an algorithm for generating an essentially arbitrary
amount of keying material from ZZ. Our algorithm is derived from that
algorithm by mandating some optional fields and omitting others.
KM = H ( ZZ || OtherInfo)
H is the message digest function SHA-1 [FIPS-180] ZZ is the shared
secret value computed in Section 2.1.1. Leading zeros MUST be
preserved, so that ZZ occupies as many octets as p.
Note that I have discovered a bug in the Bouncy Castle libraries up to 1.49 (that's the current version at this date) in the DH implementation regarding the secret extraction - it does strip the spurious leading 00h valued bytes, but it forgets to left-pad the result up to the prime size p. This will lead to an incorrect derived key once in 192 times (!)

Decrypting “long” message encrypted with RSA java

Hi this is the same question, that was asked two years ago:
Java/JCE: Decrypting “long” message encrypted with RSA
I had a large byte array and rsa keypair, initiated by value 1024.
Using rsa encryption and the specified size of the key is strong requirement, I can't change it. So I can't use symmetric encryption with asymetric encryption symmetric key. I can't use any other keys. I had a byte array and need ciphered byte array to be returned. I wonder if there is any ready tool, that can manage with this problem?
Sorry for such an amateurish question, but I really need a help.

As stated, your question has a single answer, and that's "no". RSA encryption is an algorithm which encrypts messages up to a given size, which depends on the key size; with a 1024-bit RSA key, and RSA as the standard describes it, the maximum size is 117 bytes, no more. There is no way to encrypt a larger message with RSA alone, and that's a definite, mathematical certainty.
If you really need to process longer messages, then you necessarily have to add something else. In that case, please, please, do not try to do anything fancy of your own devising with some oh-so-clever splitting of data into small blocks and the like. That path leads to doom. You might produce something which appears to compile and run, but which will be invariably weak in some way, like almost every other home-made variation on cryptography. That's because security cannot be tested: it is not a case of "works" or "does not work".
The well-trodden path of asymmetric encryption goes thus:
You select a random sequence of bytes of some appropriate length, e.g. 128 bits (that's 16 bytes). Let's call it K.
You encrypt K with the RSA public key; this yields E.
You encrypt the message with K using a symmetric encryption algorithm ("AES/CBC/PKCS5Padding"). Since this is a one-shot key, you can use an all-zeros IV. This yields a bunch of bytes, let's call it F.
The encrypted message is then the concatenation of E and F.
Decryption proceeds in the reverse order: the RSA private key is used to recover K from E, then K is used to decrypt F into the original message. The key K is never stored anywhere, and a new key K is generated every time (even if you encrypt the same message twice). That's important, do not change that unless you understand what you are doing (and if you do, then you already know that).
Given what you state about your problem, you have to do something else than "just RSA". The procedure I describe above is about the best "something else" that you could come up with, security-wise.
Assembling some cryptographic elements into such a protocol is a process fraught with pitfalls so you may have better luck using an already defined format and support library. Two common formats for asymmetric encryption are CMS and OpenPGP. A library which supports both and has good reputation is Bouncy Castle.

If you do need to encrypt/decrypt long strings using RSA, then you can break the bytes up in to smaller "chunks" and process each chunk of bytes through the cipher one at a time while storing the results in a ByteBuffer.
Encryption:
byte[] encData = null;
try {
// create public key
X509EncodedKeySpec publicKeySpec = new X509EncodedKeySpec(key);
KeyFactory kf = KeyFactory.getInstance("RSA");
PublicKey pk = kf.generatePublic(publicKeySpec);
Cipher pkCipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
pkCipher.init(Cipher.ENCRYPT_MODE, pk);
int chunkSize = 117; // 1024 / 8 - 11(padding) = 117
int encSize = (int) (Math.ceil(data.length/117.0)*128);
int idx = 0;
ByteBuffer buf = ByteBuffer.allocate(encSize);
while (idx < data.length) {
int len = Math.min(data.length-idx, chunkSize);
byte[] encChunk = pkCipher.doFinal(data, idx, len);
buf.put(encChunk);
idx += len;
}
// fully encrypted data
encData = buf.array();
} catch (Exception e) {
e.printStackTrace();
Decryption
Cipher rsaCipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
rsaCipher.init(Cipher.DECRYPT_MODE, rsaPk);
int chunkSize = 128;
int idx = 0;
ByteBuffer buf = ByteBuffer.allocate(data.length);
while(idx < data.length) {
int len = Math.min(data.length-idx, chunkSize);
byte[] chunk = rsaCipher.doFinal(data, idx, len);
buf.put(chunk);
idx += len;
}
// fully decrypted data
byte[] decryptedData = buf.array();