I intend to parition md5 hash (2^128) into four quadrants. I'm using BigInteger java class to represent md5 hash. Below is the code I tried to partition it but seems to be not working:
private static final BigInterger chunkSize_ = (new BigInterger(2)).pow(125);
...
byte[] hashValue = new byte[hash.remaining()]; // hash is md5 bytebuffer
hash.get(hashValue);
BigInteger hashNumber = new BigInteger(hashValue);
String chunk = hashNumber.divide(chunkSize_).toString();
Collecting stats I see chunk value to as high as 34. Idea was to divide 2^128 into 4 halves. Please let me know what is wrong with above code snippet.
Thanks
Related
Since Ed25519 has not been around for long (in JDK), there are very few resources on how to use it.
While their example is very neat and useful, I have some trouble understanding what am I doing wrong regarding key parsing.
They Public Key is being read from a packet sent by an iDevice.
(Let's just say, it's an array of bytes)
From the searching and trying my best to understand how the keys are encoded, I stumbled upon this message.
4. The public key A is the encoding of the point [s]B. First,
encode the y-coordinate (in the range 0 <= y < p) as a little-
endian string of 32 octets. The most significant bit of the
final octet is always zero. To form the encoding of the point
[s]B, copy the least significant bit of the x coordinate to the
most significant bit of the final octet. The result is the
public key.
That means that if I want to get y and isXOdd I have to do some work.
(If I understood correctly)
Below is the code for it, yet the verifying still fails.
I think, I did it correctly by reversing the array to get it back into Big Endian for BigInteger to use.
My questions are:
Is this the correct way to parse the public key from byte arrays
If it is, what could possibly be the reason for it to fail the verifying process?
// devicePublicKey: ByteArray
val lastIndex = devicePublicKey.lastIndex
val lastByte = devicePublicKey[lastIndex]
val lastByteAsInt = lastByte.toInt()
val isXOdd = lastByteAsInt.and(255).shr(7) == 1
devicePublicKey[lastIndex] = (lastByteAsInt and 127).toByte()
val y = devicePublicKey.reversedArray().asBigInteger
val keyFactory = KeyFactory.getInstance("Ed25519")
val nameSpec = NamedParameterSpec.ED25519
val point = EdECPoint(isXOdd, y)
val keySpec = EdECPublicKeySpec(nameSpec, point)
val key = keyFactory.generatePublic(keySpec)
Signature.getInstance("Ed25519").apply {
initVerify(key)
update(deviceInfo)
println(verify(deviceSignature))
}
And the data (before manipulation) (all in HEX):
Device identifier: 34444432393531392d463432322d343237442d414436302d444644393737354244443533
Device public key: e0a611c84db0ae91abfe2e6db91b6a457a4b41f9d8e09afdc7207ce3e4942e94
Device signature: a0383afb3bcbd43d08b04274a9214036f16195dc890c07a81aa06e964668955b29c5026d73d8ddefb12160529eeb66f843be4a925b804b575e6a259871259907
Device info: a86a71d42874b36e81a0acc65df0f2a84551b263b80b61d2f70929cd737176a434444432393531392d463432322d343237442d414436302d444644393737354244443533e0a611c84db0ae91abfe2e6db91b6a457a4b41f9d8e09afdc7207ce3e4942e94
// Device info is simply concatenated [hkdf, identifier, public key]
And the public key after the manipulation:
e0a611c84db0ae91abfe2e6db91b6a457a4b41f9d8e09afdc7207ce3e4942e14
Thank you very much, and every bit of help is greatly appreciated.
This will help many more who will stumble upon this problem at a later point, when the Ed25519 implementation will not be so fresh.
Helped me a lot. Would never have figured it out without your example.
I did it in java.
public static PublicKey getPublicKey(byte[] pk)
throws NoSuchAlgorithmException, InvalidKeySpecException, InvalidParameterSpecException {
// key is already converted from hex string to a byte array.
KeyFactory kf = KeyFactory.getInstance("Ed25519");
// determine if x was odd.
boolean xisodd = false;
int lastbyteInt = pk[pk.length - 1];
if ((lastbyteInt & 255) >> 7 == 1) {
xisodd = true;
}
// make sure most significant bit will be 0 - after reversing.
pk[pk.length - 1] &= 127;
// apparently we must reverse the byte array...
pk = ReverseBytes(pk);
BigInteger y = new BigInteger(1, pk);
NamedParameterSpec paramSpec = new NamedParameterSpec("Ed25519");
EdECPoint ep = new EdECPoint(xisodd, y);
EdECPublicKeySpec pubSpec = new EdECPublicKeySpec(paramSpec, ep);
PublicKey pub = kf.generatePublic(pubSpec);
return pub;
Actually, the whole encoding and decoding is correct.
The one thing in the end, that was the problem was that I (by mistake) reversed the array I read one too many times.
Reversing arrays since certain keys are encoded in little endian, while in order to represent it as a BigInteger in JVM, you have to reverse the little endian so it becomes big endian.
Hopefully this helps everyone in the future who will get stuck on any similar problems.
If there will be any questions, simply comment here or send me a message here.
I'll do my best to help you out.
I the code here is way way way more than you probably need. I investigated this and came up with what I think is equivalent, only much simpler. Anyhow, here is the blog piece: https://www.tbray.org/ongoing/When/202x/2021/04/19/PKI-Detective. and here is the Java code: https://github.com/timbray/blueskidjava
You can check how it is done in the OpenJDK implementation:
https://github.com/openjdk/jdk15/blob/master/src/jdk.crypto.ec/share/classes/sun/security/ec/ed/EdDSAPublicKeyImpl.java#L65
Basically encodedPoint is your byte array (just the plain bytes, without ASN.1 encoding).
I have been trying to replicate the java password authenticate to python, however the resulted hash is different.
password: abcd1234
password token (java): $31$16$sWy1dDEx52vwQUCswXDYMQMzTJC39g1_nmrK384T4-w
generated password token (python): $pbkdf2$16$c1d5MWRERXg1MnZ3UVVDcw$qPQvE4QbrnYJTmRXk0M7wlfhH5U
From the Java code, the Iteration is 16, SALT should the first 16 char in sWy1dDEx52vwQUCswXDYMQMzTJC39g1_nmrK384T4-w, which is sWy1dDEx52vwQUCs and the hash should be wXDYMQMzTJC39g1_nmrK384T4-w
however, applying the variables to python gave me a different hash result which, qPQvE4QbrnYJTmRXk0M7wlfhH5U which is different from Java's hash.
Where did i missed out?
Java:
private static final String ALGORITHM = "PBKDF2WithHmacSHA1";
private static final int SIZE = 128;
private static final Pattern layout = Pattern.compile("\\$31\\$(\\d\\d?)\\$(.{43})");
public boolean authenticate(char[] password, String token)
{
Matcher m = layout.matcher(token);
if (!m.matches())
throw new IllegalArgumentException("Invalid token format");
int iterations = iterations(Integer.parseInt(m.group(1)));
byte[] hash = Base64.getUrlDecoder().decode(m.group(2));
byte[] salt = Arrays.copyOfRange(hash, 0, SIZE / 8);
byte[] check = pbkdf2(password, salt, iterations);
int zero = 0;
for (int idx = 0; idx < check.length; ++idx)
zero |= hash[salt.length + idx] ^ check[idx];
return zero == 0;
}
Python:
from passlib.hash import pbkdf2_sha1
def hasher(password):
size = 128
key0 = "abcd1234"
iter = int(password.split("$")[2])
salt0 = password.split("$")[3][0: 16]
hash = pbkdf2_sha1.using(rounds=iter, salt = salt0.encode()).hash(key0)
print(hash.split('$')[4])
return hash
Original Link for Java code: How can I hash a password in Java?
There's a bunch of things different between how that java code does things, and how passlib's pbkdf2_sha1 hasher does things.
The java hash string contains a log cost parameter, which needs passing through 1<<cost to get the number of rounds / iterations.
The salt+digest needs to be base64 decoded, then take the first 16 bytes as the salt (which actually corresponds to first 21 1/3 characters of base64 data).
Similarly, since the digest's bits start in the middle of a base64 character, when the salt+digest is decoded, and digest is then encoded separately, the base64 string would be
AzNMkLf2DX-easrfzhPj7A (noticably different from the original encoded string).
Based on that, the following bit of code converts a java hash into the format used by pbkdf1_sha1.verify:
from passlib.utils.binary import b64s_decode, ab64_encode
def adapt_java_hash(jhash):
_, ident, cost, data = jhash.split("$")
assert ident == "31"
data = b64s_decode(data.replace("_", ".").replace("-", "+"))
return "$pbkdf2$%d$%s$%s" % (1<<int(cost), ab64_encode(data[:16]),
ab64_encode(data[16:]))
>>> adapt_java_hash("$31$16$sWy1dDEx52vwQUCswXDYMQMzTJC39g1_nmrK384T4-w")
'$pbkdf2$65536$sWy1dDEx52vwQUCswXDYMQ$AzNMkLf2DX.easrfzhPj7A'
The resulting string should be suitable for passing into pbkdf2_sha1.verify("abcd1234", hash); except for one issue:
The java code truncates the sha1 digest to 16 bytes, rather than the full 20 bytes; and way passlib's hasher is coded, the digest must be the full 20 bytes.
If you alter the java code to use SIZE=160 instead of SIZE=128, running the hash through the above adapt() function should then work in passlib.
I am looking to implement my ConsistentHashing to which I can supply a good HashingFunction. A decent implementation using a SortedMap is explained here: https://weblogs.java.net/blog/tomwhite/archive/2007/11/consistent_hash.html
Now like suggested on the post I would like to use a Crypto function like MD5 which has good randomization. I understand MD5 gives back an inherent 128 bit output, however I need a randomized 32 bits. Would the following have high cardinality?
(1) Would the first 4 bytes of MD5 output be random enough? In which case I could just take first 32 bits of 128 bits of MD5 hash:
class MD5Hashing implements HashFunction{
#Override
public int getHash(String key) throws Exception{
MessageDigest digest = MessageDigest.getInstance("MD5");
byte[] byteArray = digest.digest(key.getBytes("UTF-8"));
ByteBuffer buffer = ByteBuffer.wrap(byteArray);
return buffer.getInt()& 0x7fffffff;
}
}
(2) What if I use just the String's internal Horner's algorithm which uses 31x+y on all characters in String?
class StringHashing implements HashFunction{
#Override
public int getHash(String key) throws Exception{
return key.hashCode()& 0x7fffffff;
}
}
(3) My internal Consistent Hashing like in link above is just a TreeMap Should I be using a BigInteger instead to still be able to get all 128 bits from MD5 or other Crypto algorithm?
private final SortedMap<Integer, T> circle = new TreeMap<Integer, T>();
EDIT:
Looks like both are bad, I even tried taking the last 4 bytes from MD5 hash. buffer.getInt(12).
Running for 5000 random strings following was the distribution.
{host4.a.b.com=1599, host3.a.b.com=1075, host2.a.b.com=238, host1.a.b.com=2088}
Found Murmur hash which has API to convert String input to a 32 bit hash output. Gave me a very nice distribution as well.
{host4.a.b.com=1665, host3.a.b.com=1373, host2.a.b.com=648, host1.a.b.com=1314}
http://d3s.mff.cuni.cz/~holub/sw/javamurmurhash/
public static int hash32( final String text) {...}
I'm making a system where I want to verify the server's identity via RSA, but I can't seem to get the server to properly decrypt the client's message.
The public and private keys are in slot 0 of the array, and mod is in slot 1, so they are setup correctly.
Client side code
int keyLength = 3072 / 8;//RSA key size
byte[] data = new byte[keyLength];
//Generate some random data. Note that
//Only the fist half of this will be used.
new SecureRandom().nextBytes(data);
int serverKeySize = in.readInt();
if (serverKeySize != keyLength) {//Definitely not the right heard
return false;
}
//Take the server's half of the random data and pass ours
in.readFully(data, keyLength / 2 , keyLength / 2);
//Encrypt the data
BigInteger[] keys = getKeys();
BigInteger original = new BigInteger(data);
BigInteger encrypted = original.modPow(keys[0], keys[1]);
data = encrypted.toByteArray();
out.write(data);
//If the server's hash doesn't match, the server has the wrong key!
in.readFully(data, 0, data.length);
BigInteger decrypted = new BigInteger(data);
return original.equals(decrypted);
Server side code
int keyLength = 3072 / 8;//Key length
byte[] data = new byte[keyLength];
//Send the second half of the key
out.write(data, keyLength / 2, keyLength / 2);
in.readFully(data);
BigInteger[] keys = getKeys();
BigInteger encrypted = new BigInteger(data);
BigInteger original = encrypted.modPow(keys[0], keys[1]);
data = original.toByteArray();
out.write(data);
AFAIK that implementation is correct however it doesn't seem to produce the correct output. Also no, I do not wish to use a Cipher for various reasons.
There are some critical details that are not being accounted for. The data you want to apply RSA to must be encoded as BigInteger x, with 0 <= x < n, where n is your modulus. You aren't doing that. In fact, because you are filling your entire data array with random data you cannot guarantee that. The PKCS#1 padding algorithm is designed to do this correctly, but since you are rolling your own you'll have to fix this in your code. Also, examine carefully how the BigInteger(byte[]) constructor and BigInteger.toByteArray() decode/encode integers. Naively many expect simply the base 256 encoding, and forget that BigInteger must accommodate negative integer also. It does so by using the ASN.1 DER integer rules. If the positive integer's high-order byte would be >= 128 then a leading zero byte is added.
Background: I have an iterative hash algorithm I need to compute from a Python script and a Java web application.
Psuedo code:
hash = sha256(raw)
for x=1 to 64000 hash = sha256(hash)
where hash is a byte array of length 32, and not a hex string of length 64.
The reason I want to keep it in bytes is because, though Python can convert to hex string in between each iteration and keep the processing time under a second, Java takes 3 seconds for the String overhead.
So, the Java code looks like this:
// hash one time...
byte[] result = sha256(raw.getBytes("UTF-8"));
// then hash 64k-1 more times
for (int x = 0; x < 64000-1; x++) {
result = sha256(result);
}
// hex encode and print result
StringBuilder sb = new StringBuilder();
Formatter formatter = new Formatter(sb);
for (int i=0; i<buf.length; i++) {
formatter.format("%02x", buf[i]);
}
System.out.println(sb.toString());
And the Python code looks like this:
import hashlib
# hash 1 time...
hasher = hashlib.sha256()
hasher.update(raw)
digest = hasher.digest()
# then hash 64k-1 times
for x in range (0, 64000-1):
# expect digest is bytes and not hex string
hasher.update(digest)
digest = hasher.digest()
print digest.encode("hex")
The Python result calculated the hash on the hex representation of the first digest (String), rather than the raw digest bytes. So, I get varying outputs.
Method .update of hasher appends argument to previous text (Python docs). Instead you should create new hasher each time you want to compute digest.
import hashlib
# hash 1 time...
digest = hashlib.sha256(raw).digest()
# then hash 64k-1 times
for x in range(0, 64000-1):
digest = hashlib.sha256(digest).digest()
print digest.encode("hex")