from Hbase rowkey design
I dont unstand the sample in "Byte Pattern":
// hash
//
MessageDigest md = MessageDigest.getInstance("MD5");
byte[] digest = md.digest(Bytes.toBytes(s));
System.out.println("md5 digest bytes length: " + digest.length); // returns 16
String sDigest = new String(digest);
byte[] sbDigest = Bytes.toBytes(sDigest);
System.out.println("md5 digest as string length: " + sbDigest.length); // returns 26
why sbDigent.length != digest.length ?
Thank you
in my eclipse,
"digest" is 0xE807F1FCF82D132F9BB018CA6738A19F, 16 bytes, so its length is 16.
"sbDigest" is 0xEFBFBD07EFBFBDEFBFBDEFBFBD2D132FEFBFBDEFBFBD18EFBFBD6738EFBFBDEFBFBD,34 bytes, so its length is 34 (not 26).
The root reason is java.lang.String's constructor:
public String(byte[] bytes)
Constructs a new String by decoding the specified array of bytes using
the platform's default charset. The length of the new String is a
function of the charset, and hence may not be equal to the length
of the byte array.
I would like to compare two byte arrays. One is calculated from plaintext with MessageDigest SHA1, the other is the hex itself in byte array, without calculation.
MessageDigest returns 20 byte long result, String.getBytes() returns 40 byte long array. bytesToHex() function is the same that was provided in this answer, used only for printing.
The question:
How can I convert a string to byte array (and then compare with one calculated with MessageDigest) without the additional overhead? String comparison with bytesToHex() and .toUppercase() is working, but not an option, since speed is cruital in the application.
The code:
MessageDigest md;
try {
md = MessageDigest.getInstance("SHA-1");
byte[] toEncode = "test".getBytes();
byte[] encoded = md.digest(toEncode);
System.out.println("String to encode:\t\t" + new String(toEncode));
System.out.println("Encoded in hex:\t\t\t" + bytesToHex(encoded));
System.out.println("Encoded length:\t\t\t" + encoded.length);
byte[] hash = new String("a94a8fe5ccb19ba61c4c0873d391e987982fbbd3").getBytes(); // "test" representation in SHA1
System.out.println("\nHash to compare with:\t\t" + new String(hash));
System.out.println("Hash length:\t\t\t" + hash.length);
System.out.println("Two byte array equals:\t\t" + Arrays.equals(hash, encoded));
System.out.println("Two equals in string:\t\t" + new String(hash).equals(bytesToHex(encoded).toLowerCase()));
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
The result:
String to encode: test
Encoded in hex: A94A8FE5CCB19BA61C4C0873D391E987982FBBD3
Encoded length: 20
Hash to compare with: a94a8fe5ccb19ba61c4c0873d391e987982fbbd3
Hash length: 40
Two byte array equals: false
Two equals in string: true
You're not decoding your hex representation to bytes. If you would, for example using the solution from this answer, the two arrays would match:
try {
byte[] encoded = MessageDigest.getInstance("SHA-1").digest("test".getBytes());
byte[] hash = DatatypeConverter.parseHexBinary("a94a8fe5ccb19ba61c4c0873d391e987982fbbd3");
System.out.println("Two byte array equals:\t\t" + Arrays.equals(hash, encoded));
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
I'm using this:
import com.sun.org.apache.xml.internal.security.utils.Base64; to encode/decode Base64 strings and byte arrays to store into a db.
I'm testing out encoding and decoding to see if I can get back the original string:
SecureRandom srand = new SecureRandom();
byte[] randomSalt = new byte[64];
srand.nextBytes(randomSalt);
System.out.println("rand salt bytes: " + randomSalt); // first line
String salt = Base64.encode(randomSalt);
try {
System.out.println(Base64.decode(salt)); // second line
}catch(Base64DecodingException e){
System.out.println(e);
}
However, this prints out:
rand salt bytes: [B#68286c59
[B#44d01f20
Why are these not the same, so that I can get back the original byte array?
What you are doing is actually dealing with the java pointer instead of the actual bytes.
This is the correct way to implement
byte[] bytesEncoded = Base64.encodeBase64(str .getBytes());
System.out.println("ecncoded value is " + new String(bytesEncoded ));
// Decode data on other side, by processing encoded data
byte[] valueDecoded= Base64.decodeBase64(bytesEncoded );
System.out.println("Decoded value is " + new String(valueDecoded));
With the following Code - I'm generally able to decrypt a token that I'm being passed. The token string when decoded to json will look like the following:
\"id\":\"9efef759-15a3-4cd0-b1f1-fceab7ad0a6e\",
\"exp\":\"2016-07-23T15:27:50.758+12:00\",
\"iv\":\"OOqNpy9puM5jPjTwrWHSNb+d5NYDEwIq2pZFqx6mraI14Kkh0bzEWADoU2d/KGu6cp9/FrVt4epheIP5Fw9qUFrdVcNYjLO5HWdJ0V5GhpdLJlFbMnFy4vS1rJ+4X1qTNZrqPwZh2deLceoHmxnqw7ml8JVFeIaz9H8BQXkgcNo=\",
\"ver\":\"1\",
\"iat\":\"2016-07-13T15:27:50.758+12:00\",
\"key\":\"d7R9blmqBYMywOEdYpRbd+gvKPfOqmxsRQMlDipkuGoWZobJ0dnK0MGBFAXq4wOdHbHVbfisjqm+6HoRSZ2w0KcfY+enPoKL5yptvlULkwpDtATEP8pnRmCh6ycWntbanL1gJI7RoNWTkomItBp/yODdL5kSMue76xAtIzc9+no=\",
\"sig\":\"X6A58tRDSUC5HJEP1VVmQjo17Qk2rJC9pYZiV5ccIjdcLmz7HPIkpm0ZCsFcQX4ps1k32asSojqOyegYFIdDqHypdrV9c5sHchIrp6Ak8MOjNTpy+SweTGPzkjlEHCMkWLVHjrkBq9mmoMk2o0sYyZes+/ARuYB8IjtAINtbAQE=\",
\"enc\":\"n+exbDhicBLuUtbYPXrrKESIktgyaidSreD5FWAxErGJeOyjTWv9QOqCGfEou5yJq2njCddf0mu0JOEP9i1mlhe1MUUa1hE4J+qnqxre+tSxWRNszHQL8Pk+0FV6cZ1nqk+aCfw9VOjlOLYXYmNF0NSZBqQIqzpobM3twHIf5u7pvJkvbnfP8Db0S83ZchNgMWyH1t+UEb+jbpcg1Um3U7Yb8Q==\"
The IV and key are pulled from the token and Asymmetrically decrypted then the enc text is symmetrically decrypted before they are passed to a gzip decompression.
internal virtual UserObj decrypt(string jsonToken, UserObj cls, System.Security.Cryptography.AsymmetricAlgorithm certPrivateKey)
{
Org.BouncyCastle.Crypto.AsymmetricCipherKeyPair bcPrivateKey;
try
{
//Make a bouncyCastle private key for feeding to the rsa Engine.
bcPrivateKey = Org.BouncyCastle.Security.DotNetUtilities.GetKeyPair(certPrivateKey);
// Attempt to unmarshal the JSON token
Token token = encoder.unmarshalJsonString<Token>(jsonToken);
// Asymmetrically encrypt the symmetric encryption iv
Org.BouncyCastle.Crypto.Engines.RsaEngine rsaCipher = new Org.BouncyCastle.Crypto.Engines.RsaEngine();
Org.BouncyCastle.Crypto.Encodings.Pkcs1Encoding rsaEncoder = new Org.BouncyCastle.Crypto.Encodings.Pkcs1Encoding(rsaCipher);
rsaEncoder.Init(false, bcPrivateKey.Private);
// Asymmetrically decrypt the symmetric encryption key
byte[] encryptedAesKeyBytes = encoder.fromBase64String(token.Get(Token.ENCRYPTED_KEY));
byte[] aesKeyBytes = rsaEncoder.ProcessBlock(encryptedAesKeyBytes, 0, encryptedAesKeyBytes.Length);
// Asymmetrically decrypt the symmetric encryption IV
byte[] encryptedAesIvBytes = encoder.fromBase64String(token.Get(Token.IV));
byte[] aesIvBytes = rsaEncoder.ProcessBlock(encryptedAesIvBytes, 0, encryptedAesIvBytes.Length);
//Setting equivalent excyption to "AES/CTR/NoPadding"
Org.BouncyCastle.Crypto.Engines.AesEngine aes = new Org.BouncyCastle.Crypto.Engines.AesEngine();
Org.BouncyCastle.Crypto.Modes.SicBlockCipher blockCipher = new Org.BouncyCastle.Crypto.Modes.SicBlockCipher(aes);
Org.BouncyCastle.Crypto.Paddings.PaddedBufferedBlockCipher aesCipher = new Org.BouncyCastle.Crypto.Paddings.PaddedBufferedBlockCipher(blockCipher, new Org.BouncyCastle.Crypto.Paddings.ZeroBytePadding());
Org.BouncyCastle.Crypto.Parameters.KeyParameter keyParam2 = new Org.BouncyCastle.Crypto.Parameters.KeyParameter(aesKeyBytes);
// Symmetrically decrypt the data
Org.BouncyCastle.Crypto.Parameters.ParametersWithIV keyParamWithIv = new Org.BouncyCastle.Crypto.Parameters.ParametersWithIV(keyParam2, aesIvBytes, 0, TokenEncryptor.IV_SIZE_BYTES);
//
// Symmetrically decrypt the data
aesCipher.Init(false, keyParamWithIv);
string encryptedData = token.Get(Token.ENCRYPTED_DATA);
byte[] inputBytes = encoder.fromBase64String(encryptedData);
byte[] compressedJsonBytes = new byte[aesCipher.GetOutputSize(inputBytes.Length)];
//Do the decryption. length is the proper size of the compressed data, compressedJsonBytes will
//contain extra nulls at the end.
int length = aesCipher.ProcessBytes(inputBytes, compressedJsonBytes, 0);
//String to look at the compressed data (debug)
string compressed = encoder.toBase64String(compressedJsonBytes);
byte[] compressedJsonBytesProperSize = new byte[length];
Array.Copy(compressedJsonBytes, compressedJsonBytesProperSize, length);
//String to look at the compressed data (debug)
compressed = encoder.toBase64String(compressedJsonBytesProperSize);
byte[] jsonBytes = null;
try
{
jsonBytes = encoder.decompress(compressedJsonBytesProperSize);
}
catch (Exception)
{
jsonBytes = encoder.decompress(compressedJsonBytes);
}
string tmep = System.Text.Encoding.UTF8.GetString(jsonBytes, 0, jsonBytes.Length);
UserObj dataObj = encoder.fromJsonBytes<UserObj>(jsonBytes);
return dataObj;
}
catch (Exception e)
{
throw new Exceptions.TokenDecryptionException(e);
}
}
The final decrypted token looking something like this:
\"domain\":\"GLOBAL\",
\"user\":\"someuser\",
\"groups\":[\"GROUP1\",\"GROUP2\",\"GROUP3\"],
\"branchId\":\"0000\"
My problem occurs when depending on the number of items in the group, the GZ Decompression will fail. On some tokens if I pass the full compressedJsonByte array (with nulls at the end), it complains about a CRC error (why I have the try/catch around the decompression), so then I pass it the trimmed byte array. But for other tokens with more groups, it decompresses with the full byte array.
I have a comparable encryption routine and found that with everything else being equal if a alerted the user name from 17 to 19 characters I'd need to use the untrimmed byte array for decompression. But have since found the problem runs deeper.
Any help would be appreciated. I'm hoping its a decompression issue, but I suspect maybe something in the decrypting is fudging the end of the output byte array.
I'm not able to change the decryption type as it's coming from an external entity and their side is written in Java.
For reference the decompression routine is:
public virtual byte[] decompress(byte[] compressedData)
{
try
{
//Push to a file for debug
System.IO.FileStream fs = new System.IO.FileStream(#"C:\temp\file.gz", System.IO.FileMode.OpenOrCreate);
fs.Write(compressedData,0,compressedData.Length);
fs.Flush();
fs.Close();
byte[] outputBytes = new byte[4096];
byte[] buffer = new byte[4096];
using (System.IO.MemoryStream msInput = new System.IO.MemoryStream(compressedData))
{
System.IO.MemoryStream msOutput = new System.IO.MemoryStream();
//using (System.IO.Compression.GZipStream gzs = new System.IO.Compression.GZipStream(msInput, System.IO.Compression.CompressionMode.Decompress))
using (ZLibNet.GZipStream gzs = new ZLibNet.GZipStream(msInput, ZLibNet.CompressionMode.Decompress))
{
int nRead;
bool canR = gzs.CanRead;
while ((nRead = gzs.Read(buffer, 0, buffer.Length)) > 0)
{
msOutput.Write(buffer, 0, nRead);
}
}
outputBytes = msOutput.ToArray();
if (outputBytes.Length == 0)
throw new Exception("Could not decompress");
}
return outputBytes;
}
catch (Exception e)
{
throw new Exceptions.ServiceException(e);
}
}
You're using ZeroBytePadding while CTR mode doesn't require any padding at all, you might as well directly use the SicCipher instance.
Zero byte padding will strip all the zero valued bytes from the end of the data, which is why you may end up with compromised data.
Zero byte padding is not deterministic and should not be used unless you're either:
sure of the data not ending with a zero byte or bit;
able to determine the plaintext length in other ways.
I may have just cracked it. For the larger payload token, ProcessBytes would work in one pass.
For the smaller ones it took me a long while to notice that the length reported from ProcessBytes was less that the length of the inputBytes array.
I tried a DoFinal after the process bytes:
int length = aesCipher.ProcessBytes(inputBytes, compressedJsonBytes, 0);
int length2= aesCipher.DoFinal(compressedJsonBytes, length);
And for a token I encoded it worked...but for the given ones I have, it posted a Org.BouncyCastle.Crypto.DataLengthException: last block incomplete in decryption
So finally I tried
int length2 = aesCipher.ProcessBytes(inputBytes,length,inputBytes.Length-length,compressedJsonBytes,length);
To process the remain data - and this worked.
So my resulting code now looks as follows - replacing the single line:
int length = aesCipher.ProcessBytes(inputBytes, compressedJsonBytes, 0);
With:
int length = 0;
while (length < inputBytes.Length)
{
length += aesCipher.ProcessBytes(inputBytes, length, inputBytes.Length-length, compressedJsonBytes, length);
}
And this seems to have removed the need to take the full length compressed byte array. The UncompressedProperSize byte array now down it's job properly.
I need to convert a salt value randomly generated and store it in the database. To store it in the database I converted it to a string. Then for retrieving the original value, I convert it back to byte. But both value are not matching. I have tried "UTF-8","UTF-16", BASE64Encoder.
SecureRandom ranGen = new SecureRandom();
byte[] aesKey = new byte[16]; // 16 bytes = 128 bits
ranGen.nextBytes(aesKey);
System.out.println(aesKey);
String a=new String(aesKey,"UTF-16");
byte[] b=new byte[16];
b=a.getBytes("UTF-16");
System.out.println(b);
Outputs for the above code(Executed it 2 times):
[B#11563ff
[B#1581593
and
[B#170888e
[B#11563ff
You really ought to use Base64 for converting binary data to Strings. There are lots of free implementations available, for example the one found in Apache Commons Codec.
Also, it's really easy to use, for example:
For encoding:
import org.apache.commons.codec.binary.Base64;
...
byte[] abValue = {...}; // Your data to encode
Base64 base64 = new Base64();
String strEncodedData = base64.encodeToString(abValue).trim();
For decoding:
import org.apache.commons.codec.binary.Base64;
...
String strEncodedData = "..."; // Your previously encoded data
Base64 base64 = new Base64();
byte[] abValue = base64.decode(strValue);
As your code is written above, printing aesKey and then b, what you are actually printing is the output of the toString method for an array object, which is just the default Object toString method. So I don't see how you can expect them to be the same.
If you really want to check they are the same you should compare them byte by byte.
In terms of your actual question regarding storing a byte[] as a String in the DB, your best bet is to Base64 encode it. I would suggest using the Apache Commons Codec library for this. See the user guide.
EDIT:
Using the BASE64Encode and BASE64Decoder you have referred to, the code would be like this:
SecureRandom ranGen = new SecureRandom();
byte[] aesKey = new byte[16]; // 16 bytes = 128 bits
ranGen.nextBytes(aesKey);
String a = new BASE64Encoder().encode(aesKey);
System.out.println(a);
byte[] b = new BASE64Decoder().decodeBuffer(a);
System.out.println(new BASE64Encoder().encode(b));
for (int i = 0; i < aesKey.length; i++) {
System.out.println(aesKey[i] + " " + b[i]);
}
Here, I have also looped through the bytes individually, to show that they are indeed equal.