Develop Reference

How to convert byte array to Base 64 string in unsigned way in java?

I am struggling to get the same Base64 string in both C# and Java
I want Java to treat bytes as unsigned ones when converting to Base64.
Here's my C# code
private static void Main(string[] args)
{
long baseTimeStamp = 1501492600;
byte[] bytes = BitConverter.GetBytes(baseTimeStamp * 114);
for (int i = 0; i < bytes.Length; i++)
{
bytes[i] = (byte)(bytes[i] >> 2);
}
string base64 = Convert.ToBase64String(bytes);
Console.WriteLine(base64);
}
In Java, I want to get the same Base64 for the same long value
Here's the code
public static void main(String[] args) {
long myLong = 1501492600;
byte[] bytes = longToBytes(myLong);
for(int i = 0; i < bytes.length / 2; i++)
{
int temp = bytes[i];
bytes[i] = bytes[bytes.length - i - 1];
bytes[bytes.length - i - 1] = (byte) temp;
bytes[i] = (byte)((bytes[i] >> 2));
}
System.out.println(DatatypeConverter.printBase64Binary(bytes));
}
private static byte[] longToBytes(long x) {
ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
buffer.putLong(x);
return buffer.array();
}
I tried both the commented way, and the DatatypeConverter way, but I get different String values. Is there a standard JDK way, or should I write my own base64 method to treat bytes as unsigned ones?

Base64 converts bits. It doesn't know or care about whether you think of the bytes as signed or unsigned. If you're getting different values, the problem is somewhere else.
However, since you're doing bit shifting, you need to use the zero-fill version >>> instead of the sign extend >>. That may be the source of your problem.
DatatypeConverter is still the easiest way to go.

How to convert minus hex value to String and byte array

I want to convert decimal -10 value to hex in a String and byte array format.
I have tried
String minHex = Integer.toHexString(Integer.valueOf(-10));
System.out.println(minHex);
Which results in fffffff6, and I think it is not correct. And for converting byte array I am using below function which I found from
Convert a string representation of a hex dump to a byte array using Java?
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
}
So also not sure it will work for minus hex value or not.

To convert an hex to a String the way you expect it i.e. -10 is converted to -a, use:
String hex = Integer.toString(-10, 16);
To convert to a byte array, simply convert the int to a byte array, it is represented the same way:
byte[] bytes = ByteBuffer.allocate(4).putInt(-10).array();

How do I convert a large string into hex and then into byte?

I work with cellphones and deal with MEID numbers on a daily basis. So instead of searching online for a MEID (a hex number of length 14) to pseudo ESN (a hex number of length 8) calculator, I figured I can make my own program. The way to obtain a pESN from MEID is fairly simple in theory. For example, given MEID 0xA0000000002329, to make a pESN, SHA-1 needs to be applied to the MEID. SHA-1 on A0000000002329 gives e3be267a2cd5c861f3c7ea4224df829a3551f1ab. Take the last 6 hex numbers of this result, and append it to 0x80 - the result is 0x8051F1AB.
Now here is the code I have so far:
public void sha1() throws NoSuchAlgorithmException {
String hexMEID = "A0000000002329";
MessageDigest mDigest = MessageDigest.getInstance("SHA1");
byte[] result = mDigest.digest(hexMEID.getBytes());
StringBuilder sb = new StringBuilder();
for (int i = 0; i < result.length; i++) {
sb.append(Integer.toString((result[i] & 0xff) + 0x100, 16).substring(1));
}
System.out.println(sb.toString());
}
The problem is that using this method, SHA-1 on A0000000002329 gives a89b611b421f57705bd013297ce3fc835f706ab0 instead of e3be267a2cd5c861f3c7ea4224df829a3551f1ab. What am I doing wrong here??
Someone gave me a hint that "the trick is to apply SHA-1 to the number representing the MEID, not the string representing the MEID. You'll need to process it byte-by-byte, so you must give it two hex numbers at a time (since two hex numbers make a byte) and make sure they are interpreted as numbers and not ASCII characters". If this is true then how do I change my string into hex and then into byte so that SHA1 can give me the correct result???

Without libraries, you can follow the example here:
In Java, how do you convert a hex string to a byte[]?
byte[] b = new BigInteger(s,16).toByteArray();
One library (I'm sure there are many) that also provides this is POJava:
<dependency>
<groupId>org.pojava</groupId>
<artifactId>pojava</artifactId>
<version>2.8.1</version>
</dependency>
byte[] hexMEIDBytes=EncodingTool.hexDecode(hexMEID);
[EDIT] ==============
Here's a more complete example per your followup question:
byte[] hexMEIDBytes = EncodingTool.hexDecode(hexMEID);
byte[] hash = HashingTool.hash(hexMEIDBytes, HashingAlgorithm.SHA);
String pESN="0x80" + EncodingTool.hexEncode(hash).substring(34).toUpperCase();
// a hexMEID value of "A0000000002329" results in a pESN value of "0x8051F1AB"

For String to Hex:
public String StrToHex(String arg) {
return String.format("%040x", new BigInteger(arg.getBytes(//Your Charset//)));
}
For Hex to byte:
This below code wont work for "0".
public byte[] hexStrToByteArray(String s) {
int leng = s.length();
byte[] data = new byte[leng / 2];
for (int i = 0; i < leng; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i+1), 16));
}
return data;
}

You can use the following two methods
public static synchronized String bytesToHex(byte [] buf){
StringBuffer strbuf = new StringBuffer(buf.length * 2);
int i;
for (i = 0; i < buf.length; i++) {
if (((int) buf[i] & 0xff) < 0x10){
strbuf.append("0");
}
strbuf.append(Long.toString((int) buf[i] & 0xff, 16));
}
return strbuf.toString();
}
public synchronized static byte[] hexToBytes(String hexString) {
byte[] b = new BigInteger(hexString,16).toByteArray();
return b;
}

How to convert java BigDecimal to normal byte array (not 2's complement)

How do I convert from big integer to a byte array which is not in 2's complement format. Bascially I only need to convert positive numbers and do not need the sign bit.
So something like 10 would become a byte 0x0a i.e-> 00001010
[Update]
As per comment I tried this
public void testBinary()
{
BigDecimal test = new BigDecimal(35116031);
BigInteger theInt = test.unscaledValue();
byte[] arr = theInt.toByteArray();
System.out.println(getCounterVal(arr, new BigInteger("256")));
}
public BigInteger getCounterVal(byte[] arr, BigInteger multiplier)
{
BigInteger counter = BigInteger.ZERO;
for(int i = (arr.length - 1); i >=0; i--)
{
int b = arr[i];
//int val = (int) b & 0xFF;
BigInteger augend = BigInteger.valueOf(b);
counter = counter.add(augend.multiply(multiplier.pow(i)));
}
return counter;
}
The out put value I got was -19720446 And with the //int val = (int) b & 0xFF; uncommented and used as augend, I got the value 4292024066
[Update2]
Here is a test I ran which works. Not sure if it is bug free but looks fine.
#Test
public void bigIntegerToArray()
{
BigInteger bigInt = new BigInteger("35116444");
byte[] array = bigInt.toByteArray();
if (array[0] == 0)
{
byte[] tmp = new byte[array.length - 1];
System.arraycopy(array, 1, tmp, 0, tmp.length);
array = tmp;
}
BigInteger derived = BigInteger.ZERO;
BigInteger twofiftysix = new BigInteger("256");
int j = 0;
for (int i = array.length - 1; i >= 0; i--)
{
int val = (int) array[i] & 0xFF;
BigInteger addend = BigInteger.valueOf(val);
BigInteger multiplier = twofiftysix.pow(j);
addend = addend.multiply(multiplier);
derived = derived.add(addend);
j++;
}
Assert.assertEquals(bigInt, derived);
}

The difference is largely conceptual. Unsigned numbers are the same in 2's compliment. 2's compliment just describes how to represent negative numbers which you say you don't have.
i.e. 10 is 00001010 in signed and unsigned representation.
To get the bytes from a BigDecimal or BigInteger you can use the methods it provides.
BigDecimal test = new BigDecimal(35116031);
BigInteger theInt = test.unscaledValue();
byte[] arr = theInt.toByteArray();
System.out.println(Arrays.toString(arr));
BigInteger bi2 = new BigInteger(arr);
BigDecimal bd2 = new BigDecimal(bi2, 0);
System.out.println(bd2);
prints
[2, 23, -45, -1]
35116031
The bytes are correct and reproduce the same value.
There is a bug in the way you rebuild your BigInteger. You assume the byte serialization is little endian when Java typically uses big endian http://en.wikipedia.org/wiki/Endianness

Try to split the number in bytes, by dividing by 256 in each iteration and using the remainder, and place all these bytes into an array.

the sign bit in 2-compliment for positive numbers is 0
so signed or unsigned doesn't make a difference for positive numbers

If the value is less than the size of a long then use longValue, then chop the long into bytes. If the value is bigger than a long then probably you need to use an iterative approach, repeatedly dividing the number by 256, taking the remainder as the next byte, then repeating until you get zero. The bytes would be generated right to left. Signed numbers require thought (to generate 2s-complement results) but aren't much more complicated.

Store binary sequence in byte array?

I need to store a couple binary sequences that are 16 bits in length into a byte array (of length 2). The one or two binary numbers don't change, so a function that does conversion might be overkill. Say for example the 16 bit binary sequence is 1111000011110001. How do I store that in a byte array of length two?

String val = "1111000011110001";
byte[] bval = new BigInteger(val, 2).toByteArray();
There are other options, but I found it best to use BigInteger class, that has conversion to byte array, for this kind of problems. I prefer if, because I can instantiate class from String, that can represent various bases like 8, 16, etc. and also output it as such.
Edit: Mondays ... :P
public static byte[] getRoger(String val) throws NumberFormatException,
NullPointerException {
byte[] result = new byte[2];
byte[] holder = new BigInteger(val, 2).toByteArray();
if (holder.length == 1) result[0] = holder[0];
else if (holder.length > 1) {
result[1] = holder[holder.length - 2];
result[0] = holder[holder.length - 1];
}
return result;
}
Example:
int bitarray = 12321;
String val = Integer.toString(bitarray, 2);
System.out.println(new StringBuilder().append(bitarray).append(':').append(val)
.append(':').append(Arrays.toString(getRoger(val))).append('\n'));

I have been disappointed with all of the solutions I have found to converting strings of bits to byte arrays and vice versa -- all have been buggy (even the BigInteger solution above), and very few are as efficient as they should be.
I realize the OP was only concerned with a bit string to an array of two bytes, which the BitInteger approach seems to work fine for. However, since this post is currently the first search result when searching "bit string to byte array java" in Google, I am going to post my general solution here for people dealing with huge strings and/or huge byte arrays.
Note that my solution below is the only solution I have ran that passes all of my test cases -- many online solutions to this relatively simple problem simply do not work.
Code
/**
* Zips (compresses) bit strings to byte arrays and unzips (decompresses)
* byte arrays to bit strings.
*
* #author ryan
*
*/
public class BitZip {
private static final byte[] BIT_MASKS = new byte[] {1, 2, 4, 8, 16, 32, 64, -128};
private static final int BITS_PER_BYTE = 8;
private static final int MAX_BIT_INDEX_IN_BYTE = BITS_PER_BYTE - 1;
/**
* Decompress the specified byte array to a string.
* <p>
* This function does not pad with zeros for any bit-string result
* with a length indivisible by 8.
*
* #param bytes The bytes to convert into a string of bits, with byte[0]
* consisting of the least significant bits in the byte array.
* #return The string of bits representing the byte array.
*/
public static final String unzip(final byte[] bytes) {
int byteCount = bytes.length;
int bitCount = byteCount * BITS_PER_BYTE;
char[] bits = new char[bitCount];
{
int bytesIndex = 0;
int iLeft = Math.max(bitCount - BITS_PER_BYTE, 0);
while (bytesIndex < byteCount) {
byte value = bytes[bytesIndex];
for (int b = MAX_BIT_INDEX_IN_BYTE; b >= 0; --b) {
bits[iLeft + b] = ((value % 2) == 0 ? '0' : '1');
value >>= 1;
}
iLeft = Math.max(iLeft - BITS_PER_BYTE, 0);
++bytesIndex;
}
}
return new String(bits).replaceFirst("^0+(?!$)", "");
}
/**
* Compresses the specified bit string to a byte array, ignoring trailing
* zeros past the most significant set bit.
*
* #param bits The string of bits (composed strictly of '0' and '1' characters)
* to convert into an array of bytes.
* #return The bits, as a byte array with byte[0] containing the least
* significant bits.
*/
public static final byte[] zip(final String bits) {
if ((bits == null) || bits.isEmpty()) {
// No observations -- return nothing.
return new byte[0];
}
char[] bitChars = bits.toCharArray();
int bitCount = bitChars.length;
int left;
for (left = 0; left < bitCount; ++left) {
// Ignore leading zeros.
if (bitChars[left] == '1') {
break;
}
}
if (bitCount == left) {
// Only '0's in the string.
return new byte[] {0};
}
int cBits = bitCount - left;
byte[] bytes = new byte[((cBits) / BITS_PER_BYTE) + (((cBits % BITS_PER_BYTE) > 0) ? 1 : 0)];
{
int iRight = bitCount - 1;
int iLeft = Math.max(bitCount - BITS_PER_BYTE, left);
int bytesIndex = 0;
byte _byte = 0;
while (bytesIndex < bytes.length) {
while (iLeft <= iRight) {
if (bitChars[iLeft] == '1') {
_byte |= BIT_MASKS[iRight - iLeft];
}
++iLeft;
}
bytes[bytesIndex++] = _byte;
iRight = Math.max(iRight - BITS_PER_BYTE, left);
iLeft = Math.max((1 + iRight) - BITS_PER_BYTE, left);
_byte = 0;
}
}
return bytes;
}
}
Performance
I was bored at work so I did some performance testing comparing against the accepted answer here for when N is large. (Pretending to ignore the fact that the BigInteger approach posted above doesn't even work properly as a general approach.)
This is running with a random bit string of size 5M and a random byte array of size 1M:
String -> byte[] -- BigInteger result: 39098ms
String -> byte[] -- BitZip result: 29ms
byte[] -> String -- Integer result: 138ms
byte[] -> String -- BitZip result: 71ms
And the code:
public static void main(String[] argv) {
int testByteLength = 1000000;
int testStringLength = 5000000;
// Independently random.
final byte[] randomBytes = new byte[testByteLength];
final String randomBitString;
{
StringBuilder sb = new StringBuilder();
Random rand = new Random();
for (int i = 0; i < testStringLength; ++i) {
int value = rand.nextInt(1 + i);
sb.append((value % 2) == 0 ? '0' : '1');
randomBytes[i % testByteLength] = (byte) value;
}
randomBitString = sb.toString();
}
byte[] resultCompress;
String resultDecompress;
{
Stopwatch s = new Stopwatch();
TimeUnit ms = TimeUnit.MILLISECONDS;
{
s.start();
{
resultCompress = compressFromBigIntegerToByteArray(randomBitString);
}
s.stop();
{
System.out.println("String -> byte[] -- BigInteger result: " + s.elapsed(ms) + "ms");
}
s.reset();
}
{
s.start();
{
resultCompress = zip(randomBitString);
}
s.stop();
{
System.out.println("String -> byte[] -- BitZip result: " + s.elapsed(ms) + "ms");
}
s.reset();
}
{
s.start();
{
resultDecompress = decompressFromIntegerParseInt(randomBytes);
}
s.stop();
{
System.out.println("byte[] -> String -- Integer result: " + s.elapsed(ms) + "ms");
}
s.reset();
}
{
s.start();
{
resultDecompress = unzip(randomBytes);
}
s.stop();
{
System.out.println("byte[] -> String -- BitZip result: " + s.elapsed(ms) + "ms");
}
s.reset();
}
}
}