Develop Reference

What is the purpose of low and high nibble when converting a string to a HexString

Recently I have been going through some examples of MD5 to start getting an understanding of security and MD5 has been fairly simple to understand for the most part and a good starting point even though it is no longer secure. Despite this I have a question regarding high and lo nibbles when it comes to converting a string to a hex string.
So I know high and low nibbles are equal to half a byte or also can be hex digits that represent a single hexadecimal digit. What I am not understanding though is exactly how they work and the purpose that they serve. I have been searching on google and I can't find much of an answer that will help explain what they do in the context that they are in. Here is the context of the conversion:
private static String toHexString( byte[] byteArray )
{
final String HEX_CHARS = "0123456789ABCDEF";
byte[] result = new byte[byteArray.length << 1];
int len = byteArray.length;
for( int i = 0 ; i < len ; i++ )
{
byte b = byteArray[i]
int lo4 = b & 0x0F;
int hi4 = ( b & 0xF0 ) >> 4;
result[i * 2] = (byte)HEX_CHARS.charAt( hi4 );
result[i * 2 + 1] = (byte)HEX_CHARS.charAt( lo4 );
}
return new String( result );
}
I don't exactly understand what is going on in the for statement. I would appreciate any help understanding this and if there is some link to some places that I can learn more about this please also leave it.
I understand the base definition of nibble but not the operations and what the assignment to the number 4 is doing either.
If I need to post the full example code I will just ask as I am unsure if it is needed.

This code simply converts a byte array to hexadecimal representation. Within the for-loop, each byte is converted into two characters. I think it's easier to understand it on an example.
Assume one of the bytes in your array is, say, 218 (unsigned). That's 1101 1010 in binary.
lo4 gets the lowest 4 bits by AND-ing the byte with the bitmask 00001111:
int lo4 = b & 0x0F;
This results in 1010, 10 in decimal.
hi4 gets the highest 4 bits by AND-ing with the bitmask 1111 0000 and shifting 4 bits to the right:
int hi4 = ( b & 0xF0 ) >> 4;
This results in 1101, 13 in decimal.
Now to get the hexadecimal representation of this byte you only need to convert 10 and 13 to their hexadecimal representations and concatenate. For this you simply look up the character in the prepared HEX_CHARS string at the specific index. 10 -> A, 13 -> D, resulting in 218 -> DA.

It's just bit operations. The & character takes the literal bit value of each and does a logical and on them.
int lo4 = b & 0x0F;
for instance if b = 24 then it will evaluate to this
00011000
+00001111
=00001000
The second such line does the same on the first four bits.
00011000
+11110000
=00010000
the '>>' shifts all of the bits a certain number in that direction so
00010000 >> 4 = 00000001.
This is done so that you can derive the hex value from the number. Since each character in hex can represent 4 bits by splitting the number into pieces of 4 bits we can convert it.
in the case of b = 24 we no have lo4 = 1000 or 8 and hi4 = 0001 or 1. The last part of the loop assigns the character value for each.
Hex_chars[hi4] = '1' and Hex_chars[lo4] = '8' which gives you "18" for that part of the string which is 24 in hex.

How to convert binary string to the byte array of 2 bytes in java

I have binary string String A = "1000000110101110". I want to convert this string into byte array of length 2 in java
I have taken the help of this link
I have tried to convert it into byte by various ways
I have converted that string into decimal first and then apply the code to store into the byte array
int aInt = Integer.parseInt(A, 2);
byte[] xByte = new byte[2];
xByte[0] = (byte) ((aInt >> 8) & 0XFF);
xByte[1] = (byte) (aInt & 0XFF);
System.arraycopy(xByte, 0, record, 0,
xByte.length);
But the values get store into the byte array are negative
xByte[0] :-127
xByte[1] :-82
Which are wrong values.
2.I have also tried using
byte[] xByte = ByteBuffer.allocate(2).order(ByteOrder.BIG_ENDIAN).putInt(aInt).array();
But it throws the exception at the above line like
java.nio.Buffer.nextPutIndex(Buffer.java:519) at
java.nio.HeapByteBuffer.putInt(HeapByteBuffer.java:366) at
org.com.app.convert.generateTemplate(convert.java:266)
What should i do now to convert the binary string to byte array of 2 bytes?Is there any inbuilt function in java to get the byte array

The answer you are getting
xByte[0] :-127
xByte[1] :-82
is right.
This is called 2's compliment Represantation.
1st bit is used as signed bit.
0 for +ve
1 for -ve
if 1st bit is 0 than it calculates as regular.
but if 1st bit is 1 than it deduct the values of 7 bit from 128 and what ever the answer is presented in -ve form.
In your case
1st value is10000001
so 1(1st bit) for -ve and 128 - 1(last seven bits) = 127
so value is -127
For more detail read 2's complement representation.

Use putShort for putting a two byte value. int has four bytes.
// big endian is the default order
byte[] xByte = ByteBuffer.allocate(2).putShort((short)aInt).array();
By the way, your first attempt is perfect. You can’t change the negative sign of the bytes as the most significant bit of these bytes is set. That’s always interpreted as negative value.
10000001₂ == -127
10101110₂ == -82

try this
String s = "1000000110101110";
int i = Integer.parseInt(s, 2);
byte[] a = {(byte) ( i >> 8), (byte) i};
System.out.println(Arrays.toString(a));
System.out.print(Integer.toBinaryString(0xFF & a[0]) + " " + Integer.toBinaryString(0xFF & a[1]));
output
[-127, -82]
10000001 10101110
that is -127 == 0xb10000001 and -82 == 0xb10101110

Bytes are signed 8 bit integers. As such your result is completely correct.
That is: 01111111 is 127, but 10000000 is -128. If you want to get numbers in 0-255 range you need to use a bigger variable type like short.
You can print byte as unsigned like this:
public static String toString(byte b) {
return String.valueOf(((short)b) & 0xFF);
}

Convert a byte array to integer in Java and vice versa

I want to store some data into byte arrays in Java. Basically just numbers which can take up to 2 Bytes per number.
I'd like to know how I can convert an integer into a 2 byte long byte array and vice versa. I found a lot of solutions googling but most of them don't explain what happens in the code. There's a lot of shifting stuff I don't really understand so I would appreciate a basic explanation.

Use the classes found in the java.nio namespace, in particular, the ByteBuffer. It can do all the work for you.
byte[] arr = { 0x00, 0x01 };
ByteBuffer wrapped = ByteBuffer.wrap(arr); // big-endian by default
short num = wrapped.getShort(); // 1
ByteBuffer dbuf = ByteBuffer.allocate(2);
dbuf.putShort(num);
byte[] bytes = dbuf.array(); // { 0, 1 }

byte[] toByteArray(int value) {
return ByteBuffer.allocate(4).putInt(value).array();
}
byte[] toByteArray(int value) {
return new byte[] {
(byte)(value >> 24),
(byte)(value >> 16),
(byte)(value >> 8),
(byte)value };
}
int fromByteArray(byte[] bytes) {
return ByteBuffer.wrap(bytes).getInt();
}
// packing an array of 4 bytes to an int, big endian, minimal parentheses
// operator precedence: <<, &, |
// when operators of equal precedence (here bitwise OR) appear in the same expression, they are evaluated from left to right
int fromByteArray(byte[] bytes) {
return bytes[0] << 24 | (bytes[1] & 0xFF) << 16 | (bytes[2] & 0xFF) << 8 | (bytes[3] & 0xFF);
}
// packing an array of 4 bytes to an int, big endian, clean code
int fromByteArray(byte[] bytes) {
return ((bytes[0] & 0xFF) << 24) |
((bytes[1] & 0xFF) << 16) |
((bytes[2] & 0xFF) << 8 ) |
((bytes[3] & 0xFF) << 0 );
}
When packing signed bytes into an int, each byte needs to be masked off because it is sign-extended to 32 bits (rather than zero-extended) due to the arithmetic promotion rule (described in JLS, Conversions and Promotions).
There's an interesting puzzle related to this described in Java Puzzlers ("A Big Delight in Every Byte") by Joshua Bloch and Neal Gafter . When comparing a byte value to an int value, the byte is sign-extended to an int and then this value is compared to the other int
byte[] bytes = (…)
if (bytes[0] == 0xFF) {
// dead code, bytes[0] is in the range [-128,127] and thus never equal to 255
}
Note that all numeric types are signed in Java with exception to char being a 16-bit unsigned integer type.

You can also use BigInteger for variable length bytes. You can convert it to long, int or short, whichever suits your needs.
new BigInteger(bytes).intValue();
or to denote polarity:
new BigInteger(1, bytes).intValue();
To get bytes back just:
new BigInteger(bytes).toByteArray()
Although simple, I just wanted to point out that if you run this many times in a loop, this could lead to a lot of garbage collection. This may be a concern depending on your use case.

A basic implementation would be something like this:
public class Test {
public static void main(String[] args) {
int[] input = new int[] { 0x1234, 0x5678, 0x9abc };
byte[] output = new byte[input.length * 2];
for (int i = 0, j = 0; i < input.length; i++, j+=2) {
output[j] = (byte)(input[i] & 0xff);
output[j+1] = (byte)((input[i] >> 8) & 0xff);
}
for (int i = 0; i < output.length; i++)
System.out.format("%02x\n",output[i]);
}
}
In order to understand things you can read this WP article: http://en.wikipedia.org/wiki/Endianness
The above source code will output 34 12 78 56 bc 9a. The first 2 bytes (34 12) represent the first integer, etc. The above source code encodes integers in little endian format.

/** length should be less than 4 (for int) **/
public long byteToInt(byte[] bytes, int length) {
int val = 0;
if(length>4) throw new RuntimeException("Too big to fit in int");
for (int i = 0; i < length; i++) {
val=val<<8;
val=val|(bytes[i] & 0xFF);
}
return val;
}

As often, guava has what you need.
To go from byte array to int: Ints.fromBytesArray, doc here
To go from int to byte array: Ints.toByteArray, doc here

Someone with a requirement where they have to read from bits, lets say you have to read from only 3 bits but you need signed integer then use following:
data is of type: java.util.BitSet
new BigInteger(data.toByteArray).intValue() << 32 - 3 >> 32 - 3
The magic number 3 can be replaced with the number of bits (not bytes) you are using.

i think this is a best mode to cast to int
public int ByteToint(Byte B){
String comb;
int out=0;
comb=B+"";
salida= Integer.parseInt(comb);
out=out+128;
return out;
}
first comvert byte to String
comb=B+"";
next step is comvert to a int
out= Integer.parseInt(comb);
but byte is in rage of -128 to 127 for this reasone, i think is better use rage 0 to 255 and you only need to do this:
out=out+256;

How to send such complicated hex, binary protocol data accurately using java byte[]?

I am very confused with this kind of stuffs, What should I send as final command, get always confused 8 bits to 1 byte but how do we make it? Is it only the command packet [hex] as shown in screen shot? or Is it header + command packet[hex] shown in two screen shots?
Confusion details:
In such diagram header block shows mostly like "Bit 7, Bit 6,..,Bit 0" instead of "Bit 0, Bit 1, Bit 2, ... Bit 7", i always wondering why?.
But when I apply in code what is following order for byte st[0] = Bit 7 or Bit 1?
Also according to this diagram, does it mean every command I send will have Header fixed always?
This is the code I was trying by taking Bit 1 as st[0], Bit 2 as st1 instead of Bit 7 as st[0]. To apply Power off/on test.
import java.io.*;
import java.net.*;
public class test
{
public static void main(String[] args) throws UnknownHostException, IOException
{
byte st[]=new byte[256];
st[0]=0x01; // get
st[1]=0x02; // header 2 byte
st[2]=0x02; // header length
st[3]=0; // command byte
st[4]=0; // reserved
st[5]=0;
st[6]=0;
st[7]=0;
st[8]=0x01; // power off
st[9]=0x30;
st[10]=0x01;
st[11]=0x00;
System.out.println(st); // Should this work, am i correct based on diagram?
Socket s = new Socket("192.168.1.2", 49137);
DataInputStream input = new DataInputStream(s.getInputStream());
DataOutputStream outToServer = new DataOutputStream(s.getOutputStream());
BufferedReader i = new BufferedReader(new InputStreamReader(s.getInputStream()));
outToServer.write(st);
String get;
get = i.readLine();
System.out.println("FROM SERVER: " + get);
s.close();
}
}
P.S: How would you do this really? Every hex command make hand by hand, this PDF file has almost 100 commands, it would take lot of time. Or you manage them in a different way?

As #Rocky mentioned, it looks like in your code you are confusing bits and bytes.
It might help if you think of a byte in binary:
Binary Decimal Hex
00000000 0 0x00
00000001 1 0x01
00110000 48 0x30
If you look at the binary representation, you count the bits from the right: A byte has 8 bits, so bit 7 is the left-most bit and bit 0 is the right-most bit.
The reason why hexadecimal (base-16) notation is so convenient is because it is easier to convert between binary to hex than binary to hex.
Take the binary number 00110000. If you split these up into two parts (0011) and (0000) called the high nibble (bits 7-4) and the low nibble (bits 3-0). Then you can easily convert the two nibbles into hex:
Nibble Hex Decimal
0000 0 0
0001 1 1
0010 2 2
0011 3 3
0100 4 4
0101 5 5
0110 6 6
0111 7 7
1000 8 8
1001 9 9
1010 A 10
1011 B 11
1100 C 12
1101 D 13
1110 E 14
1111 F 15
Putting the two nibbles together, you can see the relationship between hex and binary:
Binary
0011 1100
Hex
3 C
so binary 00110100 = hex 34 = dec 60
So back to your binary format:
In the request packet, you are getting the response (hex 30), so if you convert that into your bit:
Hex 30 = binary 0011 0000
You can see that bit 5 and 4 are set.
In order to dynamically set bits in a byte, you'll need to use boolean logic AND and OR. See the following for results of and and or on a single bit:
Bit Bit Result Result Result
A B AND OR XOR
0 0 0 0 0
0 1 0 1 1
1 0 0 1 1
1 1 1 1 0
You've also got the NOT operation
Bit Result NOT
0 1
1 0
With multiple bits, you just perform the operation on each bit (bit 7 to 0), for example:
01101010 (hex 6A)
AND 11100110 (hex E6)
= 01100010 (hex 62)
01101010 (hex 6A)
OR 11100110 (hex E6)
= 11101110 (hex EE)
NOT 00111001 (hex 3B)
= 11000110 (hex C6)
So with that in mind, you can use the following operations to set and clear individual bits in a byte:
If you want to ensure that bit 6 is set (1), you just have to OR it with 01000000 (hex 40)
xxxxxxxx (any value)
OR 01000000 (hex 40)
= x1xxxxxx
If you wanted to ensure that bit 6 is clear (0), you just have to AND it with NOT (hex 40), so
NOT 01000000 (hex 40)
= 10111111 (hex BF)
xxxxxxxx (any value)
AND 10111111 (hex BF)
= x0xxxxxx
To put all this into Java code, you've got the following binary operators:
| binary OR
& binary AND
~ binary NOT
So if you wanted to set a bit in a byte:
byte anyByte;
anyByte = anyByte | 0x40;
which can be shortened to
anyByte |= 0x40;
If you want to clear a bit:
anyByte &= ~0x40;
If you want to test whether a bit is set, you'd use the following:
if ((anyByte & 0x40) == 0x40) ...
If you want to test whether bit 4 and bit 1 was set, you'd do the following:
if ((anyByte & 0x12) == 0x12) ...
Why 0x12? Because hex 12 is binary 0001 0010 which "masks" bit 4 and 1.
Back to your question:
In order to send the correct command string, you just need to create the right byte array as specified in the manual, though I hope it is clearer now how to set bits in bytes:
Socket s = new Socket("192.168.1.2", 49137);
InputStream in = s.getInputStream());
// send the request
// (Note, for different requests, you'll need different, byte arrays and may
// even have different lengths (can't tell without having seen the whole manual)
byte[] st = new byte[] { 0x01, 0x30, 0x01, 0x00 };
OutputStream out = s.getOutputStream();
out.write(st);
out.flush();
// read the first header byte (bits indicate the rest of the header structure)
byte header = (byte)in.read();
// bit 1 shows whether the header represented in 2 bytes
boolean header2Byte = (header & 0x2) != 0;
// bit 2 shows whether the length is represented in 2 bytes
boolean len2Byte = (header & 0x4) != 0;
if (header2Byte) {
// need to read the extra header (discarded here)
in.read();
}
// missed off reading the command byte/s
int len = 0;
if (len2Byte) {
byte[] lenByte = new byte[2];
in.read(lenByte);
if (isLittleEndian) {
len = (lenByte[1] << 8) + lenByte[0];
} else {
len = (lenByte[0] << 8) + lenByte[1];
}
} else {
// only one byte signifies the length
len = is.read();
}
byte[] data = new byte[len];
in.read(data);
// it is unclear what format the data, has but if it is a string, and encoded as
// UTF-8, then you can use the following
String stringData = new String(data, "UTF-8");
System.out.println(stringData);
// note, try-catch-finally omitted for brevity
in.close();
out.close();
s.close();
Note, I'm not using a DataInputStream here as the way Java encodes integers may be different from how the device encodes it's integers. E.g. in Java and integer is 4 bytes and Big Endian (see also this SO article).
N.B. the << operator is the left shift operator, which shifts the bits along in a byte and the above case is used to combine two bytes into a 16-bit number. Left-shifting by 8 is equivalent to multiplying by 256.

In such diagram header block shows mostly like "Bit 7, Bit 6,..,Bit 0"
instead of "Bit 0, Bit 1, Bit 2, ... Bit 7", i always wondering
why?.
In typical number writing 0 is the least significant bit 7 the most significant and the byte is written in most significant to least significant 7-0.
byte 11111111
place 76543210
But when i apply in code what is following order for byte st[0] = Bit 7 or Bit 1?
In your code this is not setting bits this is setting a byte in the byte array
If you are nervous about setting bits try a class like this:
public class SimpleByteSetter {
/* b is is byte you are setting
on is if the bit is set to on or 1
place is the bit place in the range of 0-7
*/
public static byte set(final byte b, final boolean on, final int place) {
if (on) { return (byte) (b | ((1 << place) & 0xFF)); }
return (byte) (b & (~((1 << place) & 0xFF)));
}
// 1 == on everything else off (but only use 0!)
public static byte set(final byte b, final int on, final int place) {
return set(b, 1==on, place);
}
}
use it in you code like:
byte header = 0;
// get = 0, set = 1, place = 0
header = SimpleByteSetter(header, 0, 0 ); // get
// header 2 byte = 0, header 2 byte = 1, place = 1
header = SimpleByteSetter(header, 0, 1 ); // header 1 byte
...
st[0] = header;
Also according to this diagram, does it mean every command i send will have Header fixed always
Yes
This is the code i was trying by taking Bit 1 as st[0], Bit 2 as st1 instead of Bit 7 as st[0]. To apply Power off/on test.
I don't have enough information to actually build the packet but:
// the header is 1 byte I don't see how to make it two
// the command is 2 bytes per the table
// the parameter length is 0 so 1 byte (is there suppose to be a device id?)
byte[] st = new byte[ 1 + 2 + 1 ];
byte header = 0;
// get = 0, set = 1, place = 0
header = SimpleByteSetter(header, 0, 0 ); // get
// header 2 byte = 0, header 2 byte = 1, place = 1
header = SimpleByteSetter(header, 0, 1 ); // header 1 byte
// length 1 byte = 0, length 2 byte = 1
header = SimpleByteSetter(header, 0, 2 );
// command 1 byte = 0, command 2 byte = 1;
header = SimpleByteSetter(header, 1, 3 );
st[0] = header;
st[1] = 0x0130 & 0xFF; // poweroff command first byte
st[2] = 0x0100 & 0xFF; // poweroff second byte
st[3] = 0;

You don't need to care about bit order because IP packets contain bytes and the lower level components make sure every byte is transferred correctly.
Instead of System.out.println(st) I'd make a small method that prints the command buffer nicely in hexadeciaml.

How does the call to the hashing algorithm work, particularly the use of bit shifting 0xff?

The following code snippet, sourced from Core Java Vol 2 (7th Ed), shows how to create an SHA1 and an MD5 fingerprint using Java.
It turns out that the only function that works is when I load the cleartext from a textfile.
How does MessageDigestFrame.computeDigest() work out the fingerprint, and, particularly the use of the bit shifting pattern (Line 171 - 172)?
public void computeDigest(byte[] b)
{
currentAlgorithm.reset();
currentAlgorithm.update(b);
byte[] hash = currentAlgorithm.digest();
String d = "";
for (int i = 0; i < hash.length; i++)
{
int v = hash[i] & 0xFF;
if (v < 16) d += "0";
d += Integer.toString(v, 16).toUpperCase() + " ";
}
digest.setText(d);
}

The method should work fine whatever you give it - if you're getting the wrong results, I suspect you're loading the file incorrectly. Please show that code, and we can help you work out what's going wrong.
In terms of the code, this line:
int v = hash[i] & 0xFF;
is basically used to treat a byte as unsigned. Bytes are signed in Java - an acknowledged design mistake in the language - but we want to print out the hex value as if it were an unsigned integer. The bitwise AND with just the bottom 8 bits effectively converts it to the integer value of the byte treated as unsigned.
(There are better ways to convert a byte array to a hex string, but that's a separate matter.)

It is not bit shifting, it is bit masking. hash[i] is a byte. When it is widened to integer you need to mask off the higher integer bits because of possible sign extension.
byte b = (byte)0xEF;
System.out.println("No masking: " + (int)b);
System.out.println("Masking: " + (int)(b & 0xFF));

This snipped:
int v = hash[i] & 0xFF;
if (v < 16) d += "0";
d += Integer.toString(v, 16).toUpperCase() + " ";
First you set all but the lowest 8 bits of v to 0 (because 0xFF is 11111111 in binary).
Then if the resulting number is only one digit in hex (< 16) you add a leading "0".
Finally convert the result to hex and add it to the string.