Develop Reference

How can I understand the byte operation between Java and Nodejs [duplicate]

I'm reading a file into a byte array in chunks and sending it over the network via a POST request to a webserver. It's not anything complicated, I've done it before using this exact same code. This time, I noticed that my images are looking really odd when they get to the server, so I decided to look at the byte array being sent and the one being received just to make sure it was the same. It's not. On the java sending side the byte array contains negative numbers. On the C# receiving side, there are no negative numbers.
The first 15 bytes on the receiving side (C#)
137
80
78
71
13
10
26
10
0
0
0
13
73
72
68
Those same bytes but on the sending side (java)
-119
80
78
71
13
10
26
10
0
0
0
13
73
72
68
All of the non-negative numbers are the same, and the -119 isn't the only negative number, they are all over. I did notice that -119 and 137 are 256 apart and wondered if that has something to do with it.
The code I'm using to read the image (java)
public static byte[] readPart(String fileName, long offset, int length) throws FileNotFoundException, Exception
{
byte[] data = new byte[length];
File file = new File(fileName);
InputStream is = new FileInputStream(file);
is.skip(offset);
is.read(data,0,data.length);
is.close();
return data;
}
The code I'm using to write the data (c#)
private void writeFile(string fileName, Stream contents)
{
using (FileStream fs = new FileStream(fileName, FileMode.Append, FileAccess.Write, FileShare.ReadWrite))
{
int bufferLen = 65000;
byte[] buffer = new byte[bufferLen];
int count = 0;
while ((count = contents.Read(buffer, 0, bufferLen)) > 0)
{
fs.Write(buffer, 0, count);
}
fs.Close();
}
contents.Close();
}
I don't know if that is something that always happens and I just never noticed it before or if it is something that decided to go horribly wrong. What I do know is that this code worked before for something very similar and that it's not working now.
If anyone has any suggestions or an explanation I would really appreciate it.
EDIT:
The reason my images were looking odd is how I was calling the readPart method.
byte[] data = FileUtilities.readPart(fileName,counter,maxFileSize);//counter is the current chunk number
How I should have been calling it
byte[] data = FileUtilities.readPart(fileName,counter*maxFileSize,maxFileSize);//the current chunk * cuhnksize for the offset...
Thanks everyone, I'm significantly less confused now :)

In Java, byte is a signed value (using two's complement to encode negative values), so what you see it correct if unexpected by most people.
To convert a byte to an unsigned int value, use b & 0xff

Java doesn't have unsigned bytes; all bytes are treated as signed. That's all.
All that really matters is how you think of the bytes, since you rarely ever actually need to do comparisons on bytes. The only significant difference is that they print out as signed, as you've discovered.
If you like, you can use e.g. Guava's UnsignedBytes utilities to view Java bytes as unsigned, but there's really not much practical difference.

As a further explanation, assume you have 137 as an unsigned byte. That is represented as:
1000 1001
This binary value, when expressed as a signed two's complement number, turns out to be -119. (-128 + 9)
Any unsigned byte values over 128 will be affected by the difference since the left-most bit is used in this way by the two's complement scheme.

Maybe it has something to do with the fact that Java's byte is signed (range -128 to 127) while C#'s is unsigned (0 to 255) :). The information is the same in binary, it's just interpreted differently.

The range of byte is from -128 to 127, so if you try to assign a byte 128, it will cycle around and the result will be -128.
System.out.println("Max val = " + Byte.MAX_VALUE); //prints: Max val = 127
System.out.println("Min val = " + Byte.MIN_VALUE); //prints: Min val = -128
System.out.println("(byte)137 = " + (byte)137); //prints: (byte)137 = -119
System.out.println("(byte)128 = " + (byte)128); //prints: (byte)128 = -128
System.out.println("(byte)-129 = " + (byte)-129); //prints: (byte)-129 = 127

Java not treating 0xFC as a signed integer?

Just when I though I had a fair grasp on how Java treats all Integers/Bytes etc.. as signed numbers, it hit me with another curve ball and got me thinking if I really understand this treatment after all.
This is a piece of assembly code that is supposed to jump to an address if a condition is met (which indeed is met). PC just before the jump is C838h and then after condition check it is supposed to be: C838h + FCh (h = hex) which I thought would be treated as signed so the PC would jump backwards: FCh = -4 in two's compliment negative number. But To my surprise, java ADDED FCh to the PC making it incorrectly jump to C934h instead of back to C834h.
C832: LD B,04 06 0004 (PC=C834)
C834: LD (HL), A 77 9800 (PC=C835)
C835: INC L:00 2C 0001 (PC=C836)
C836: JR NZ, n 20 00FC (PC=C934)
I tested this in a java code and indeed the result was the same:
int a = 0xC838;
int b = 0xFC;
int result = a + b;
System.out.printf("%04X\n", result); //prints C934 (incorrect)
To fix this I had to cast FCh into a byte after checking if the first bit is 1, which it is: 11111100
int a = 0xC838;
int b = (byte) 0xFC;
int result = a + b;
System.out.printf("%04X\n", result); //prints C834 (correct)
In short I guess my question is that I thought java would know that FCh is a negative number but that is not the case unless I cast it to a byte. Why? Sorry I know this question is asked many times and I seem to be asking it myself alot.

0xfc is a positive number. If you want a negative number, then write a negative number. -0x4 would do just fine.
But if you want to apply this to non-constant data, you'll need to tell Java that you want it sign-extended in some way.
The core of the problem is that you have a 32-bit signed integer, but you want it treated like an 8-bit signed integer. The easiest way to achieve that would be to just use byte as you did above.
If you really don't want to write byte, you can write (0xfc << 24) >> 24:
class Main
{
public static void main(String[] args)
{
int a = 0xC838;
int b = (0xfc << 24) >> 24;
int result = a + b;
System.out.printf("%04X\n", result);
}
}
(The 24 derives from the difference of the sizes of int (32 bits) and byte (8 bits)).

What is the purpose of low and high nibble when converting a string to a HexString

Recently I have been going through some examples of MD5 to start getting an understanding of security and MD5 has been fairly simple to understand for the most part and a good starting point even though it is no longer secure. Despite this I have a question regarding high and lo nibbles when it comes to converting a string to a hex string.
So I know high and low nibbles are equal to half a byte or also can be hex digits that represent a single hexadecimal digit. What I am not understanding though is exactly how they work and the purpose that they serve. I have been searching on google and I can't find much of an answer that will help explain what they do in the context that they are in. Here is the context of the conversion:
private static String toHexString( byte[] byteArray )
{
final String HEX_CHARS = "0123456789ABCDEF";
byte[] result = new byte[byteArray.length << 1];
int len = byteArray.length;
for( int i = 0 ; i < len ; i++ )
{
byte b = byteArray[i]
int lo4 = b & 0x0F;
int hi4 = ( b & 0xF0 ) >> 4;
result[i * 2] = (byte)HEX_CHARS.charAt( hi4 );
result[i * 2 + 1] = (byte)HEX_CHARS.charAt( lo4 );
}
return new String( result );
}
I don't exactly understand what is going on in the for statement. I would appreciate any help understanding this and if there is some link to some places that I can learn more about this please also leave it.
I understand the base definition of nibble but not the operations and what the assignment to the number 4 is doing either.
If I need to post the full example code I will just ask as I am unsure if it is needed.

This code simply converts a byte array to hexadecimal representation. Within the for-loop, each byte is converted into two characters. I think it's easier to understand it on an example.
Assume one of the bytes in your array is, say, 218 (unsigned). That's 1101 1010 in binary.
lo4 gets the lowest 4 bits by AND-ing the byte with the bitmask 00001111:
int lo4 = b & 0x0F;
This results in 1010, 10 in decimal.
hi4 gets the highest 4 bits by AND-ing with the bitmask 1111 0000 and shifting 4 bits to the right:
int hi4 = ( b & 0xF0 ) >> 4;
This results in 1101, 13 in decimal.
Now to get the hexadecimal representation of this byte you only need to convert 10 and 13 to their hexadecimal representations and concatenate. For this you simply look up the character in the prepared HEX_CHARS string at the specific index. 10 -> A, 13 -> D, resulting in 218 -> DA.

It's just bit operations. The & character takes the literal bit value of each and does a logical and on them.
int lo4 = b & 0x0F;
for instance if b = 24 then it will evaluate to this
00011000
+00001111
=00001000
The second such line does the same on the first four bits.
00011000
+11110000
=00010000
the '>>' shifts all of the bits a certain number in that direction so
00010000 >> 4 = 00000001.
This is done so that you can derive the hex value from the number. Since each character in hex can represent 4 bits by splitting the number into pieces of 4 bits we can convert it.
in the case of b = 24 we no have lo4 = 1000 or 8 and hi4 = 0001 or 1. The last part of the loop assigns the character value for each.
Hex_chars[hi4] = '1' and Hex_chars[lo4] = '8' which gives you "18" for that part of the string which is 24 in hex.

Java byte array contains negative numbers

I'm reading a file into a byte array in chunks and sending it over the network via a POST request to a webserver. It's not anything complicated, I've done it before using this exact same code. This time, I noticed that my images are looking really odd when they get to the server, so I decided to look at the byte array being sent and the one being received just to make sure it was the same. It's not. On the java sending side the byte array contains negative numbers. On the C# receiving side, there are no negative numbers.
The first 15 bytes on the receiving side (C#)
137
80
78
71
13
10
26
10
0
0
0
13
73
72
68
Those same bytes but on the sending side (java)
-119
80
78
71
13
10
26
10
0
0
0
13
73
72
68
All of the non-negative numbers are the same, and the -119 isn't the only negative number, they are all over. I did notice that -119 and 137 are 256 apart and wondered if that has something to do with it.
The code I'm using to read the image (java)
public static byte[] readPart(String fileName, long offset, int length) throws FileNotFoundException, Exception
{
byte[] data = new byte[length];
File file = new File(fileName);
InputStream is = new FileInputStream(file);
is.skip(offset);
is.read(data,0,data.length);
is.close();
return data;
}
The code I'm using to write the data (c#)
private void writeFile(string fileName, Stream contents)
{
using (FileStream fs = new FileStream(fileName, FileMode.Append, FileAccess.Write, FileShare.ReadWrite))
{
int bufferLen = 65000;
byte[] buffer = new byte[bufferLen];
int count = 0;
while ((count = contents.Read(buffer, 0, bufferLen)) > 0)
{
fs.Write(buffer, 0, count);
}
fs.Close();
}
contents.Close();
}
I don't know if that is something that always happens and I just never noticed it before or if it is something that decided to go horribly wrong. What I do know is that this code worked before for something very similar and that it's not working now.
If anyone has any suggestions or an explanation I would really appreciate it.
EDIT:
The reason my images were looking odd is how I was calling the readPart method.
byte[] data = FileUtilities.readPart(fileName,counter,maxFileSize);//counter is the current chunk number
How I should have been calling it
byte[] data = FileUtilities.readPart(fileName,counter*maxFileSize,maxFileSize);//the current chunk * cuhnksize for the offset...
Thanks everyone, I'm significantly less confused now :)

In Java, byte is a signed value (using two's complement to encode negative values), so what you see it correct if unexpected by most people.
To convert a byte to an unsigned int value, use b & 0xff

Java doesn't have unsigned bytes; all bytes are treated as signed. That's all.
All that really matters is how you think of the bytes, since you rarely ever actually need to do comparisons on bytes. The only significant difference is that they print out as signed, as you've discovered.
If you like, you can use e.g. Guava's UnsignedBytes utilities to view Java bytes as unsigned, but there's really not much practical difference.

As a further explanation, assume you have 137 as an unsigned byte. That is represented as:
1000 1001
This binary value, when expressed as a signed two's complement number, turns out to be -119. (-128 + 9)
Any unsigned byte values over 128 will be affected by the difference since the left-most bit is used in this way by the two's complement scheme.

Maybe it has something to do with the fact that Java's byte is signed (range -128 to 127) while C#'s is unsigned (0 to 255) :). The information is the same in binary, it's just interpreted differently.

The range of byte is from -128 to 127, so if you try to assign a byte 128, it will cycle around and the result will be -128.
System.out.println("Max val = " + Byte.MAX_VALUE); //prints: Max val = 127
System.out.println("Min val = " + Byte.MIN_VALUE); //prints: Min val = -128
System.out.println("(byte)137 = " + (byte)137); //prints: (byte)137 = -119
System.out.println("(byte)128 = " + (byte)128); //prints: (byte)128 = -128
System.out.println("(byte)-129 = " + (byte)-129); //prints: (byte)-129 = 127

How to send such complicated hex, binary protocol data accurately using java byte[]?

I am very confused with this kind of stuffs, What should I send as final command, get always confused 8 bits to 1 byte but how do we make it? Is it only the command packet [hex] as shown in screen shot? or Is it header + command packet[hex] shown in two screen shots?
Confusion details:
In such diagram header block shows mostly like "Bit 7, Bit 6,..,Bit 0" instead of "Bit 0, Bit 1, Bit 2, ... Bit 7", i always wondering why?.
But when I apply in code what is following order for byte st[0] = Bit 7 or Bit 1?
Also according to this diagram, does it mean every command I send will have Header fixed always?
This is the code I was trying by taking Bit 1 as st[0], Bit 2 as st1 instead of Bit 7 as st[0]. To apply Power off/on test.
import java.io.*;
import java.net.*;
public class test
{
public static void main(String[] args) throws UnknownHostException, IOException
{
byte st[]=new byte[256];
st[0]=0x01; // get
st[1]=0x02; // header 2 byte
st[2]=0x02; // header length
st[3]=0; // command byte
st[4]=0; // reserved
st[5]=0;
st[6]=0;
st[7]=0;
st[8]=0x01; // power off
st[9]=0x30;
st[10]=0x01;
st[11]=0x00;
System.out.println(st); // Should this work, am i correct based on diagram?
Socket s = new Socket("192.168.1.2", 49137);
DataInputStream input = new DataInputStream(s.getInputStream());
DataOutputStream outToServer = new DataOutputStream(s.getOutputStream());
BufferedReader i = new BufferedReader(new InputStreamReader(s.getInputStream()));
outToServer.write(st);
String get;
get = i.readLine();
System.out.println("FROM SERVER: " + get);
s.close();
}
}
P.S: How would you do this really? Every hex command make hand by hand, this PDF file has almost 100 commands, it would take lot of time. Or you manage them in a different way?

As #Rocky mentioned, it looks like in your code you are confusing bits and bytes.
It might help if you think of a byte in binary:
Binary Decimal Hex
00000000 0 0x00
00000001 1 0x01
00110000 48 0x30
If you look at the binary representation, you count the bits from the right: A byte has 8 bits, so bit 7 is the left-most bit and bit 0 is the right-most bit.
The reason why hexadecimal (base-16) notation is so convenient is because it is easier to convert between binary to hex than binary to hex.
Take the binary number 00110000. If you split these up into two parts (0011) and (0000) called the high nibble (bits 7-4) and the low nibble (bits 3-0). Then you can easily convert the two nibbles into hex:
Nibble Hex Decimal
0000 0 0
0001 1 1
0010 2 2
0011 3 3
0100 4 4
0101 5 5
0110 6 6
0111 7 7
1000 8 8
1001 9 9
1010 A 10
1011 B 11
1100 C 12
1101 D 13
1110 E 14
1111 F 15
Putting the two nibbles together, you can see the relationship between hex and binary:
Binary
0011 1100
Hex
3 C
so binary 00110100 = hex 34 = dec 60
So back to your binary format:
In the request packet, you are getting the response (hex 30), so if you convert that into your bit:
Hex 30 = binary 0011 0000
You can see that bit 5 and 4 are set.
In order to dynamically set bits in a byte, you'll need to use boolean logic AND and OR. See the following for results of and and or on a single bit:
Bit Bit Result Result Result
A B AND OR XOR
0 0 0 0 0
0 1 0 1 1
1 0 0 1 1
1 1 1 1 0
You've also got the NOT operation
Bit Result NOT
0 1
1 0
With multiple bits, you just perform the operation on each bit (bit 7 to 0), for example:
01101010 (hex 6A)
AND 11100110 (hex E6)
= 01100010 (hex 62)
01101010 (hex 6A)
OR 11100110 (hex E6)
= 11101110 (hex EE)
NOT 00111001 (hex 3B)
= 11000110 (hex C6)
So with that in mind, you can use the following operations to set and clear individual bits in a byte:
If you want to ensure that bit 6 is set (1), you just have to OR it with 01000000 (hex 40)
xxxxxxxx (any value)
OR 01000000 (hex 40)
= x1xxxxxx
If you wanted to ensure that bit 6 is clear (0), you just have to AND it with NOT (hex 40), so
NOT 01000000 (hex 40)
= 10111111 (hex BF)
xxxxxxxx (any value)
AND 10111111 (hex BF)
= x0xxxxxx
To put all this into Java code, you've got the following binary operators:
| binary OR
& binary AND
~ binary NOT
So if you wanted to set a bit in a byte:
byte anyByte;
anyByte = anyByte | 0x40;
which can be shortened to
anyByte |= 0x40;
If you want to clear a bit:
anyByte &= ~0x40;
If you want to test whether a bit is set, you'd use the following:
if ((anyByte & 0x40) == 0x40) ...
If you want to test whether bit 4 and bit 1 was set, you'd do the following:
if ((anyByte & 0x12) == 0x12) ...
Why 0x12? Because hex 12 is binary 0001 0010 which "masks" bit 4 and 1.
Back to your question:
In order to send the correct command string, you just need to create the right byte array as specified in the manual, though I hope it is clearer now how to set bits in bytes:
Socket s = new Socket("192.168.1.2", 49137);
InputStream in = s.getInputStream());
// send the request
// (Note, for different requests, you'll need different, byte arrays and may
// even have different lengths (can't tell without having seen the whole manual)
byte[] st = new byte[] { 0x01, 0x30, 0x01, 0x00 };
OutputStream out = s.getOutputStream();
out.write(st);
out.flush();
// read the first header byte (bits indicate the rest of the header structure)
byte header = (byte)in.read();
// bit 1 shows whether the header represented in 2 bytes
boolean header2Byte = (header & 0x2) != 0;
// bit 2 shows whether the length is represented in 2 bytes
boolean len2Byte = (header & 0x4) != 0;
if (header2Byte) {
// need to read the extra header (discarded here)
in.read();
}
// missed off reading the command byte/s
int len = 0;
if (len2Byte) {
byte[] lenByte = new byte[2];
in.read(lenByte);
if (isLittleEndian) {
len = (lenByte[1] << 8) + lenByte[0];
} else {
len = (lenByte[0] << 8) + lenByte[1];
}
} else {
// only one byte signifies the length
len = is.read();
}
byte[] data = new byte[len];
in.read(data);
// it is unclear what format the data, has but if it is a string, and encoded as
// UTF-8, then you can use the following
String stringData = new String(data, "UTF-8");
System.out.println(stringData);
// note, try-catch-finally omitted for brevity
in.close();
out.close();
s.close();
Note, I'm not using a DataInputStream here as the way Java encodes integers may be different from how the device encodes it's integers. E.g. in Java and integer is 4 bytes and Big Endian (see also this SO article).
N.B. the << operator is the left shift operator, which shifts the bits along in a byte and the above case is used to combine two bytes into a 16-bit number. Left-shifting by 8 is equivalent to multiplying by 256.

In such diagram header block shows mostly like "Bit 7, Bit 6,..,Bit 0"
instead of "Bit 0, Bit 1, Bit 2, ... Bit 7", i always wondering
why?.
In typical number writing 0 is the least significant bit 7 the most significant and the byte is written in most significant to least significant 7-0.
byte 11111111
place 76543210
But when i apply in code what is following order for byte st[0] = Bit 7 or Bit 1?
In your code this is not setting bits this is setting a byte in the byte array
If you are nervous about setting bits try a class like this:
public class SimpleByteSetter {
/* b is is byte you are setting
on is if the bit is set to on or 1
place is the bit place in the range of 0-7
*/
public static byte set(final byte b, final boolean on, final int place) {
if (on) { return (byte) (b | ((1 << place) & 0xFF)); }
return (byte) (b & (~((1 << place) & 0xFF)));
}
// 1 == on everything else off (but only use 0!)
public static byte set(final byte b, final int on, final int place) {
return set(b, 1==on, place);
}
}
use it in you code like:
byte header = 0;
// get = 0, set = 1, place = 0
header = SimpleByteSetter(header, 0, 0 ); // get
// header 2 byte = 0, header 2 byte = 1, place = 1
header = SimpleByteSetter(header, 0, 1 ); // header 1 byte
...
st[0] = header;
Also according to this diagram, does it mean every command i send will have Header fixed always
Yes
This is the code i was trying by taking Bit 1 as st[0], Bit 2 as st1 instead of Bit 7 as st[0]. To apply Power off/on test.
I don't have enough information to actually build the packet but:
// the header is 1 byte I don't see how to make it two
// the command is 2 bytes per the table
// the parameter length is 0 so 1 byte (is there suppose to be a device id?)
byte[] st = new byte[ 1 + 2 + 1 ];
byte header = 0;
// get = 0, set = 1, place = 0
header = SimpleByteSetter(header, 0, 0 ); // get
// header 2 byte = 0, header 2 byte = 1, place = 1
header = SimpleByteSetter(header, 0, 1 ); // header 1 byte
// length 1 byte = 0, length 2 byte = 1
header = SimpleByteSetter(header, 0, 2 );
// command 1 byte = 0, command 2 byte = 1;
header = SimpleByteSetter(header, 1, 3 );
st[0] = header;
st[1] = 0x0130 & 0xFF; // poweroff command first byte
st[2] = 0x0100 & 0xFF; // poweroff second byte
st[3] = 0;

You don't need to care about bit order because IP packets contain bytes and the lower level components make sure every byte is transferred correctly.
Instead of System.out.println(st) I'd make a small method that prints the command buffer nicely in hexadeciaml.