Develop Reference

How can I understand the byte operation between Java and Nodejs [duplicate]

I'm reading a file into a byte array in chunks and sending it over the network via a POST request to a webserver. It's not anything complicated, I've done it before using this exact same code. This time, I noticed that my images are looking really odd when they get to the server, so I decided to look at the byte array being sent and the one being received just to make sure it was the same. It's not. On the java sending side the byte array contains negative numbers. On the C# receiving side, there are no negative numbers.
The first 15 bytes on the receiving side (C#)
137
80
78
71
13
10
26
10
0
0
0
13
73
72
68
Those same bytes but on the sending side (java)
-119
80
78
71
13
10
26
10
0
0
0
13
73
72
68
All of the non-negative numbers are the same, and the -119 isn't the only negative number, they are all over. I did notice that -119 and 137 are 256 apart and wondered if that has something to do with it.
The code I'm using to read the image (java)
public static byte[] readPart(String fileName, long offset, int length) throws FileNotFoundException, Exception
{
byte[] data = new byte[length];
File file = new File(fileName);
InputStream is = new FileInputStream(file);
is.skip(offset);
is.read(data,0,data.length);
is.close();
return data;
}
The code I'm using to write the data (c#)
private void writeFile(string fileName, Stream contents)
{
using (FileStream fs = new FileStream(fileName, FileMode.Append, FileAccess.Write, FileShare.ReadWrite))
{
int bufferLen = 65000;
byte[] buffer = new byte[bufferLen];
int count = 0;
while ((count = contents.Read(buffer, 0, bufferLen)) > 0)
{
fs.Write(buffer, 0, count);
}
fs.Close();
}
contents.Close();
}
I don't know if that is something that always happens and I just never noticed it before or if it is something that decided to go horribly wrong. What I do know is that this code worked before for something very similar and that it's not working now.
If anyone has any suggestions or an explanation I would really appreciate it.
EDIT:
The reason my images were looking odd is how I was calling the readPart method.
byte[] data = FileUtilities.readPart(fileName,counter,maxFileSize);//counter is the current chunk number
How I should have been calling it
byte[] data = FileUtilities.readPart(fileName,counter*maxFileSize,maxFileSize);//the current chunk * cuhnksize for the offset...
Thanks everyone, I'm significantly less confused now :)

In Java, byte is a signed value (using two's complement to encode negative values), so what you see it correct if unexpected by most people.
To convert a byte to an unsigned int value, use b & 0xff

Java doesn't have unsigned bytes; all bytes are treated as signed. That's all.
All that really matters is how you think of the bytes, since you rarely ever actually need to do comparisons on bytes. The only significant difference is that they print out as signed, as you've discovered.
If you like, you can use e.g. Guava's UnsignedBytes utilities to view Java bytes as unsigned, but there's really not much practical difference.

As a further explanation, assume you have 137 as an unsigned byte. That is represented as:
1000 1001
This binary value, when expressed as a signed two's complement number, turns out to be -119. (-128 + 9)
Any unsigned byte values over 128 will be affected by the difference since the left-most bit is used in this way by the two's complement scheme.

Maybe it has something to do with the fact that Java's byte is signed (range -128 to 127) while C#'s is unsigned (0 to 255) :). The information is the same in binary, it's just interpreted differently.

The range of byte is from -128 to 127, so if you try to assign a byte 128, it will cycle around and the result will be -128.
System.out.println("Max val = " + Byte.MAX_VALUE); //prints: Max val = 127
System.out.println("Min val = " + Byte.MIN_VALUE); //prints: Min val = -128
System.out.println("(byte)137 = " + (byte)137); //prints: (byte)137 = -119
System.out.println("(byte)128 = " + (byte)128); //prints: (byte)128 = -128
System.out.println("(byte)-129 = " + (byte)-129); //prints: (byte)-129 = 127

Converting int to byte[]

I'm reading a value from the screen in Android and try to convert it to a byte[] to send it over Bluetooth but every time I it takes 128 from the screen, it converts it to -128 in the byte[] and than I don't get anything on the other side..What is the problem..Why is it giving negative number?..
This is the convert code :
ByteBuffer.allocate(4).putInt(yourInt).array();
EDIT : On the other side I have to transform the byte[] to String.
And another problem..If I lower the number in the allocate() I get a BufferOverflowException even though I use only 1 position in the array. Why?
EDIT 2 : I'm working with 8bit numbers (0-255)

public static void main(String[] args) {
byte[] num = BigInteger.valueOf(-2147483648).toByteArray();
System.out.println(new BigInteger(num).intValue());
}
if you want to transfer only one byte, valued 0-255, then
public static void main(String[] args) {
for(int i=0; i<=255; i++){
byte a = (byte) i;
byte[] num = new byte [] {(byte) a};
System.out.println(num[0] + " : " + i + " : " + ((256+num[0]) % 256));
}
}
The second code will convert a byte variable to equal integer output.

The reason the code is doing this is because your integer (128) looks like
1000000 in binary with a bunch of leading zeroes. Since all Java primitives are signed, the leading bit is 0 and this comes out OK. However, when you convert to bytes the leading bit gets interpreted as the sign bit, so the last byte is interpreted as negative.
The documentation is also pretty clear that putInt() writes all 4 bytes of the integer to the buffer, instead of only the bytes which are "used".

You are seeing the correct values in the ByteBuffer. The last byte of the buffer has the byte value 0x80, which if you sign extend to a longer int value APPEARS as -128 when displaying in a dumb terminal.
BYTE VALUES
1000 0000
This hight bit typically designates a negative number in signed byte/short/int systems.
If you convert this signed byte to a larger type like an int by just prepending 1 bits, you would see this:
1111 1111 1111 1111 1111 1111 1000 0000
Which indicates -128 as a signed int.

How to convert binary string to the byte array of 2 bytes in java

I have binary string String A = "1000000110101110". I want to convert this string into byte array of length 2 in java
I have taken the help of this link
I have tried to convert it into byte by various ways
I have converted that string into decimal first and then apply the code to store into the byte array
int aInt = Integer.parseInt(A, 2);
byte[] xByte = new byte[2];
xByte[0] = (byte) ((aInt >> 8) & 0XFF);
xByte[1] = (byte) (aInt & 0XFF);
System.arraycopy(xByte, 0, record, 0,
xByte.length);
But the values get store into the byte array are negative
xByte[0] :-127
xByte[1] :-82
Which are wrong values.
2.I have also tried using
byte[] xByte = ByteBuffer.allocate(2).order(ByteOrder.BIG_ENDIAN).putInt(aInt).array();
But it throws the exception at the above line like
java.nio.Buffer.nextPutIndex(Buffer.java:519) at
java.nio.HeapByteBuffer.putInt(HeapByteBuffer.java:366) at
org.com.app.convert.generateTemplate(convert.java:266)
What should i do now to convert the binary string to byte array of 2 bytes?Is there any inbuilt function in java to get the byte array

The answer you are getting
xByte[0] :-127
xByte[1] :-82
is right.
This is called 2's compliment Represantation.
1st bit is used as signed bit.
0 for +ve
1 for -ve
if 1st bit is 0 than it calculates as regular.
but if 1st bit is 1 than it deduct the values of 7 bit from 128 and what ever the answer is presented in -ve form.
In your case
1st value is10000001
so 1(1st bit) for -ve and 128 - 1(last seven bits) = 127
so value is -127
For more detail read 2's complement representation.

Use putShort for putting a two byte value. int has four bytes.
// big endian is the default order
byte[] xByte = ByteBuffer.allocate(2).putShort((short)aInt).array();
By the way, your first attempt is perfect. You can’t change the negative sign of the bytes as the most significant bit of these bytes is set. That’s always interpreted as negative value.
10000001₂ == -127
10101110₂ == -82

try this
String s = "1000000110101110";
int i = Integer.parseInt(s, 2);
byte[] a = {(byte) ( i >> 8), (byte) i};
System.out.println(Arrays.toString(a));
System.out.print(Integer.toBinaryString(0xFF & a[0]) + " " + Integer.toBinaryString(0xFF & a[1]));
output
[-127, -82]
10000001 10101110
that is -127 == 0xb10000001 and -82 == 0xb10101110

Bytes are signed 8 bit integers. As such your result is completely correct.
That is: 01111111 is 127, but 10000000 is -128. If you want to get numbers in 0-255 range you need to use a bigger variable type like short.
You can print byte as unsigned like this:
public static String toString(byte b) {
return String.valueOf(((short)b) & 0xFF);
}

How to store unsigned short in java?

So generally I'm using Netty and it's BigEndianHeapChannelBuffer to receive unsinged short (from c++ software) in Java.
When I do it like this:
buf.readUnsignedByte(); buf.readUnsignedByte();
it returns:
149 and 00. Till now everything is fine. Because server sent 149 as unsigned short [2 bytes].
Instead of this I would like to receive unsigned short (ofc after restarting my application):
buf.readUnsignedShort();
and magic happens. It returns: 38144.
Next step is to retrieve unsigned byte:
short type = buf.readUnsignedByte();
System.out.println(type);
and it returns: 1 which is correct output.
Could anyone help me with this?
I looked deeper and this is what netty does with it:
public short readShort() {
checkReadableBytes(2);
short v = getShort(readerIndex);
readerIndex += 2;
return v;
}
public int readUnsignedShort() {
return readShort() & 0xFFFF;
}
But still I can't figure what is wrong. I would like just be able to read that 149.

You could also borrow a page from the Java DataInputStream.readUnsignedShort() implementation:
public final int readUnsignedShort() throws IOException {
int ch1 = in.read();
int ch2 = in.read();
if ((ch1 | ch2) < 0)
throw new EOFException();
return (ch1 << 8) + (ch2 << 0);
}

The answer is to change endianness. Thanks to Roger who in comments wrote:
Not much magic, 149*256+0=38144. You have specified BigEndian so this seems correct, the most significant byte is sent first
and:
#mickula The short is two bytes, where one of the bytes is "worth" 256 times as much as the other, Since you use BigEndian the first byte is the one "worth" more. Similar to the decimal number 123 of the digits 1, 2, and 3. The first position is with 10 times the next position which is worth 10 times the next, and so on. So 1 * 100 + 2 * 10 + 3 = 123 when the digits are transferred one at a time. If you see 1, 2, and 3 as little endian you would use 1 + 2 * 10 + 3 * 100 = 321. The 256 is because the size of a byte is 256. –
Thanks to his comments I just switched endianess in server bootstrap by adding:
bootstrap.setOption("child.bufferFactory", new
HeapChannelBufferFactory(ByteOrder.LITTLE_ENDIAN));

How to send such complicated hex, binary protocol data accurately using java byte[]?

I am very confused with this kind of stuffs, What should I send as final command, get always confused 8 bits to 1 byte but how do we make it? Is it only the command packet [hex] as shown in screen shot? or Is it header + command packet[hex] shown in two screen shots?
Confusion details:
In such diagram header block shows mostly like "Bit 7, Bit 6,..,Bit 0" instead of "Bit 0, Bit 1, Bit 2, ... Bit 7", i always wondering why?.
But when I apply in code what is following order for byte st[0] = Bit 7 or Bit 1?
Also according to this diagram, does it mean every command I send will have Header fixed always?
This is the code I was trying by taking Bit 1 as st[0], Bit 2 as st1 instead of Bit 7 as st[0]. To apply Power off/on test.
import java.io.*;
import java.net.*;
public class test
{
public static void main(String[] args) throws UnknownHostException, IOException
{
byte st[]=new byte[256];
st[0]=0x01; // get
st[1]=0x02; // header 2 byte
st[2]=0x02; // header length
st[3]=0; // command byte
st[4]=0; // reserved
st[5]=0;
st[6]=0;
st[7]=0;
st[8]=0x01; // power off
st[9]=0x30;
st[10]=0x01;
st[11]=0x00;
System.out.println(st); // Should this work, am i correct based on diagram?
Socket s = new Socket("192.168.1.2", 49137);
DataInputStream input = new DataInputStream(s.getInputStream());
DataOutputStream outToServer = new DataOutputStream(s.getOutputStream());
BufferedReader i = new BufferedReader(new InputStreamReader(s.getInputStream()));
outToServer.write(st);
String get;
get = i.readLine();
System.out.println("FROM SERVER: " + get);
s.close();
}
}
P.S: How would you do this really? Every hex command make hand by hand, this PDF file has almost 100 commands, it would take lot of time. Or you manage them in a different way?

As #Rocky mentioned, it looks like in your code you are confusing bits and bytes.
It might help if you think of a byte in binary:
Binary Decimal Hex
00000000 0 0x00
00000001 1 0x01
00110000 48 0x30
If you look at the binary representation, you count the bits from the right: A byte has 8 bits, so bit 7 is the left-most bit and bit 0 is the right-most bit.
The reason why hexadecimal (base-16) notation is so convenient is because it is easier to convert between binary to hex than binary to hex.
Take the binary number 00110000. If you split these up into two parts (0011) and (0000) called the high nibble (bits 7-4) and the low nibble (bits 3-0). Then you can easily convert the two nibbles into hex:
Nibble Hex Decimal
0000 0 0
0001 1 1
0010 2 2
0011 3 3
0100 4 4
0101 5 5
0110 6 6
0111 7 7
1000 8 8
1001 9 9
1010 A 10
1011 B 11
1100 C 12
1101 D 13
1110 E 14
1111 F 15
Putting the two nibbles together, you can see the relationship between hex and binary:
Binary
0011 1100
Hex
3 C
so binary 00110100 = hex 34 = dec 60
So back to your binary format:
In the request packet, you are getting the response (hex 30), so if you convert that into your bit:
Hex 30 = binary 0011 0000
You can see that bit 5 and 4 are set.
In order to dynamically set bits in a byte, you'll need to use boolean logic AND and OR. See the following for results of and and or on a single bit:
Bit Bit Result Result Result
A B AND OR XOR
0 0 0 0 0
0 1 0 1 1
1 0 0 1 1
1 1 1 1 0
You've also got the NOT operation
Bit Result NOT
0 1
1 0
With multiple bits, you just perform the operation on each bit (bit 7 to 0), for example:
01101010 (hex 6A)
AND 11100110 (hex E6)
= 01100010 (hex 62)
01101010 (hex 6A)
OR 11100110 (hex E6)
= 11101110 (hex EE)
NOT 00111001 (hex 3B)
= 11000110 (hex C6)
So with that in mind, you can use the following operations to set and clear individual bits in a byte:
If you want to ensure that bit 6 is set (1), you just have to OR it with 01000000 (hex 40)
xxxxxxxx (any value)
OR 01000000 (hex 40)
= x1xxxxxx
If you wanted to ensure that bit 6 is clear (0), you just have to AND it with NOT (hex 40), so
NOT 01000000 (hex 40)
= 10111111 (hex BF)
xxxxxxxx (any value)
AND 10111111 (hex BF)
= x0xxxxxx
To put all this into Java code, you've got the following binary operators:
| binary OR
& binary AND
~ binary NOT
So if you wanted to set a bit in a byte:
byte anyByte;
anyByte = anyByte | 0x40;
which can be shortened to
anyByte |= 0x40;
If you want to clear a bit:
anyByte &= ~0x40;
If you want to test whether a bit is set, you'd use the following:
if ((anyByte & 0x40) == 0x40) ...
If you want to test whether bit 4 and bit 1 was set, you'd do the following:
if ((anyByte & 0x12) == 0x12) ...
Why 0x12? Because hex 12 is binary 0001 0010 which "masks" bit 4 and 1.
Back to your question:
In order to send the correct command string, you just need to create the right byte array as specified in the manual, though I hope it is clearer now how to set bits in bytes:
Socket s = new Socket("192.168.1.2", 49137);
InputStream in = s.getInputStream());
// send the request
// (Note, for different requests, you'll need different, byte arrays and may
// even have different lengths (can't tell without having seen the whole manual)
byte[] st = new byte[] { 0x01, 0x30, 0x01, 0x00 };
OutputStream out = s.getOutputStream();
out.write(st);
out.flush();
// read the first header byte (bits indicate the rest of the header structure)
byte header = (byte)in.read();
// bit 1 shows whether the header represented in 2 bytes
boolean header2Byte = (header & 0x2) != 0;
// bit 2 shows whether the length is represented in 2 bytes
boolean len2Byte = (header & 0x4) != 0;
if (header2Byte) {
// need to read the extra header (discarded here)
in.read();
}
// missed off reading the command byte/s
int len = 0;
if (len2Byte) {
byte[] lenByte = new byte[2];
in.read(lenByte);
if (isLittleEndian) {
len = (lenByte[1] << 8) + lenByte[0];
} else {
len = (lenByte[0] << 8) + lenByte[1];
}
} else {
// only one byte signifies the length
len = is.read();
}
byte[] data = new byte[len];
in.read(data);
// it is unclear what format the data, has but if it is a string, and encoded as
// UTF-8, then you can use the following
String stringData = new String(data, "UTF-8");
System.out.println(stringData);
// note, try-catch-finally omitted for brevity
in.close();
out.close();
s.close();
Note, I'm not using a DataInputStream here as the way Java encodes integers may be different from how the device encodes it's integers. E.g. in Java and integer is 4 bytes and Big Endian (see also this SO article).
N.B. the << operator is the left shift operator, which shifts the bits along in a byte and the above case is used to combine two bytes into a 16-bit number. Left-shifting by 8 is equivalent to multiplying by 256.

In such diagram header block shows mostly like "Bit 7, Bit 6,..,Bit 0"
instead of "Bit 0, Bit 1, Bit 2, ... Bit 7", i always wondering
why?.
In typical number writing 0 is the least significant bit 7 the most significant and the byte is written in most significant to least significant 7-0.
byte 11111111
place 76543210
But when i apply in code what is following order for byte st[0] = Bit 7 or Bit 1?
In your code this is not setting bits this is setting a byte in the byte array
If you are nervous about setting bits try a class like this:
public class SimpleByteSetter {
/* b is is byte you are setting
on is if the bit is set to on or 1
place is the bit place in the range of 0-7
*/
public static byte set(final byte b, final boolean on, final int place) {
if (on) { return (byte) (b | ((1 << place) & 0xFF)); }
return (byte) (b & (~((1 << place) & 0xFF)));
}
// 1 == on everything else off (but only use 0!)
public static byte set(final byte b, final int on, final int place) {
return set(b, 1==on, place);
}
}
use it in you code like:
byte header = 0;
// get = 0, set = 1, place = 0
header = SimpleByteSetter(header, 0, 0 ); // get
// header 2 byte = 0, header 2 byte = 1, place = 1
header = SimpleByteSetter(header, 0, 1 ); // header 1 byte
...
st[0] = header;
Also according to this diagram, does it mean every command i send will have Header fixed always
Yes
This is the code i was trying by taking Bit 1 as st[0], Bit 2 as st1 instead of Bit 7 as st[0]. To apply Power off/on test.
I don't have enough information to actually build the packet but:
// the header is 1 byte I don't see how to make it two
// the command is 2 bytes per the table
// the parameter length is 0 so 1 byte (is there suppose to be a device id?)
byte[] st = new byte[ 1 + 2 + 1 ];
byte header = 0;
// get = 0, set = 1, place = 0
header = SimpleByteSetter(header, 0, 0 ); // get
// header 2 byte = 0, header 2 byte = 1, place = 1
header = SimpleByteSetter(header, 0, 1 ); // header 1 byte
// length 1 byte = 0, length 2 byte = 1
header = SimpleByteSetter(header, 0, 2 );
// command 1 byte = 0, command 2 byte = 1;
header = SimpleByteSetter(header, 1, 3 );
st[0] = header;
st[1] = 0x0130 & 0xFF; // poweroff command first byte
st[2] = 0x0100 & 0xFF; // poweroff second byte
st[3] = 0;

You don't need to care about bit order because IP packets contain bytes and the lower level components make sure every byte is transferred correctly.
Instead of System.out.println(st) I'd make a small method that prints the command buffer nicely in hexadeciaml.