Develop Reference

Getting signed short values when I wanted to write out unsigned short values

Using the solution provided here -Java input and output binary file byte size are not matching after transformation I wrote out a 2-dimensional array to disk that consists of unsigned shorts. When I checked the values of this file by reading them in I still see signed values i.e. negative values. What is the error in the code ?
fos = new FileOutputStream(destFile);
short[][] data = getData(); //defined in another class
if (ByteOrder.nativeOrder().equals(ByteOrder.BIG_ENDIAN)
{
for (int i = 0;i<1200;i++)
{
for (int j = 0; j <1200;j++)
{
fos.write((data[i][j] >> 8) & 0xFF);
fos.write(data[i][j] & 0xFF);
}
}
}
else
{
for (int i = 0;i<1200;i++)
{
for (int j = 0; j <1200;j++)
{
fos.write(data[i][j] & 0xFF);
fos.write((data[i][j] >> 8) & 0xFF);
}
}
}

Java doesn't have an unsigned short datatype, but don't worry, the code works fine. The bit patterns for signed and unsigned are the same, the difference is in how the first bit is interpreted. If you ever want the unsigned value for a short you can use the AND operator:
int unsignedValue = signedShortValue & 0xFFFF;

java how to convert a byte-value in a byte-array in an int/long value? [duplicate]

How do I convert a long to a byte[] and back in Java?
I'm trying convert a long to a byte[] so that I will be able to send the byte[] over a TCP connection. On the other side I want to take that byte[] and convert it back into a double.

public byte[] longToBytes(long x) {
ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
buffer.putLong(x);
return buffer.array();
}
public long bytesToLong(byte[] bytes) {
ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
buffer.put(bytes);
buffer.flip();//need flip
return buffer.getLong();
}
Or wrapped in a class to avoid repeatedly creating ByteBuffers:
public class ByteUtils {
private static ByteBuffer buffer = ByteBuffer.allocate(Long.BYTES);
public static byte[] longToBytes(long x) {
buffer.putLong(0, x);
return buffer.array();
}
public static long bytesToLong(byte[] bytes) {
buffer.put(bytes, 0, bytes.length);
buffer.flip();//need flip
return buffer.getLong();
}
}
Since this is getting so popular, I just want to mention that I think you're better off using a library like Guava in the vast majority of cases. And if you have some strange opposition to libraries, you should probably consider this answer first for native java solutions. I think the main thing my answer really has going for it is that you don't have to worry about the endian-ness of the system yourself.

You could use the Byte conversion methods from Google Guava.
Example:
byte[] bytes = Longs.toByteArray(12345L);

I tested the ByteBuffer method against plain bitwise operations but the latter is significantly faster.
public static byte[] longToBytes(long l) {
byte[] result = new byte[8];
for (int i = 7; i >= 0; i--) {
result[i] = (byte)(l & 0xFF);
l >>= 8;
}
return result;
}
public static long bytesToLong(final byte[] b) {
long result = 0;
for (int i = 0; i < 8; i++) {
result <<= 8;
result |= (b[i] & 0xFF);
}
return result;
}
For Java 8+ we can use the static variables that were added:
public static byte[] longToBytes(long l) {
byte[] result = new byte[Long.BYTES];
for (int i = Long.BYTES - 1; i >= 0; i--) {
result[i] = (byte)(l & 0xFF);
l >>= Byte.SIZE;
}
return result;
}
public static long bytesToLong(final byte[] b) {
long result = 0;
for (int i = 0; i < Long.BYTES; i++) {
result <<= Byte.SIZE;
result |= (b[i] & 0xFF);
}
return result;
}

If you are looking for a fast unrolled version, this should do the trick, assuming a byte array called "b" with a length of 8:
byte[] -> long
long l = ((long) b[7] << 56)
| ((long) b[6] & 0xff) << 48
| ((long) b[5] & 0xff) << 40
| ((long) b[4] & 0xff) << 32
| ((long) b[3] & 0xff) << 24
| ((long) b[2] & 0xff) << 16
| ((long) b[1] & 0xff) << 8
| ((long) b[0] & 0xff);
long -> byte[] as an exact counterpart to the above
byte[] b = new byte[] {
(byte) lng,
(byte) (lng >> 8),
(byte) (lng >> 16),
(byte) (lng >> 24),
(byte) (lng >> 32),
(byte) (lng >> 40),
(byte) (lng >> 48),
(byte) (lng >> 56)};

Why do you need the byte[]? why not just write it to the socket?
I assume you mean long rather than Long, the latter needs to allow for null values.
DataOutputStream dos = new DataOutputStream(
new BufferedOutputStream(socket.getOutputStream()));
dos.writeLong(longValue);
DataInputStream dis = new DataInputStream(
new BufferedInputStream(socket.getInputStream()));
long longValue = dis.readLong();

I find this method to be most friendly.
var b = BigInteger.valueOf(x).toByteArray();
var l = new BigInteger(b);

Just write the long to a DataOutputStream with an underlying ByteArrayOutputStream. From the ByteArrayOutputStream you can get the byte-array via toByteArray():
class Main
{
public static byte[] long2byte(long l) throws IOException
{
ByteArrayOutputStream baos=new ByteArrayOutputStream(Long.SIZE/8);
DataOutputStream dos=new DataOutputStream(baos);
dos.writeLong(l);
byte[] result=baos.toByteArray();
dos.close();
return result;
}
public static long byte2long(byte[] b) throws IOException
{
ByteArrayInputStream baos=new ByteArrayInputStream(b);
DataInputStream dos=new DataInputStream(baos);
long result=dos.readLong();
dos.close();
return result;
}
public static void main (String[] args) throws java.lang.Exception
{
long l=123456L;
byte[] b=long2byte(l);
System.out.println(l+": "+byte2long(b));
}
}
Works for other primitives accordingly.
Hint: For TCP you do not need the byte[] manually. You will use a Socket socket and its streams
OutputStream os=socket.getOutputStream();
DataOutputStream dos=new DataOutputStream(os);
dos.writeLong(l);
//etc ..
instead.

All of the current answers are more complicated than they need to be and I’d hate for anyone locating this thread to walk away without a more concise option.
You can do both of these conversions in a single line.
byte[] to long:
ByteBuffer.wrap(yourBytes).getLong();
long to byte[]:
ByteBuffer.wrap(new byte[8]).putLong(yourLong).array();

You could use the implementation in org.apache.hadoop.hbase.util.Bytes http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/util/Bytes.html
The source code is here:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hbase/hbase/0.89.20100924-28/org/apache/hadoop/hbase/util/Bytes.java#Bytes.toBytes%28long%29
Look for the toLong and toBytes methods.
I believe the software license allows you to take parts of the code and use it but please verify that.

public static long bytesToLong(byte[] bytes) {
if (bytes.length > 8) {
throw new IllegalMethodParameterException("byte should not be more than 8 bytes");
}
long r = 0;
for (int i = 0; i < bytes.length; i++) {
r = r << 8;
r += bytes[i];
}
return r;
}
public static byte[] longToBytes(long l) {
ArrayList<Byte> bytes = new ArrayList<Byte>();
while (l != 0) {
bytes.add((byte) (l % (0xff + 1)));
l = l >> 8;
}
byte[] bytesp = new byte[bytes.size()];
for (int i = bytes.size() - 1, j = 0; i >= 0; i--, j++) {
bytesp[j] = bytes.get(i);
}
return bytesp;
}

I will add another answer which is the fastest one possible ׂ(yes, even more than the accepted answer), BUT it will not work for every single case. HOWEVER, it WILL work for every conceivable scenario:
You can simply use String as intermediate. Note, this WILL give you the correct result even though it seems like using String might yield the wrong results AS LONG AS YOU KNOW YOU'RE WORKING WITH "NORMAL" STRINGS. This is a method to increase effectiveness and make the code simpler which in return must use some assumptions on the data strings it operates on.
Con of using this method: If you're working with some ASCII characters like these symbols in the beginning of the ASCII table, the following lines might fail, but let's face it - you probably will never use them anyway.
Pro of using this method: Remember that most people usually work with some normal strings without any unusual characters and then the method is the simplest and fastest way to go.
from Long to byte[]:
byte[] arr = String.valueOf(longVar).getBytes();
from byte[] to Long:
long longVar = Long.valueOf(new String(byteArr)).longValue();

Kotlin extensions for Long and ByteArray types:
fun Long.toByteArray() = numberToByteArray(Long.SIZE_BYTES) { putLong(this#toByteArray) }
private inline fun numberToByteArray(size: Int, bufferFun: ByteBuffer.() -> ByteBuffer): ByteArray =
ByteBuffer.allocate(size).bufferFun().array()
#Throws(NumberFormatException::class)
fun ByteArray.toLong(): Long = toNumeric(Long.SIZE_BYTES) { long }
#Throws(NumberFormatException::class)
private inline fun <reified T: Number> ByteArray.toNumeric(size: Int, bufferFun: ByteBuffer.() -> T): T {
if (this.size != size) throw NumberFormatException("${T::class.java.simpleName} value must contains $size bytes")
return ByteBuffer.wrap(this).bufferFun()
}
You can see full code in my library https://github.com/ArtemBotnev/low-level-extensions

If you are already using an OutputStream to write to the socket, then DataOutputStream might be a good fit. Here is an example:
// Assumes you are currently working with a SocketOutputStream.
SocketOutputStream outputStream = ...
long longValue = ...
DataOutputStream dataOutputStream = new DataOutputStream(outputStream);
dataOutputStream.writeLong(longValue);
dataOutputStream.flush();
There are similar methods for short, int, float, etc. You can then use DataInputStream on the receiving side.

Here's another way to convert byte[] to long using Java 8 or newer:
private static int bytesToInt(final byte[] bytes, final int offset) {
assert offset + Integer.BYTES <= bytes.length;
return (bytes[offset + Integer.BYTES - 1] & 0xFF) |
(bytes[offset + Integer.BYTES - 2] & 0xFF) << Byte.SIZE |
(bytes[offset + Integer.BYTES - 3] & 0xFF) << Byte.SIZE * 2 |
(bytes[offset + Integer.BYTES - 4] & 0xFF) << Byte.SIZE * 3;
}
private static long bytesToLong(final byte[] bytes, final int offset) {
return toUnsignedLong(bytesToInt(bytes, offset)) << Integer.SIZE |
toUnsignedLong(bytesToInt(bytes, offset + Integer.BYTES));
}
Converting a long can be expressed as the high- and low-order bits of two integer values subject to a bitwise-OR. Note that the toUnsignedLong is from the Integer class and the first call to toUnsignedLong may be superfluous.
The opposite conversion can be unrolled as well, as others have mentioned.

From Java 9, the best approach is to use VarHandle, which will read from the byte array as if it is a long array, for performance make the VarHandle instance a static final field.
static final VarHandle HANDLE = MethodHandles.byteArrayViewVarHandle(Long.TYPE.arrayType(), ByteOrder.nativeOrder());
static long bytesToLong(byte[] bytes, int offset) {
return (long)HANDLE.get(bytes, offset);
}
static void longToBytes(byte[] bytes, int offset, long value) {
HANDLE.set(bytes, offset, value);
}

new ObjectMapper().writeValueAsString(1234L).getBytes(); //*
Clearly this is not the ideal solution in terms of bytes, but it's what everybody seems to do these days.
Given op wants to convert it back to a double at the other end precision is probably not a concern.
(* if its not clear, this says: send it over TCP as JSON)

public static long ToInt64(byte[] buffer) throws OutOfRangeException {
if (buffer.length < 8) {
throw new OutOfRangeException();
}
long int64 = 0;
int64 = buffer[0] & 0xffL;
int64 |= ((long) buffer[1] << 8) & 0xff00L;
int64 |= ((long) buffer[2] << 16) & 0xff0000L;
int64 |= ((long) buffer[3] << 24) & 0xff000000L;
int64 |= ((long) buffer[4] << 32) & 0xff00000000L;
int64 |= ((long) buffer[5] << 40) & 0xff0000000000L;
int64 |= ((long) buffer[6] << 48) & 0xff000000000000L;
int64 |= ((long) buffer[7] << 56);
return int64;
}

static byte[] longToBytes(Long l) {
return (l + "").getBytes(StandardCharsets.UTF_8);
}

Java 16bit CRC packet validation

I have an Avl Packet that I'm recieving through GPRS from a Device.
The protocol manual says the packet has a 16bit CRC on the last 4 bytes and a source code for CRC calculation is given:
public static int getCrc16(Byte[] buffer) {
return getCrc16(buffer, 0, buffer.length, 0xA001, 0);
}
public synchronized static int getCrc16(Byte[] buffer, int offset, int bufLen, int polynom, int preset) {
preset &= 0xFFFF;
polynom &= 0xFFFF;
int crc = preset;
for (int i = 0; i < bufLen; i++) {
int data = buffer[i + offset] & 0xFF;
crc ^= data;
for (int j = 0; j < 8; j++) {
if ((crc & 0x0001) != 0) {
crc = (crc >> 1) ^ polynom;
} else {
crc = crc >> 1;
}
}
}
return crc & 0xFFFF;
}
So I get the CRC number the packet sends me,then I call getCrc16 for the Byte array in which I have stored the packet and then compare the two numbers right??
Here is the code I use inside my program:
public static String toBinaryString(byte n) {
StringBuilder sb = new StringBuilder("00000000");
for (int bit = 0; bit < 8; bit++) {
if (((n >> bit) & 1) > 0) {
sb.setCharAt(7 - bit, '1');
}
}
return sb.toString();
}
int CalculatedCRC = getCrc16(AvlPacket);
System.out.println("Calculated CRC= " + CalculatedCRC);
int index = (AvlPacket.length)-4;
String BinaryRecievedCRC = "";
for (int j = 0; j < 4; j++) {
BinaryRecievedCRC+= toBinaryString(AvlPacket[index]);
index+=1;
}
int RecievedCRC = Integer.parseInt(BinaryRecievedCRC, 2);
System.out.println("Recieved CRC= " + RecievedCRC);
toBinaryString() converts a byte to it's binary from and puts it into a string!
So I calculate the CRC through getCrc16() given to me from the manual.Then take an index 4 bytes before the end of the packet so I can read the last 4 bytes and get the CRC sent with the packet!
The for loop takes each of the last bytes and with toBinaryString() combines all them in binary form and into a String!So I got something like 0000000000000000101011011101001 (The manual states that first two bytes are always zeroes because its a 16bit CRC)
So I just parse the Binary String into a signed int and Compare the two CRCs...!
Yet I get Results like :
Calculated CRC= 21395
-----Recieved CRC= 30416
or
Calculated CRC= 56084
-----Recieved CRC= 10504
I've tested with many packets and they can't all have loss of data..And I'm parsing the data too so I know that the data I get is correct!
What am I missing in all this??

There is probably something wrong with the documentation wording (or your understanding of it). If there is a 16-Bit CRC in a packet, its most likely occupying two bytes, not four (in binary form). If it were a decimal, even four bytes wouldn't suffice (you would need 5 didgts to store it as unsigned decimal string).
Your code shows you do conversions (but I can't see what kind of conversion its supposed to do):
BinaryRecievedCRC+= toBinaryString(AvlPacket[index]);
I would expect the CRC to be store in the data in binary form, so I assume the only thing you need to figure out are which endianess is used and where the CRC is stored in the data.
Edit: Judging from your comment you would need to extract the CRC like this:
public int getCRC(byte[] data, int index) {
return ((data[index] & 0xFF) << 8)) | (data[index + 1] & 0xFF);
}

So I get the CRC number the packet sends me,then I call getCrc16 for
the Byte array in which I have stored the packet and then compare the
two numbers right??
Wrong. You calculate the CRC over the entire message, including the CRC bytes, and the result should be zero.

Problem Solved!
The problem was that the Packet had 8 other bytes before getting into the Data part!
So I had to exclude those first 8 bytes along with the last 4 bytes of the sent CRC before calculating the CRC!
Now the numbers agree and the above code is correct with the exception that the for loop in the getCrc16 function starts from i=8 (so as to skip the first 8 bytes of the packet which do no belong to the Data part!)
Thank you all for your time!

Transforming byte values so that their MSB is removed / restored

I have a binary protocol which extracts the MSB of each payload byte into a MSB collection byte (septett) for transmission, and re-injects the MSBs on receiver side. The payload consists of n four byte frames, depending on sender (six in my case).
Those are two example frames, with their septett (last byte), as seen on the wire:
0x2E 0x00 0x5F 0x00 0x04
0x79 0x01 0x38 0x22 0x04
Those are the same frames, client side, with the MSBs re-injected:
0x2E 0x00 0xDF 0x00
0x79 0x01 0xB8 0x22
The C functions that do the transformation are defined on pages 9 and 10 in this document. My version of these, in Java, is below. The problem I have is that none of this works, and I'm confused as to why. I pass my four bytes from the wire, and get same bytes out, untouched. I could use some help figuring out what's wrong here (probably something trivial that I fail to see).
private static byte[] vbusExtractSeptett(byte[] data, int offset, int length) {
byte septett = 0;
for (int i = 0; i < length; i++) {
if ((data[offset + i] & 0x80) != 0) {
data[offset + i] &= 0x7F;
septett |= 1 << i;
}
}
data[offset + length] = septett;
return data;
}
private static byte[] vbusInjectSeptett(final byte[] data, int offset, int length) {
byte septett = data[offset + length];
for (int i = 0; i < length; i++) {
if ((septett & (1 << i)) != 0)
data[offset + i] |= 0x80;
}
return data;
}

In Java, a byte is signed. Without reading through all your code, I bet that is the problem.
The C code in your document uses unsigned char math.
Since Java doesn't have "unsigned", you probably need to do all your math in shorts (or ints) and then convert back to bytes. Be sure to mask off the sign bit, e.g. something like
byte theResult = theIntIDidtheMathOn & 0xFF;
data[index] = theResult;

The other answer is correct, you are not taking into account Java's use of signed bytes.
There are a few possibilities for solving this:
You can do the stuff above with "& 0xFF"
You can just treat everything as ints (as I have done below)
You could use a library to help. Some examples are JOOU (which was started in response to this lack of unsigned types) or netty channelbuffers (NB: although netty is focussed on network IO, like sockets etc., its channel buffer class is great for dealing with bytestreams and signed/unsigned types of different lengths and I have used it quite a lot for transforming binary protocols like the one you're handling.)
I have written a "solution" to your question below.
public class SOAnswer
{
private static void vbusExtractSeptett(int[] data, int offset, int length) {
int septett = 0;
for (int i = 0; i < length; i++) {
if ((data[offset + i] & 0x80) != 0) {
data[offset + i] &= 0x7F;
septett |= 1 << i;
}
}
data[offset + length] = septett;
}
private static void vbusInjectSeptett(final int[] data, int offset, int length) {
int septett = data[offset + length];
for (int i = 0; i < length; i++) {
if ((septett & (1 << i)) != 0)
data[offset + i] |= 0x80;
}
// clear the septett byte
data[offset + length] = 0x00;
}
private static void printIntArrayAsHEX(int[] array)
{
StringBuilder builder = new StringBuilder();
for ( int a : array )
{
String s = Integer.toHexString( a );
if (s.length() == 1)
builder.append( "0" );
builder.append(s + ":");
}
builder.substring( 0, builder.lastIndexOf( ":" ) - 1 );
System.out.println(builder.toString());
}
public static void main( String[] args )
{
// Create an array long enough for the extracting/injecting
int[] arr = new int[]{0x2E, 0x00, 0xDF, 0x00, 0x00};
// see what it looks like
printIntArrayAsHEX(arr);
// perform extraction
vbusExtractSeptett( arr, 0, 4 );
// see what it looks like
printIntArrayAsHEX(arr);
// Perform injection
vbusInjectSeptett( arr, 0, 4 );
// see what it looks like
printIntArrayAsHEX(arr);
}
}
One recommendation I would have for you is to think about if you really want to to re-implement the C code verbatim (especially the very functional programming style of passing an array of primitive types, an offset into the array and the length of the array). Maybe something more like this would be more OO:
private static int[] vbusExtractSeptettJAVASTYLE(int[] data) {
int[] extractedData = Arrays.copyOf( data, data.length +1 );
int septett = 0;
for (int i = 0; i < data.length; i++) {
if ((data[i] & 0x80) != 0) {
extractedData[i] = data[i] &= 0x7F;
septett |= 1 << i;
}
}
extractedData[extractedData.length-1] = septett;
return extractedData;
}

Java byte arrays and copying ints into them

What's the best way to put an int at a certain point in a byte[] array?
Say you have a byte array:
byte[] bytes = {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
int someInt = 12355; //0x43, 0x30
How can I do like bytes[4] = someInt; so that now bytes[4] will equal 0x43 and bytes[5] will be equal to 0x30?
I'm used to just using memcpy with C++ and don't know the alternatives in Java.
Thanks

If you want also high 0-bytes of the int put into the byte[]:
void place(int num, byte[] store, int where){
for(int i = 0; i < 4; ++i){
store[where+i] = (byte)(num & 0xFF);
num >>= 8;
}
}
If you only want the bytes to the highest nonzero byte:
void place(int num, byte[] store, int where){
while(num != 0){
store[where++] = (byte)(num & 0xFF);
num >>>= 8;
}
}
If you want the bytes big-endian (highest byte at lowest index), the version storing all four bytes is very easy, the other one slightly more difficult:
void placeBigEndian(int num , byte[] store, int where){
for(int i = 3; i >= 0; --i){
store[where+i] = (byte)(num & 0xFF);
num >>= 8;
}
}
void placeBigEndian(int num, byte[] store, int where){
in mask = 0xFF000000, shift = 24;
while((mask & num) == 0){
mask >>>= 8;
shift -= 8;
}
while(shift > 0){
store[where++] = (byte)((num & mask) >>> shift);
mask >>>= 8;
shift -= 8;
}
}

Note, you assume a big endian ordering! x86 is little endian... What's more, your int is 32bits long, hence 0x00004330 in big endian.
If this is what you want, use a ByteBuffer (which uses big endian ordering by default):
ByteBuffer buf = ByteBuffer.allocate(8);
// then use buf.putInt(yourint, index)
// buf.get(index) will read byte at index index, starting from 0

I don't see the problem, it looks like you solved it your own way:
public static void putShort(bytes[] array, int position, short value)
{
byte leftByte = (byte) (value >>> 8);
byte rightByte = (byte) (value & 0xFF);
array[position] = leftByte;
array[position + 1] = rightByte;
}
Note that an int is 4 bytes and a short is 2 bytes.

First of all, in Java you don't need to initialize byte arrays to zeroes. All arrays are initialized on construction time to 0/false/null.
Second, ints are signed 32-bit big-endian integers, so 12355 is actually 0x00003043. If you want to use 16-bit integers, use the short type.
Then, to get the individual bytes in your integer, you could do:
bytes[ i ] = (byte) (someInt >> 24);
bytes[ i+1 ] = (byte) (someInt >> 16);
bytes[ i+2 ] = (byte) (someInt >> 8);
bytes[ i+3 ] = (byte) (someInt);
The conversion to byte truncates the remaining bits, so no & 0xFF mask is needed. I'm assuming i is the index of the array. To change the endianness, swap the offsets at the indices.

One approach would be to use a DataOutputStream and it's writeInt() method, wrapped around a ByteArrayOutputStream. e.g. (with no error-handling)
public byte[] writeIntAtPositionX(int position, int iVal) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(baos);
// now, advancing to a specific spot is awkward.
// presumably you are actually writing other stuff out before the integer
// but if you really want to advance to a specific position
for (int i = 0; i < position; i++)
dos.writeByte(0);
dos.writeInt(iVal);
dos.flush();
dos.close();
return baos.toByteArray();
}
The big advantage of this method is that the guys who wrote Java figured out the byte ordering and the masking with 0xFF and all that stuff. Plus, if you ever envision writing doubles, shorts, longs or Strings etc. to your buffer you won't need to add all those methods, the work is already done.