In my current project I need to send over the network a parsed structure that contains some n-bit fields. for instance:
protocol version: 1 byte
messageId: 1 byte
creationTime: 6 bytes
traceId: 3 bits
reliability: 7 bits
Etc...
Thus, I created a simple POJO class to represent this for parsing & unparsing, but I have some doubts on what type to use for those fields, since this decision can make parsing & unparsing easy or a bit of a nightmare. I must say that the Message to be sent over the network has a very specific size constraint: it cannot surpass the sum of all fields.
I first tought in using bytes for everything and then have a Message.getBytes() method that will convert the message and for those fields that are less than a byte, use bitwise operations to discard the unnecessary bits.
Am I going in the right direction or there is another much simpler way to do this? I just feel I'm re-inventing the wheel here, this feels kinda boilerplate code...
Thanks!
EDIT: If anybody else stumbles here, I'll post how I solved this (thanks to a mate at work that helped me with these), so just keep reading:
Luckily enough, my protocol socket size rounds up to a fixed number of bytes (49) and those fields that are less than byte size, sum up a byte at then end, resulting in that I can combine both fields in a single byte prior parsing/unparsing.
That said, imagine I have two fields, say field1 and field2, first in 7 bits and the other just a bit. To combine those, I just do this trick:
byte resultingByte = short2Byte((short) ((field1 % 128) * 2 + (field2 ? 1 : 0)));
Note that both field1 and field2 are short type. I found this the most convenient way to work at the bit level. Thus, I mod first field, making sure I'm only getting 7 bits, move the bits to the left divining by 2 since only one bit is needed to be moved. Lastly I add the field 2 short, which can be 1 or 0. Then I have a short with the required values in the 8 Less Signigicant Bits.
I created commodity methods to convert from short2Byte, Long, and some others:
private byte [] to2Bytes(int in) {
ByteBuffer ret = ByteBuffer.allocate(2);
int val = in % 65536;
short s1 = (short) (val / 256);
short s0 = (short) (val % 256);
ret.put(short2Byte(s1));
ret.put(short2Byte(s0));
return ret.array();
}
private byte [] to4Bytes(long in) {
ByteBuffer ret = ByteBuffer.allocate(4);
long div = 4294967296L;
long val = in % div;
int rem = 0;
short s3 = (short) (val / 16777216L);
rem = (int) (val % 16777216L);
short s2 = (short) (rem / 65536);
rem = rem % 65536;
short s1 = (short) (rem / 256);
short s0 = (short) (rem % 256);
ret.put(short2Byte(s3));
ret.put(short2Byte(s2));
ret.put(short2Byte(s1));
ret.put(short2Byte(s0));
return ret.array();
}
private byte [] time2Bytes(Long time) {
ByteBuffer ret = ByteBuffer.allocate(6);
String hex = Long.toHexString(time).toUpperCase();
while (hex.length() < 12) {
hex = "0" + hex;
}
while (hex.length() > 12) {
hex = hex.substring(1);
}
try {
for (int i = 0; i < 6; i++) {
String strByte = "" + hex.charAt(i*2) + hex.charAt(i*2 + 1);
short b = Short.parseShort(strByte, 16);
if (b > 127) {
b -= 256;
}
ret.put((byte) b);
}
}
catch (NumberFormatException e) {
// Exception captured for correctness
e.printStackTrace();
}
return ret.array();
}
private long bytes2time(byte b5, byte b4, byte b3, byte b2, byte b1, byte b0) {
long l5, l4, l3, l2, l1, l0;
l5 = byte2short(b5) * 1099511627776L;
l4 = byte2short(b4) * 4294967296L;
l3 = byte2short(b3) * 16777216L;
l2 = byte2short(b2) * 65536L;
l1 = byte2short(b1) * 256L;
l0 = byte2short(b0) * 1L;
return l5 + l4 + l3 + l2 + l1 + l0;
}
private long bytes2long(byte b3, byte b2, byte b1, byte b0) {
long l3, l2, l1, l0;
l3 = byte2short(b3) * 16777216L;
l2 = byte2short(b2) * 65536L;
l1 = byte2short(b1) * 256L;
l0 = byte2short(b0) * 1L;
return l3 + l2 + l1 + l0;
}
private int bytes2int(byte b1, byte b0) {
return (int)byte2short(b1) * 256 + (int)byte2short(b0);
}
private short byte2short(byte b) {
if (b < 0) {
return (short) (b+256);
}
return (short)b;
}
private byte short2Byte(short s) {
if (s < 128) {
return (byte) s;
}
else {
return (byte) (s-256);
}
}
At the end I'm sending a byte array with 49 bytes. Unparsing is very similar process, obviously. There must be a proper way to do this, but well, it works...Hope this helps someone!
You can to use ByteBuffer and BitSet class to write and read messages, but it can still become a nightmare even worse then pure bit manipulation (and it will affect performance)
Related
Given an array filled with 4 bytes inside (R,G,B,A), I'm trying to translate this array full of 4 8bits numbers into its translation in 32bits. To be more clear, if I get an array such as:
byte[] tab = {1,2,3,4};
with translated in binary in 8bit :
1 = 0b00000001
2 = 0b00000010
3 = 0b00000011
4 = 0b00000100
Then, my method should return a byte array such as :
newTab = {00000001_00000010_00000011_00000100};
For some reason, I'm trying to do this without using a String to concatenate the bytes.
I've already tried something with binary operators such as <<, >> or |, but without success...
So far, my code looks like this :
byte[] tab = {1,2,3,4};
int tmp,tabToInt = 0;
for (int x = 0 ; x < tab.length ; ++x){
tmp = tmp << (tab.length - 1 - x)*8;
byteToInt = byteToInt | tmp;
}
return tabToInt;
But it didn't seem to work, even less with negatives bytes... (like -1 = 0b11111111)
Thanks in advance for your answers!
You can use ByteBuffer like this.
byte[] tab = {1, 2, 3, 4};
int tabToInt = ByteBuffer.wrap(tab).getInt();
System.out.println("decimal = " + tabToInt);
System.out.println("binary = " + Integer.toBinaryString(tabToInt));
System.out.println("hexadecimal =" + Integer.toHexString(tabToInt));
output
decimal = 16909060
binary = 1000000100000001100000100
hexadecimal =1020304
ByteBuffer can do it, but only if you get passed at least 4 bytes.
The problem with your code is two-fold:
I think you typoed somewhere, your code doesn't even compile. I think you meant tmp = tab[x] << (tab.length - 1 - x)*8;. Your snippet never does anything with tab other than ask for its length.
Negative numbers extend, and java will convert any byte or short to an int the moment you do any math to it. So, 0b1111 1111, if you try to do e.g. << 8 on that, java first turns that -1 byte into a -1 int (so that's now 32 1 bits), and then dutifully left shifts it by 8, so now that's 24 1 bits, followed by 8 0 bits. You then bitwise OR that into your target, and thus now the target is mostly 1 bits. To convert a byte to an int without "sign extension", (b & 0xFF does it:
byte b = (byte) 0b1111_1111;
assert b == -1; // yup, it is
int c = b; // legal
assert c == -1; // yeah, still is. uhoh. That's...
int d = 0b11111111_11111111_11111111_11111111;
assert c == d; // yeah. We don't want that.
int e = (b & 0xFF);
assert e = 255;
int f = 0b0000000_0000000_0000000_11111111;
assert e == f; // yes!
So, my problem set is very simple. I am working with a set of three integers randomly selected from [0-65535] and my Job is to encode this integers into one unique number. Here is what I have tried so far
I have written a java function called pack to try and encode this numbers as follows
private long pack(long a, long b, long c) {
int N = 65535, M = 65536;
return (a + (b * N) + c * N * M);
}
And I have also written another java function to unpack or decode the packed number back to the original integers as follows
private long[] unpack(long packed) {
int N = 65535, M = 65536;
long a = (packed % N);
long b = (packed / N) % M;
long c = (packed % (N * M));
return new long[]{a, b, c};
}
Now when I ran the code above in my main function using sample data {67, 8192, 7168} I am getting the following as result in my console output
Packing 67, 8192, 7168
Result=30786392678467
UnPacking 30786392678467
Result=[67, 8192, 57411]
From the above, clearly my first and second values are always correct but the last value always appear to be wrong. What am I possibly missing out.Your help is greatly appreciated. Thanks alot.
I'm going to give you an alternative solution now, and then I can try to debug your current solution when I'm on a PC instead of a phone (rgettman beat me!).
Because each of the three numbers can be a maximum of 65535, that means that each number will fit into 16 bits. For that reason, you can simply build a unique long with the following:
long encoded = (a << 32L) | (b << 16) | c;
And decoding it would look like the following:
long a = (encoded >> 32) & 0xFFFFL;
long b = (encoded >> 16) & 0xFFFFL;
long c = encoded & 0xFFFFL;
Your packing and unpacking code is incorrect according to the range [0, 65535] you've given.
There are 65,536 possible numbers, and you don't want the encoding of one integer to change the encoding of another integer. You should use one constant set to 65536 (which is 216).
public static final long PACK = 65536;
Then your pack method changes slightly to:
private long pack(long a, long b, long c) {
return (a + (b * PACK) + c * PACK * PACK);
}
This "packs" a into the least significant 16 bits of the long (bits 49-64), b into bits 33-48, and c into bits 17-32. (Nothing is packed into bits 0-16, so those bits remain cleared.)
Also, your unpack method changes to:
private static long[] unpack(long packed) {
long a = (packed % PACK);
long b = (packed / PACK) % PACK;
long c = (packed / (PACK * PACK)); // Use / not %.
return new long[]{a, b, c};
}
Notice that c's operation divides by PACK squared, not using the % operator, but using /. Otherwise both M and N have each been replaced by PACK.
Output with these changes:
Packing 67, 8192, 168
Result=722091376707
UnPacking 722091376707
Result=[67, 8192, 168]
actually, your solution is almost correct: just make sure that M == N == 65536 and fix the problem in unpacking variable c.
private long pack(long a, long b, long c) {
long N = 65536;
return (a + (b * N) + c * N * N);
}
private long[] unpack(long packed) {
long N = 65536;
long a = (packed % N);
long b = (packed / N) % N;
long c = (packed / (N * N));
return new long[]{a, b, c};
}
Also, I changed the type of N to long although it would not matter as Java will convert it to long during multiplication anyway.
How do I convert long to 4 bytes? I am receiving some output from a C program and it uses unsigned long. I need to read this output and convert this to 4 bytes.
However, java uses signed long which is 64 bits. Is there any way to do this conversion?
To read 4 bytes as an unsigned 32-bit value, assuming it is little endian, the simplest thing to do is to use ByteBuffer
byte[] bytes = { 1,2,3,4 };
long l = ByteBuffer.wrap(bytes)
.order(ByteOrder.LITTLE_ENDIAN).getInt() & 0xFFFFFFFFL;
While l can be an signed 64-bit value it will only be between 0 and 2^^32-1 which is the range of a unsigned 32-bit value.
You can use the java.nio.ByteBuffer. It can parse the long, and it does the byte ordering for you.
You can code a loop where you divide the "long" by 256, take the rest, then you have the "Least Significant Byte" ...
(depending on whether you want little-endian or big-endian you can loop forwards or backwards)
long l = (3* 256 * 256 * 256 + 1 * 256 *256 + 4 * 256 + 8);
private byte[] convertLongToByteArray(long l) {
byte[] b = new byte[4];
if(java.nio.ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN){
for (int i=0; i<4; i++) {
b[i] = (byte)(l % 256) ;
l = l / 256;
}
}else{
for (int i=3; i>=0; i--) {
b[i] = (byte)(l % 256) ;
l = l / 256;
}
}
return b;
}
I try to calculate the checksum of a Sega Genesis rom file in Java. For this i want to port a code snipped from C into Java:
static uint16 getchecksum(uint8 *rom, int length)
{
int i;
uint16 checksum = 0;
for (i = 0; i < length; i += 2)
{
checksum += ((rom[i] << 8) + rom[i + 1]);
}
return checksum;
}
I understand what the code does. It sums all 16bit numbers (combined from two 8 bit ones). But what i didn't understand is what's happening with the overflow of the uint16 and how this transfers to Java code?
Edit:
This code seems to work, thanks:
int calculatedChecksum = 0;
int bufferi1=0;
int bufferi2=0;
bs = new BufferedInputStream(new FileInputStream(this.file));
bufferi1 = bs.read();
bufferi2 = bs.read();
while(bufferi1 != -1 && bufferi2 != -1){
calculatedChecksum += (bufferi1*256 + bufferi2);
calculatedChecksum = calculatedChecksum % 0x10000;
bufferi1 = bs.read();
bufferi2 = bs.read();
}
Simply put, the overflow is lost.
A more correct approach (imho) is to use uint32 for summation, and then you have the sum in the lower 16 bits, and the overflow in the upper 16 bits.
static int checksum(final InputStream in) throws IOException {
short v = 0;
int c;
while ((c = in.read()) >= 0) {
v += (c << 8) | in.read();
}
return v & 0xffff;
}
This should work equivalently; by using & 0xffff, we get to treat the value in v as if it were unsigned the entire time, since arithmetic overflow is identical w.r.t. bits.
You want addition modulo 216, which you can simply spell out manually:
checksum = (checksum + ((rom[i] << 8) + rom[i + 1])) % 0x10000;
// ^^^^^^^^^
I am trying to convert a HEX-sequence to a String encoded in either, ISO-8859-1, UTF-8 or UTF-16BE. That is, I have a String looking like: "0422043504410442" this represents the characters: "Test" in UTF-16BE.
The code I used to convert between the two formats was:
private static String hex2String(String hex, String encoding) throws UnsupportedEncodingException {
char[] hexArray = hex.toCharArray();
int length = hex.length() / 2;
byte[] rawData = new byte[length];
for(int i=0; i<length; i++){
int high = Character.digit(hexArray[i*2], 16);
int low = Character.digit(hexArray[i*2+1], 16);
int value = (high << 4) | low;
if( value > 127)
value -= 256;
rawData[i] = (byte) value;
}
return new String(rawData, encoding);
}
This seems to work fine for me, but I still have two questions regarding this:
Is there any simpler way (preferably without bit-handling) to do this conversion?
How am I to interpret the line: int value = (high << 4) | low;?
I am familiar with the basics of bit-handling, though not at all with the Java syntax. I believe the first part shift all bits to the left by 4 steps. Though the rest I don't understand and why it would be helpful in this certain situation.
I apologize for any confusion in my question, please let me know if I should clarify anything.
Thank you.
//Abeansits
Is there any simpler way (preferably without bit-handling) to do this conversion?
None I would know of - the only simplification seems to parse the whole byte at once rather than parsing digit by digit (e.g. using int value = Integer.parseInt(hex.substring(i * 2, i * 2 + 2), 16);)
public static byte[] hexToBytes(final String hex) {
final byte[] bytes = new byte[hex.length() / 2];
for (int i = 0; i < bytes.length; i++) {
bytes[i] = (byte) Integer.parseInt(hex.substring(i * 2, i * 2 + 2), 16);
}
return bytes;
}
How am I to interpret the line: int value = (high << 4) | low;?
look at this example for your last two digits (42):
int high = 4; // binary 0100
int low = 2; // binary 0010
int value = (high << 4) | low;
int value = (0100 << 4) | 0010; // shift 4 to left
int value = 01000000 | 0010; // bitwise or
int value = 01000010;
int value = 66; // 01000010 == 0x42 == 66
You can replace the << and | in this case with * and +, but I don't recommend it.
The expression
int value = (high << 4) | low;
is equivalent to
int value = high * 16 + low;
The subtraction of 256 to get a value between -128 and 127 is unnecessary. Simply casting, for example, 128 to a byte will produce the correct result. The lowest 8 bits of the int 128 have the same pattern as the byte -128: 0x80.
I'd write it simply as:
rawData[i] = (byte) ((high << 4) | low);
Is there any simpler way (preferably
without bit-handling) to do this
conversion?
You can use the Hex class in Apache commons, but internally, it will do the same thing, perhaps with minor differences.
How am I to interpret the line: int value = (high << 4) | low;?
This combines two hex digits, each of which represents 4 bits, into one unsigned 8-bit value stored as an int. The next two lines convert this to a signed Java byte.