Develop Reference

CRC16 CCITT Java implementation

There is a function written in C that calculates CRC16 CCITT. It helps reading data from RFID reader - and basically works fine. I would like to write a function in Java that would do similar thing.
I tried online converter page to do this, but the code I got is garbage.
Could you please take a look at this and advice why Java code that should do the same generates different crc?
Please find attached original C function:
void CRC16(unsigned char * Data, unsigned short * CRC, unsigned char Bytes)
{
int i, byte;
unsigned short C;
*CRC = 0;
for (byte = 1; byte <= Bytes; byte++, Data++)
{
C = ((*CRC >> 8) ^ *Data) << 8;
for (i = 0; i < 8; i++)
{
if (C & 0x8000)
C = (C << 1) ^ 0x1021;
else
C = C << 1;
}
*CRC = C ^ (*CRC << 8);
}
}
And here is the different CRC function written in JAVA that should calculate the same checksum, but it does not:
public static int CRC16_CCITT_Test(byte[] buffer) {
int wCRCin = 0x0000;
int wCPoly = 0x1021;
for (byte b : buffer) {
for (int i = 0; i < 8; i++) {
boolean bit = ((b >> (7 - i) & 1) == 1);
boolean c15 = ((wCRCin >> 15 & 1) == 1);
wCRCin <<= 1;
if (c15 ^ bit)
wCRCin ^= wCPoly;
}
}
wCRCin &= 0xffff;
return wCRCin;
}
When I try calculating 0,2,3 numbers in both functions I got different results:
for C function it is (DEC): 22017
for JAVA function it is (DEC): 28888
OK. I have converter C into Java code and got it partially working.
public static int CRC16_Test(byte[] data, byte bytes) {
int dataIndex = 0;
short c;
short [] crc= {0};
crc[0] = (short)0;
for(int j = 1; j <= Byte.toUnsignedInt(bytes); j++, dataIndex++) {
c = (short)((Short.toUnsignedInt(crc[0]) >> 8 ^ Byte.toUnsignedInt(data[dataIndex])) << 8);
for(int i = 0; i < 8; i++) {
if((Short.toUnsignedInt(c) & 0x8000) != 0) {
c = (short)(Short.toUnsignedInt(c) << 1 ^ 0x1021);
} else {
c = (short)(Short.toUnsignedInt(c) << 1);
}
}
crc[0] = (short)(Short.toUnsignedInt(c) ^ Short.toUnsignedInt(crc[0]) << 8);
}
return crc[0];
}
It gives the same CRC values as C code for 0,2,3 numbers, but i.e. for numbers 255, 216, 228 C code crc is 60999 while JAVA crc is -4537.
OK. Finally thanks to your pointers I got this working.
The last change required was changing 'return crc[0]' to:
return (int) crc[0] & 0xffff;
... and it works...
Many thanks to all :)

There is nothing wrong. For a 16 bit value, –4537 is represented as the exact same 16 bits as 60999 is. If you would like for your routine to return the positive version, convert to int (which is 32 bits) and do an & 0xffff.

RC4 encryption PRGA method throwing odd exceptions

I'm currently implementing a chat program that lets the user choose between RC4 and TEA encryption. It's a partner assignment, and I have taken RC4. I've used the wikipedia page mostly, as well as our book (which the code I believe is the same as on the wikipedia page).
https://en.wikipedia.org/wiki/RC4
Going off of the code on there, I've changed it into Java. Note that my convertKey method will be taking in a key from a separate class which utilizes Diffie-Hellman to obtain the key (I have the key hard coded right now to ensure RC4 works as intended), but only my RC4 class is non-functional, so unless specifically requested (since the other classes work fine and this RC4 one doesn't use them currently) I will just paste the RC4 class to avoid wasted space.
STACK TRACE:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -24 at cryptochat2.RC4.PRGA(RC4.java:80) at cryptochat2.RC4.main(RC4.java:132) Java Result: 1 –
/*
* To change this license header, choose License Headers in Project Properties.
* To change this template file, choose Tools | Templates
* and open the template in the editor.
*/
package cryptochat2;
/**
*
* #author Braydon
*/
//Initialize array of 256 bytes
// Run KSA on them (key stream algorithm)
// Run PRGA on KSA to generate keystream
// XOR the data with the keystream
import java.util.Arrays;
import java.util.*;
import java.util.stream.IntStream;
// The interesting feature of RC4 is that the key can be of any length from 1 to 256
//bytes. And again, the key is only used to initialize the permutation S.
//Note that the 256-byte array K is filled by simply repeating the key until
//the array is full.
public class RC4 {
byte[] lookUpTable = new byte[256]; // S
byte[] key = new byte[256];
byte[] initializedKey = new byte[256]; // key
byte[] keyStream;
byte keyStreamByte;
boolean generatingOutput;
//int key;
int keylength = 256;
// Call KSA then PRGA
public void KSA() {
// uses identity permutation while also converting to bytes
// then must process S
for (int k = 0; k < 256; k++) {
lookUpTable[k] = (byte) k;
}
// now processing permutation of array
int j = 0;
int tempOne = 0;
int tempTwo = 0;
for (int k = 0; k < 256; k++) {
j = (j + lookUpTable[k] + initializedKey[k % keylength]) % 256;
// Switching S[i] and S[j]
byte tmp;
for (int i = 0; i < 256; i++) {
j = (j + lookUpTable[i] + initializedKey[i]) & 0xFF;
tmp = lookUpTable[j];
lookUpTable[j] = lookUpTable[i];
lookUpTable[i] = tmp;
}
}
}
//Error in PRGA-- arrayIndexOutOfBounds, value differs based on key but
// error remains the same. It's an issue with the following method.
public void PRGA() {
int i = 0;
int j = 0;
byte tmp;
boolean generatingOutput = true;
while (generatingOutput) {
i = (i + 1) % 256;
j = (j + lookUpTable[i]) % 256;
for (int k = 0; k < 256; k++) {
j = (j + lookUpTable[i] + initializedKey[i]) & 0xFF;
tmp = lookUpTable[j];
lookUpTable[j] = lookUpTable[i];
lookUpTable[i] = tmp;
}
int keyStreamByteTemp = ((lookUpTable[i] + lookUpTable[j]) % 256);
try {// error's in this part vvvvvv
keyStreamByte = lookUpTable[keyStreamByteTemp];
System.out.println("keystream byte: " + keyStreamByte);
} catch (IndexOutOfBoundsException exception){
System.out.println(keyStreamByte + " has given an exception");
}
}
}
public void convertKey(int key) {
// call this first. it gives us our key.
int nonByte = key;
byte bytedKey = (byte) key;
// We create an int array and from it we initialize our key byte array
int[] data = IntStream.generate(() -> bytedKey).limit(256).toArray();
for (int i = 0; i < 256; i++) {
initializedKey[i] = (byte) data[i];
}
keylength = numberOfLeadingZeros(key);
}
public static int numberOfLeadingZeros(int i) {
// http://stackoverflow.com/questions/2935793/count-bits-used-in-int
if (i == 0) {
return 32;
}
int n = 1;
if (i >>> 16 == 0) {
n += 16;
i <<= 16;
}
if (i >>> 24 == 0) {
n += 8;
i <<= 8;
}
if (i >>> 28 == 0) {
n += 4;
i <<= 4;
}
if (i >>> 30 == 0) {
n += 2;
i <<= 2;
}
n -= i >>> 31;
return n;
}
public static void main(String[] args) {
RC4 RC4 = new RC4();
// temp hard coded key
RC4.convertKey(16);
RC4.KSA();
RC4.PRGA();
}
}
While I have a general understanding of RC4, it's very possible that I have an element or two confused, or incorrect, or something like that. The try/catch block gives me an infinite loop (due to the generatingOutput boolean I'd imagine, and since we aren't passing in an actual message yet) of numbers, positive and negative, throwing IndexOutOfBounds exceptions, until I stop the program.
keystream byte: -53
keystream byte: 105
105 has given an exception
keystream byte: 6
6 has given an exception
keystream byte: -111
keystream byte: 73
73 has given an exception
keystream byte: 66
keystream byte: -86
keystream byte: -104
keystream byte: -114
keystream byte: -117
keystream byte: 56
keystream byte: 67
67 has given an exception
keystream byte: 121
keystream byte: 10
keystream byte: -7
keystream byte: 16
keystream byte: 103
103 has given an exception
keystream byte: -65
-65 has given an exception
keystream byte: 31
31 has given an exception
keystream byte: 21
I would really appreciate any help, I'd like to learn this well and not disappoint my professor or myself.

Put byte array to JSON and vice versa

Is it possible to put a byte[] (byte array) to JSON?
if so, how can I do that in java? then read that JSON and convert that field again to byte[]?

Here is a good example of base64 encoding byte arrays. It gets more complicated when you throw unicode characters in the mix to send things like PDF documents. After encoding a byte array the encoded string can be used as a JSON property value.
Apache commons offers good utilities:
byte[] bytes = getByteArr();
String base64String = Base64.encodeBase64String(bytes);
byte[] backToBytes = Base64.decodeBase64(base64String);
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Base64_encoding_and_decoding
Java server side example:
public String getUnsecureContentBase64(String url)
throws ClientProtocolException, IOException {
//getUnsecureContent will generate some byte[]
byte[] result = getUnsecureContent(url);
// use apache org.apache.commons.codec.binary.Base64
// if you're sending back as a http request result you may have to
// org.apache.commons.httpclient.util.URIUtil.encodeQuery
return Base64.encodeBase64String(result);
}
JavaScript decode:
//decode URL encoding if encoded before returning result
var uriEncodedString = decodeURIComponent(response);
var byteArr = base64DecToArr(uriEncodedString);
//from mozilla
function b64ToUint6 (nChr) {
return nChr > 64 && nChr < 91 ?
nChr - 65
: nChr > 96 && nChr < 123 ?
nChr - 71
: nChr > 47 && nChr < 58 ?
nChr + 4
: nChr === 43 ?
62
: nChr === 47 ?
63
:
0;
}
function base64DecToArr (sBase64, nBlocksSize) {
var
sB64Enc = sBase64.replace(/[^A-Za-z0-9\+\/]/g, ""), nInLen = sB64Enc.length,
nOutLen = nBlocksSize ? Math.ceil((nInLen * 3 + 1 >> 2) / nBlocksSize) * nBlocksSize : nInLen * 3 + 1 >> 2, taBytes = new Uint8Array(nOutLen);
for (var nMod3, nMod4, nUint24 = 0, nOutIdx = 0, nInIdx = 0; nInIdx < nInLen; nInIdx++) {
nMod4 = nInIdx & 3;
nUint24 |= b64ToUint6(sB64Enc.charCodeAt(nInIdx)) << 18 - 6 * nMod4;
if (nMod4 === 3 || nInLen - nInIdx === 1) {
for (nMod3 = 0; nMod3 < 3 && nOutIdx < nOutLen; nMod3++, nOutIdx++) {
taBytes[nOutIdx] = nUint24 >>> (16 >>> nMod3 & 24) & 255;
}
nUint24 = 0;
}
}
return taBytes;
}

The typical way to send binary in json is to base64 encode it.
Java provides different ways to Base64 encode and decode a byte[]. One of these is DatatypeConverter.
Very simply
byte[] originalBytes = new byte[] { 1, 2, 3, 4, 5};
String base64Encoded = DatatypeConverter.printBase64Binary(originalBytes);
byte[] base64Decoded = DatatypeConverter.parseBase64Binary(base64Encoded);
You'll have to make this conversion depending on the json parser/generator library you use.

Amazingly now org.json now lets you put a byte[] object directly into a json and it remains readable. you can even send the resulting object over a websocket and it will be readable on the other side. but i am not sure yet if the size of the resulting object is bigger or smaller than if you were converting your byte array to base64, it would certainly be neat if it was smaller.
It seems to be incredibly hard to measure how much space such a json object takes up in java. if your json consists merely of strings it is easily achievable by simply stringifying it but with a bytearray inside it i fear it is not as straightforward.
stringifying our json in java replaces my bytearray for a 10 character string that looks like an id. doing the same in node.js replaces our byte[] for an unquoted value reading <Buffered Array: f0 ff ff ...> the length of the latter indicates a size increase of ~300% as would be expected

In line with #Qwertie's suggestion, but going further on the lazy side, you could just pretend that each byte is a ISO-8859-1 character. For the uninitiated, ISO-8859-1 is a single-byte encoding that matches the first 256 code points of Unicode.
So #Ash's answer is actually redeemable with a charset:
byte[] args2 = getByteArry();
String byteStr = new String(args2, Charset.forName("ISO-8859-1"));
This encoding has the same readability as BAIS, with the advantage that it is processed faster than either BAIS or base64 as less branching is required. It might look like the JSON parser is doing a bit more, but it's fine because dealing with non-ASCII by escaping or by UTF-8 is part of a JSON parser's job anyways. It could map better to some formats like MessagePack with a profile.
Space-wise however, it is usually a loss, as nobody would be using UTF-16 for JSON. With UTF-8 each non-ASCII byte would occupy 2 bytes, while BAIS uses (2+4n + r?(r+1):0) bytes for every run of 3n+r such bytes (r is the remainder).

If your byte array may contain runs of ASCII characters that you'd like to be able to see, you might prefer BAIS (Byte Array In String) format instead of Base64. The nice thing about BAIS is that if all the bytes happen to be ASCII, they are converted 1-to-1 to a string (e.g. byte array {65,66,67} becomes simply "ABC") Also, BAIS often gives you a smaller file size than Base64 (this isn't guaranteed).
After converting the byte array to a BAIS string, write it to JSON like you would any other string.
Here is a Java class (ported from the original C#) that converts byte arrays to string and back.
import java.io.*;
import java.lang.*;
import java.util.*;
public class ByteArrayInString
{
// Encodes a byte array to a string with BAIS encoding, which
// preserves runs of ASCII characters unchanged.
//
// For simplicity, this method's base-64 encoding always encodes groups of
// three bytes if possible (as four characters). This decision may
// unfortunately cut off the beginning of some ASCII runs.
public static String convert(byte[] bytes) { return convert(bytes, true); }
public static String convert(byte[] bytes, boolean allowControlChars)
{
StringBuilder sb = new StringBuilder();
int i = 0;
int b;
while (i < bytes.length)
{
b = get(bytes,i++);
if (isAscii(b, allowControlChars))
sb.append((char)b);
else {
sb.append('\b');
// Do binary encoding in groups of 3 bytes
for (;; b = get(bytes,i++)) {
int accum = b;
if (i < bytes.length) {
b = get(bytes,i++);
accum = (accum << 8) | b;
if (i < bytes.length) {
b = get(bytes,i++);
accum = (accum << 8) | b;
sb.append(encodeBase64Digit(accum >> 18));
sb.append(encodeBase64Digit(accum >> 12));
sb.append(encodeBase64Digit(accum >> 6));
sb.append(encodeBase64Digit(accum));
if (i >= bytes.length)
break;
} else {
sb.append(encodeBase64Digit(accum >> 10));
sb.append(encodeBase64Digit(accum >> 4));
sb.append(encodeBase64Digit(accum << 2));
break;
}
} else {
sb.append(encodeBase64Digit(accum >> 2));
sb.append(encodeBase64Digit(accum << 4));
break;
}
if (isAscii(get(bytes,i), allowControlChars) &&
(i+1 >= bytes.length || isAscii(get(bytes,i), allowControlChars)) &&
(i+2 >= bytes.length || isAscii(get(bytes,i), allowControlChars))) {
sb.append('!'); // return to ASCII mode
break;
}
}
}
}
return sb.toString();
}
// Decodes a BAIS string back to a byte array.
public static byte[] convert(String s)
{
byte[] b;
try {
b = s.getBytes("UTF8");
} catch(UnsupportedEncodingException e) {
throw new RuntimeException(e.getMessage());
}
for (int i = 0; i < b.length - 1; ++i) {
if (b[i] == '\b') {
int iOut = i++;
for (;;) {
int cur;
if (i >= b.length || ((cur = get(b, i)) < 63 || cur > 126))
throw new RuntimeException("String cannot be interpreted as a BAIS array");
int digit = (cur - 64) & 63;
int zeros = 16 - 6; // number of 0 bits on right side of accum
int accum = digit << zeros;
while (++i < b.length)
{
if ((cur = get(b, i)) < 63 || cur > 126)
break;
digit = (cur - 64) & 63;
zeros -= 6;
accum |= digit << zeros;
if (zeros <= 8)
{
b[iOut++] = (byte)(accum >> 8);
accum <<= 8;
zeros += 8;
}
}
if ((accum & 0xFF00) != 0 || (i < b.length && b[i] != '!'))
throw new RuntimeException("String cannot be interpreted as BAIS array");
i++;
// Start taking bytes verbatim
while (i < b.length && b[i] != '\b')
b[iOut++] = b[i++];
if (i >= b.length)
return Arrays.copyOfRange(b, 0, iOut);
i++;
}
}
}
return b;
}
static int get(byte[] bytes, int i) { return ((int)bytes[i]) & 0xFF; }
public static int decodeBase64Digit(char digit)
{ return digit >= 63 && digit <= 126 ? (digit - 64) & 63 : -1; }
public static char encodeBase64Digit(int digit)
{ return (char)((digit + 1 & 63) + 63); }
static boolean isAscii(int b, boolean allowControlChars)
{ return b < 127 && (b >= 32 || (allowControlChars && b != '\b')); }
}
See also: C# unit tests.

what about simply this:
byte[] args2 = getByteArry();
String byteStr = new String(args2);

Bit manipulation C source in Java

I try to calculate the checksum of a Sega Genesis rom file in Java. For this i want to port a code snipped from C into Java:
static uint16 getchecksum(uint8 *rom, int length)
{
int i;
uint16 checksum = 0;
for (i = 0; i < length; i += 2)
{
checksum += ((rom[i] << 8) + rom[i + 1]);
}
return checksum;
}
I understand what the code does. It sums all 16bit numbers (combined from two 8 bit ones). But what i didn't understand is what's happening with the overflow of the uint16 and how this transfers to Java code?
Edit:
This code seems to work, thanks:
int calculatedChecksum = 0;
int bufferi1=0;
int bufferi2=0;
bs = new BufferedInputStream(new FileInputStream(this.file));
bufferi1 = bs.read();
bufferi2 = bs.read();
while(bufferi1 != -1 && bufferi2 != -1){
calculatedChecksum += (bufferi1*256 + bufferi2);
calculatedChecksum = calculatedChecksum % 0x10000;
bufferi1 = bs.read();
bufferi2 = bs.read();
}

Simply put, the overflow is lost.
A more correct approach (imho) is to use uint32 for summation, and then you have the sum in the lower 16 bits, and the overflow in the upper 16 bits.

static int checksum(final InputStream in) throws IOException {
short v = 0;
int c;
while ((c = in.read()) >= 0) {
v += (c << 8) | in.read();
}
return v & 0xffff;
}
This should work equivalently; by using & 0xffff, we get to treat the value in v as if it were unsigned the entire time, since arithmetic overflow is identical w.r.t. bits.

You want addition modulo 216, which you can simply spell out manually:
checksum = (checksum + ((rom[i] << 8) + rom[i + 1])) % 0x10000;
// ^^^^^^^^^

How do I convert a signed decimal value to a 32 bit little-endian binary string?

I need to convert a signed decimal number into a 32 bit little-endian binary value. Does anyone by any chance know of a built-in Java class or function that can do this? Or have built one to do this?
The data is a longtitude/latitude value like -78.3829. Thanks for any help.

If it helps at all, here's a class that I made that converts longs to binary Strings and binary Strings to longs:
public class toBinary {
public static void main(String[] args) {
System.out.println(decimalToBinary(16317));
System.out.println(binaryToDecimal("11111111111111111111111111111111111100101001"));
}
public static long binaryToDecimal(String bin) {
long result = 0;
int len = bin.length();
for(int i = 0; i < len; i++) {
result += Integer.parseInt(bin.charAt(i) + "") * Math.pow(2, len - i - 1);
}
return result;
}
public static String decimalToBinary(long num) {
String result = "";
while(true) {
result += num % 2;
if(num < 2)
break;
num = num / 2;
}
for(int i = result.length(); i < 32; i++)
result += "0";
result = reverse(result);
result = toLittleEndian(result);
return result;
}
public static String toLittleEndian(String str) {
String result = "";
result += str.substring(24);
result += str.substring(16, 24);
result += str.substring(8, 16);
result += str.substring(0, 8);
return result;
}
public static String reverse(String str) {
String result = "";
for(int i = str.length() - 1; i >= 0; i--)
result += str.charAt(i);
return result;
}
}
It doesn't take decimal values, but it could probably give you a bit of guidance.

The conversion is trivial once you know what the endianess means on binary level. The question is more what do you really want to do with it?
public static int flipEndianess(int i) {
return (i >>> 24) | // shift byte 3 to byte 0
((i >> 8) & 0xFF00) | // shift byte 2 to byte 1
(i << 24) | // shift byte 0 to byte 3
((i & 0xFF00) << 8); // shift byte 1 to byte 2
}
This little method will swap around the bytes in an int to switch between little/big endian order (the conversion is symetric). Now you have a little endian int. But what would you do with that in Java?
More likely you need to write the data to a stream or something, then its only a question in which order you write the bytes out:
// write int to stream so bytes are little endian in the stream
// OutputStream out = ...
out.write(i);
out.write(i >> 8);
out.write(i >> 16);
out.write(i >> 24);
(For big endian you would just order the lines from bottom to top...)