In the past I have written code which handles incoming data from a serial port. The data has a fixed format.
Now I want to migrate this code to java (android). However, I see many obstacles.
The actual code is more complex, but I have a simplified version here:
#define byte unsigned char
#define word unsigned short
#pragma pack(1);
struct addr_t
{
byte foo;
word bar;
};
#pragma pack();
bool RxData( byte val )
{
static byte buffer[20];
static int idx = 0;
buffer[idx++] = val;
return ( idx == sizeof(addr_t) );
}
The RxData function is called everytime a byte is received. When the complete chunk of data is in, it returns true.
Some of the obstacles:
The used data types are not available to java. In other threads it is recommended to use larger datatypes, but in this case this is not a workable solution.
The size of the structure is in this case exactly 3 bytes. That's also why the #pragma statement is important. Otherwise the C compiler might "optimize" it for memory use, with a different size as a result.
Java also doesn't have a sizeof function and I have found no alternative for this kind of situation.
I could replace the 'sizeof' with a fixed value of 3, but that would be very bad practice IMO.
Is it at all possible to write such a code in java? Or is it wiser to try to add native c source into Android Studio?
Your C code has its problems too. Technically, you do not know how big a char and a short is. You probably want uint8_t and uint16_t respectively. Also, I'm not sure how portable packing is.
In Java, you need a class. The class might as well tell you how many bytes you need to initialise it.
class Addr
{
private byte foo;
private short bar;
public final static int bufferBytes = 3;
public int getUnsignedFoo()
{
return (int)foo & 0xff;
}
public int getUnsignedBar()
{
return (int)bar & 0xffff;
}
}
Probably a class for the buffer too although there may already be a suitable class in the standard library.
class Buffer
{
private final static int maxSize = 20;
private byte[] bytes = new byte[maxSize];
private int idx = 0;
private bool rxData(byte b)
{
bytes[idx++] = b;
return idx == Addr.bufferBytes;
}
}
To answer the question about the hardcodedness of the 3, this is actually the better way to do it because your the specification of your protocol should say "one byte for foo and two bytes for bar" not "a packed C struct with a char and a short in it". One way to deserialise the buffer is like this:
public class Addr
{
// All the stuff from above
public Addr(byte[] buffer)
{
foo = buffer[0];
bar = someFunctionThatGetsTheEndiannessRight(buffer[1], buffer[2]);
}
}
TI have left the way bar is calculated deliberately vague because it depends on your platform as much as anything. You can do it simply with bit shifts e.g.
(((short)buffer[1] & 0xff) << 8) | ((short)buffer[2] & 0xff)
However, there are better options available. For example, you can use a java.nio.ByteBuffer which has the machinery to cope with endian isssues.
Related
Native Code :
writing number 27 using fwrite().
int main()
{
int a = 27;
FILE *fp;
fp = fopen("/data/tmp.log", "w");
if (!fp)
return -errno;
fwrite(&a, 4, 1, fp);
fclose();
return 0;
}
Reading back the data(27) using DataInputStream.readInt() :
public int readIntDataInputStream(void)
{
String filePath = "/data/tmp.log";
InputStream is = null;
DataInputStream dis = null;
int k;
is = new FileInputStream(filePath);
dis = new DataInputStream(is);
k = dis.readInt();
Log.i(TAG, "Size : " + k);
return 0;
}
O/p
Size : 452984832
Well that in hex is 0x1b000000
0x1b is 27. But the readInt() is reading the data as big endian while my native coding is writing as little endian. . So, instead of 0x0000001b i get 0x1b000000.
Is my understanding correct? Did anyone came across this problem before?
From the Javadoc for readInt():
This method is suitable for reading bytes written by the writeInt method of interface DataOutput
If you want to read something written by a C program you'll have to do the byte swapping yourself, using the facilities in java.nio. I've never done this but I believe you would read the data into a ByteBuffer, set the buffer's order to ByteOrder.LITTLE_ENDIAN and then create an IntBuffer view over the ByteBuffer if you have an array of values, or just use ByteBuffer#getInt() for a single value.
All that aside, I agree with #EJP that the external format for the data should be big-endian for greatest compatibility.
There are multiple issues in your code:
You assume that the size of int is 4, it is not necessarily true, and since you want to deal with 32-bit ints, you should use int32_t or uint32_t.
You must open the file in binary more to write binary data reliably. The above code would fail on Windows for less trivial output. Use fopen("/data/tmp.log", "wb").
You must deal with endianness. You are using the file to exchange data between different platforms that may have different native endianness and/or endian specific APIs. Java seems to use big-endian, aka network byte order, so you should convert the values on the C platform with the hton32() utility function. It is unlikely to have significant impact on performance on the PC side, as this function is usually expanded inline, possibly as a single instruction and most of the time will be spent waiting for I/O anyway.
Here is a modified version of the code:
#include <endian.h>
#include <stdint.h>
#include <stdio.h>
int main(void) {
uint32_t a = hton32(27);
FILE *fp = fopen("/data/tmp.log", "wb");
if (!fp) {
return errno;
}
fwrite(&a, sizeof a, 1, fp);
fclose();
return 0;
}
I know that Java is no C, so no pointers and direct usable addresses but still I need it, because of the given library.
The library Andglk has been ported from native C and whoever did it, has the following constructor declaration in one of the classes:
public LineInputEvent(Window w, String s, int lineBuffer,
long maxLen, int dispatchRock, boolean unicode)
I now want to create one such event, but have no idea how to specify the lineBuffer. About the dispatchRock - it is also problematic but I believe it is not the bigger problem here.
Basically, this int lineBuffer (as the dispatchRock) is used one for all events during a session. One example is 1405421324 for all inputs.
About it (in the C documentation, there is for now almost nothing about andglk) is said:
This text will be placed in a buffer of your choice.
Any idea how can I specify it in Java for a given String?
EDIT:
This is from the andglk.c:
int andglk_loader_glk_MemoryStream_retainVmArray(JNIEnv *env, jobject this, int buffer, long length)
{
if (gli_register_arr) {
gidispatch_rock_t rock = gli_register_arr((void *)buffer, length, gidispatch_char_array);
return rock.num;
}
}
I'm writing a code generator that is replaying events recorded during a packet capture.
The JVM is pretty limited - it turns out. Methods can't be >64KB in size. So I added all kinds of trickery to make my code generator split up Java methods.
But now I have a new problem. I was taking a number of byte[] arrays and making them static variables in my class, e.g.:
public class myclass {
private static byte[] byteArray = { 0x3c, 0x3f, ...
...
};
private static byte[] byteArray2 = { 0x1a, 0x20, ...
...
};
...
private static byte[] byteArray_n = { 0x0a, 0x0d, ...
...
};
}
Now I get the error: "The code for the static initializer is exceeding the 65535 bytes limit".
I DO NOT WANT TO HAVE AN EXTERNAL FILE AND READ IN THE DATA FROM THERE. I WANT TO USE CODE GENERATED IN A SINGLE FILE.
What can I do? Can I declare the arrays outside the class? Or should I be using a string with unicode for the values 128-255 (e.g. \u009c instead of (byte)0x9c)? Or am I the only person in the world right now that wants to use statically initialised data?
UPDATE
The technique I'm now using is auto-creation of functions like the following:
private byte[] byteArray_6() {
String localString = "\u00ff\u00d8\u00ff\u00e0\u0000\u0010JFIF\u0000" +
"(0%()(\u00ff\u00db\u0000C\u0001\u0007\u0007\u0007\n\u0008\n\u0013\n" +
"\u0000\u00b5\u0010\u0000\u0002\u0001\u0003\u0003\u0002\u0004\u0003";
byte[] localBuff = new byte[ localString.length() ];
for ( int localInt = 0; localInt < localString.length(); localInt++ ) {
localBuff[localInt] = (byte)localString.charAt(localInt);
}
return localBuff;
}
Note: Java keeps on surprising. You'd think you could just encode every value in the range 0-255 as \u00XX (where XX is the 2-character hex representation). But you'd be wrong. The Java compiler actually thinks \u000A is a literal "\n" in your code - which breaks the compilation of your source code. So your strings can be littered with Unicode escapes but you'll have to use "\n" and "\r" instead of \u000a and \u000d respectively. And it doesn't hurt to put printable characters as they are in the strings instead of the 6 character Unicode escape representation.
Generally, you would put the data in a literal String and then have a method which decodes that to a byte[]. toByteArray() is of limited use as UTF-8 wont produce all possible byte sequences, and some values don't appear at all.
This technique is quite popular when trying to produce small object code. Removing huge sequences of array initialisation code will also help start up time.
Off the top of my head:
public static byte[] toBytes(String str) {
char[] src = str.toCharArray();
int len = src.length;
byte[] buff = new byte[len];
for (int i=0; i<len; ++i) {
buff[i] = (byte)src[i];
}
return buff;
}
More compact schemes are available. For instance you could limit string character contents to [1, 127] (0 is encoded in a non-normalised form for really bad reasons). Or something more complicated. I believe JDK8 will have a public API for Base64 decoding which isn't too bad and nicely standardised.
declare an arraylist and use a static constructor
May by you can use nested classes for storing static arrays.
This step is not the best in means of performans, but I think you could get it with minimum changes in your code.
I have the following C++ code to cipher a string with XOR.
#define MPI_CIPHER_KEY "qwerty"
Buffer FooClient::cipher_string(const Buffer& _landing_url)
{
String key(CIPHER_KEY);
Buffer key_buf(key.chars(), key.length());
Buffer landing_url_cipher = FooClient::XOR(_url, key_buf);
Buffer b64_url_cipher;
base64_encode(landing_url_cipher, b64_url_cipher);
return b64_url_cipher;
}
Buffer FooClient::XOR(const Buffer& _data, const Buffer& _key)
{
Buffer retval(_data);
unsigned int klen=_key.length();
unsigned int dlen=_data.length();
unsigned int k=0;
unsigned int d=0;
for(;d<dlen;d++)
{
retval[d]=_data[d]^_key[k];
k=(++k<klen?k:0);
}
return retval;
}
I have seen in this question such java impl. would that work for this case?
String s1, s2;
StringBuilder sb = new StringBuilder();
for(int i=0; i<s1.length() && i<s2.length();i++)
sb.append((char)(s1.charAt(i) ^ s2.charAt(i)));
String result = sb.toString();
or is there an easier way to do it?
doesn't look the same to me. the c++ version loops across all of _data no matter what the _key length was, cycling through _key as necessary. (k=(++k<klen?k:0); in the c++ code)
yours returns as soon as the shortest of key or data is hit.
Personally, i'd start with the closest literal translation of C++ to java that you can do, keeping param and local names the same.
Then write unit tests for it that have known inputs and outputs from C++
then start refactoring the java version into using java idioms/etc ensuring the tests still pass.
No - the java code will only XOR up to the length of the smaller string - whereas the C++ code will XOR the entire data completely.
Assuming s1 is your "key" this can be fixed by changing to
for(int i=0; i<s2.length();i++)
sb.append((char)(s1.charAt(i%s1.length()) ^ s2.charAt(i)));
Also the base-64 encoding of the return value is missing.
Is there a way to create a 128 bit object in java, that can be bit manipulated the same way as a long or int? I want to do 32 bit shifts and i want to be able to do a bit OR operation on the whole 128 bit structure.
Here, I present to you... an old idea. Now it's awfully downgraded (no code enhancer, no nothing) to simple 128 bit thingie that should be super fast, though. What I truly want is a ByteBuffer based array of C alike Struct but fully usable in java.
The main idea is allocating more than a single object at a time and using a pointer to the array. Thus, it greatly conserves memory and the memory is allocated in continuous area, so less cache misses (always good).
I did some moderate testing (but the code is still untested).
It does allow basic operations like add, xor, or, set/get with 128 bit numbers.
The standard rule: less documentation than expected applied unfortunately.
Adding extra code for extra operations should be straight forward.
Here is the code, look at main method for some usage. Cheers!
package bestsss.util;
import java.util.Random;
public class Bitz {
final int[] array;
private Bitz(int n){
array=new int[n<<2];
}
public int size(){
return size(this.array);
}
private static int size(int[] array){
return array.length>>2;
}
/**
* allocates N 128bit elements. newIdx to create a pointer
* #param n
* #return
*/
public static Bitz allocate(int n){
return new Bitz(n);
}
/**
* Main utility class - points to an index in the array
* #param idx
* #return
*/
public Idx newIdx(int idx){
return new Idx(array).set(idx);
}
public static class Idx{
private static final long mask = 0xFFFFFFFFL;
//dont make the field finals
int idx;
int[] array;//keep ref. here, reduce the indirection
Idx(int[] array){
this.array=array;
}
public Idx set(int idx) {
if (Bitz.size(array)<=idx || idx<0)
throw new IndexOutOfBoundsException(String.valueOf(idx));
this.idx = idx<<2;
return this;
}
public int index(){
return idx>>2;
}
public Idx shl32(){
final int[] array=this.array;
int idx = this.idx;
array[idx]=array[++idx];
array[idx]=array[++idx];
array[idx]=array[++idx];
array[idx]=0;
return this;
}
public Idx shr32(){
final int[] array=this.array;
int idx = this.idx+3;
array[idx]=array[--idx];
array[idx]=array[--idx];
array[idx]=array[--idx];
array[idx]=0;
return this;
}
public Idx or(Idx src){
final int[] array=this.array;
int idx = this.idx;
int idx2 = src.idx;
final int[] array2=src.array;
array[idx++]|=array2[idx2++];
array[idx++]|=array2[idx2++];
array[idx++]|=array2[idx2++];
array[idx++]|=array2[idx2++];
return this;
}
public Idx xor(Idx src){
final int[] array=this.array;
int idx = this.idx;
int idx2 = src.idx;
final int[] array2=src.array;
array[idx++]^=array2[idx2++];
array[idx++]^=array2[idx2++];
array[idx++]^=array2[idx2++];
array[idx++]^=array2[idx2++];
return this;
}
public Idx add(Idx src){
final int[] array=this.array;
int idx = this.idx+3;
final int[] array2=src.array;
int idx2 = src.idx+3;
long l =0;
l += array[idx]&mask;
l += array2[idx2--]&mask;
array[idx--]=(int)(l&mask);
l>>>=32;
l += array[idx]&mask;
l += array2[idx2--]&mask;
array[idx--]=(int)(l&mask);
l>>>=32;
l += array[idx]&mask;
l += array2[idx2--]&mask;
array[idx--]=(int)(l&mask);
l>>>=32;
l += array[idx]&mask;
l += array2[idx2--];
array[idx]=(int)(l&mask);
// l>>>=32;
return this;
}
public Idx set(long high, long low){
final int[] array=this.array;
int idx = this.idx;
array[idx+0]=(int) ((high>>>32)&mask);
array[idx+1]=(int) ((high>>>0)&mask);
array[idx+2]=(int) ((low>>>32)&mask);
array[idx+3]=(int) ((low>>>0)&mask);
return this;
}
public long high(){
final int[] array=this.array;
int idx = this.idx;
long res = (array[idx]&mask)<<32 | (array[idx+1]&mask);
return res;
}
public long low(){
final int[] array=this.array;
int idx = this.idx;
long res = (array[idx+2]&mask)<<32 | (array[idx+3]&mask);
return res;
}
//ineffective but well
public String toString(){
return String.format("%016x-%016x", high(), low());
}
}
public static void main(String[] args) {
Bitz bitz = Bitz.allocate(256);
Bitz.Idx idx = bitz.newIdx(0);
Bitz.Idx idx2 = bitz.newIdx(2);
System.out.println(idx.set(0, 0xf));
System.out.println(idx2.set(0, Long.MIN_VALUE).xor(idx));
System.out.println(idx.set(0, Long.MAX_VALUE).add(idx2.set(0, 1)));
System.out.println("==");
System.out.println(idx.add(idx));//can add itself
System.out.println(idx.shl32());//left
System.out.println(idx.shr32());//and right
System.out.println(idx.shl32());//back left
//w/ alloc
System.out.println(idx.add(bitz.newIdx(4).set(0, Long.MAX_VALUE)));
//self xor
System.out.println(idx.xor(idx));
//random xor
System.out.println("===init random===");
Random r = new Random(1112);
for (int i=0, s=bitz.size(); i<s; i++){
idx.set(i).set(r.nextLong(), r.nextLong());
System.out.println(idx);
}
Idx theXor = bitz.newIdx(0);
for (int i=1, s=bitz.size(); i<s; i++){
theXor.xor(idx.set(i));
}
System.out.println("===XOR===");
System.out.println(theXor);
}
}
Three possibilities have been identified:
The BitSet class provides some of the operations that you need, but no "shift" method. To implement this missing method, you'd need to do something like this:
BitSet bits = new BitSet(128);
...
// shift left by 32bits
for (int i = 0; i < 96; i++) {
bits.set(i, bits.get(i + 32));
}
bits.set(96, 127, false);
The BigInteger class provides all of the methods (more or less), but since BigInteger is immutable, it could result in an excessive object creation rate ... depending on how you use the bitsets. (There is also the issue that shiftLeft(32) won't chop off the leftmost bits ... but you can deal with this by using and to mask out the bits at index 128 and higher.)
If performance is your key concern, implementing a custom class with 4 int or 2 long fields will probably give best performance. (Which is actually the faster option of the two will depend on the hardware platform, the JVM, etc. I'd probably choose the long version because it will be simpler to code ... and only try to optimize further if profiling indicated that it was a potentially worthwhile activity.)
Furthermore, you can design the APIs to behave exactly as you require (modulo the constraints of Java language). The downside is that you have to implement and test everything, and you will be hard-wiring the magic number 128 into your code-base.
There is no longer data type than long (I have logged this as an RFE along with a 128 bit floating point ;)
You can create an object with four 32-bit int values and support these operations fairly easily.
You can't define any new types to which you could apply Java's built-in bitwise operators.
However, could you just use java.math.BigInteger? BigInteger defines all of the bit-wise operations that are defined for integral types (as methods). This includes, for example, BigInteger.or(BigInteger).
No.
Sorry there isn't a better answer.
One approach may be to create a wrapper object for two long values and implement the required functionality while taking signedness of the relevant operators into account. There is also BigInteger [updated from rlibby's answer], but it doesn't provide the required support.
Happy coding.
Perhaps BitSet would be useful to you.
It has the logical operations, and I imagine shifting wouldn't be all that hard to implement given their utility methods.
Afaik, the JVM will just convert whatever you code into 32 bit chunks whatever you do. JVM is 32 bit. I think even 64 bit version of JVM largely processes in 32 bit chunks. It certainly should to conserve memory... You're just going to slow down your code as the JIT tries to optimise the mess you create. In C/C++ etc. there's no point doing this either as you will still have impedance from the fact that it's 32 or 64 bit registers in the hardware you're most likely using. Even the Intel Xenon Phi (has 512bit vector registers) is just bunches of 32 and 64 bit elements.
If you want to implement something like that, you could try to do it in GLSL or OpenCL if you have GPU hardware available. In 2015 Java Sumatra will be released as part of Java 9, at least that's the plan. Then you will have the ability to integrate java with GPU code out of the box. That IS a big deal, hence the illustrious name!