Develop Reference

How to model Java primitive int[] array in protobuf message

I am new to Google Protocol buffers and trying to model a primitive int[] array in java via a protobuf message.
Currently i am using this proto file:
syntax = "proto3";
package protobuf;
message primitiveArrayMsg {
repeated int32 data = 1;
}
Which compiles to a java class with an List<Integer> data structure instead of an array of primitive int.
/**
* <code>repeated int32 data = 1;</code>
*/
java.util.List<java.lang.Integer> getDataList();
My application must hold millions of int values and for saving memory I decided to use int instead of Integer.
Is there a way to compile a protobuf message description in a java class with an int[] data structure?
Unfortunately, I found nothing in the Protocol Buffers Language Guide (proto3). A similar question was also asked in How to add a int array in protobuf message, which I tried but obviously the question author was looking for an ArrayList<Integer> and therefore the answer did not help me.
If there is no support for that, can you recommend me a more memory efficient way than boxing to Integer and using List<Integer>?

Protocol Buffer messages are not designed to handle large messages.
Even though the integers are efficiently packed by default when using proto3, a large amount of Integer objects would be needed in run-time memory (unless few distinct values are actually ever used, in which case the Integer objects can be re-used).
If you really do have to use Protocol Buffer messages for this, another option would be to transcribe the int arrays to and from a byte array format when encoding/decoding.

How to get the command data if it is more than 3 bytes?

I wanted to know how I can get the data send through an APDU command if buffer[ISO7816.OFFSET_LC] is >=3.
Actually I am using
if (numBytes == 1) {
shortAmount = (short) buffer[ISO7816.OFFSET_CDATA];
} else if (numBytes == 2) {
shortAmount = (short) Util.getShort(buffer, ISO7816.OFFSET_CDATA);
} else if(numBytes == 3) {
//how to get the all data contained in the APDU?
}

For most Java Card implementations there is no support for the int base type. This means you cannot store more than 16 bits in one single variable.
You can however store it in an array. And - when you think about it - it already is. The APDU buffer is nothing other than a byte array in transient memory (RAM, usually) *1.
So how you handle the APDU data in the APDU buffer is up to you:
you could parse structures inside the APDU buffer;
you could copy it to another array, either persistent (EEPROM / Flash using new byte[size]) or transient memory (RAM, through JCSystem.makeTransientByteArray())
you could convert it to shorts using Util.getShort() and store the result in a short array similar to a byte array;
you could use it as input to any API that allows input from a byte array, such as one of the Key#setValue commands, OwnerPIN#check(), signature verification etc. etc..
Or you can perform any combination of above.
If you want to perform calculations with larger values you'll have to implement those or have access to special libraries. For 32 bit integer calculations have a look at my X-mas special answer. This also shows that any kind of calculations on the data are possible, even if you have to implement them yourself. Java Card is Turing-complete, so as long as you don't run out of CPU time or memory you can do any possible calculation on it.
*1 OK, the APDU buffer is a rather special array in the way it is handled by the Java Card system, but during processing of data it just acts as a normal Java Card byte array.

Store the data in to a byte array.
Key point is, you need to process the data as a byte array if its size is >= 3. For processing however, you can develop your own API's or you can use the available one.

If you want to interpret the data sent through an APDU as an integer (== 4 bytes), then you need to store the data in to a byte array. Now depending upon the use case, you can use a JCInteger class provided by "Maarten Bodewes" to further process the data.
Key point is, you need to process the data as a byte array if its size is >= 3. For processing however, you can develop your own API's or you can use the available one.

best way of loading a large text file in java

I have a text file, with a sequence of integer per line:
47202 1457 51821 59788
49330 98706 36031 16399 1465
...
The file has 3 million lines of this format. I have to load this file into the memory and extract 5-grams out of it and do some statistics on it. I do have memory limitation (8GB RAM). I tried to minimize the number of objects I create (only have 1 class with 6 float variables, and some methods). And each line of that file, basically generates number of objects of this class (proportional to the size of the line in temrs of #ofwords). I started to feel that Java is not a good way to do these things when C++ is around.
Edit:
Assume that each line produces (n-1) objects of that class. Where n is the number of tokens in that line separated by space (i.e. 1457). So considering the average size of 10 words per line, each line gets mapped to 9 objects on average. So, there will be 9*3*10^6 objects.So, the memory needed is: 9*3*10^6*(8 bytes obj header + 6 x 4 byte floats) + (a map(String,Objects) and another map (Integer,ArrayList(Objects))). I need to keep everything in the memory, because there will be some mathematical optimization happening afterwards.

Reading/Parsing the file:
The best way to handle large files, in any language, is to try and NOT load them into memory.
In java, have a look at MappedByteBuffer. it allows you to map a file into process memory and access its contents without loading the whole thing into your heap.
You might also try reading the file line-by-line and discarding each line after you read it - again to avoid holding the entire file in memory at once.
Handling the resulting objects
For dealing with the objects you produce while parsing, there are several options:
Same as with the file itself - if you can perform whatever it is you want to perform without keeping all of them in memory (while "streaming" the file) - that is the best solution. you didnt describe the problem youre trying to solve so i dont know if thats possible.
Compression of some sort - switch from Wrapper objects (Float) to primitives (float), use something like the flyweight pattern to store your data in giant float[] arrays and only construct short-lived objects to access it, find some pattern in your data that allows you to store it more compactly
Caching/offload - if your data still doesnt fit in memory "page it out" to disk. this can be as simple as extending guava to page out to disk or bringing in a library like ehcache or the likes.
a note on java collections and maps in particular
For small objects java collections and maps in particular incur a large memory penalty (due mostly to everything being wrapped as Objects and the existence of the Map.Entry inner class instances). at the cost of a slightly less elegant API, you should probably look at gnu trove collections if memory consumption is an issue.

Optimal would be to hold only integers and line ends.
To that end, one way would be: convert the file to two files:
one binary file of integers (4 bytes)
one binary file with indexes where the next line would start.
For this one can use a Scanner to read, and a DataOutputStream+BufferedOutputStream to write.
Then you can load those two files in arrays of primitive type:
int[] integers = new int[(int)integersFile.length() / 4];
int[] lineEnds = new int[(int)lineEndsFile.length() / 4];
Reading can be done with a MappedByteBuffer.toIntBuffer(). (You then would not even need the arrays, but it would become a bit COBOL like verbose.)

could we read an object directly into the bytebuffer in java?

Say I have a List<Integer> ls and I know its length could we allocate length*4 bytes for bytebuffer and use put to directlty read ls into the buffer?

This should do what you're looking for:
ByteBuffer buffer = ByteBuffer.allocate(ls.size()*4);
for (Integer i: ls)
buffer.putInt(i);
One caveat: This function assumes that your list does not contain any null-entries.
I don't think you can do much better than this, since the underlying array of Integer objects in ls is an array of references (pointers) to these Integer objects rather than their contained int-values. The actual Integer objects will reside in memory in some random order, dictated roughly by when they were created. So, it's unlikely that you would find some consecutive block of memory that contains the data that you would need to copy into your ByteBuffer (I think this is what you meant by "directly reading" it.). So, something like using sun.misc.Unsafe will probably not be able to help you here.
Even if you had an int[] instead of a List<Integers> you probably won't be able to reliably read the data directly from memory into your ByteBuffer, since some JVM's on 64-bit machines will align all values on 64-bit address-locations for faster access, which would lead to "gaps" between your ints. And then there is the issue of endianness, which might differ from platform to platform...
Edit:
Hmmm... I just took a look at the OpenJDK source-code for the putInt()-function that this would be calling... It's a horrendous mess of sub-function-calls. If the "sun"-implementation is as "bad", you might be better off to just do the conversion yourself (using shifting and binary-ops) if you're looking for performance... The fastest might be to convert your Integers into a byte[] and then using ByteBuffer.wrap(...) if you need the answer as a ByteBuffer. Let me know if you need code...

Mapping java variables data to C Structures and write a c compatible file

I have a java class file with three arrayLists, one with type String, one with type Integer and other is ArrayList with type (ArrayList(String)). I have to write these these arraylists to a structure in C with character arrays, integers and short and output a file in a specefic format extension. The file has to be readable again by the same application. What is the best way to trasnfer the data from java to c structure and then output the c structure in a file. Thank you

There is no "C compatible file" format. If you have C structs written to disk file directly, then those are in an ad-hoc binary format. Exact format depends on things like packing and padding of the struct, byte order, word size of the CPU (like, 32 or 64 bit), etc.
So, start by defining the format, then forget it is produced by C.
Once you have the format defined, you can write a program to parse it in Java. If it is short with fixed length records, I'd probably create a class, which internally has just a private byte[] array, and then methods to manipulate it, save it and load it.

I suggest you write/read the data to a ByteBuffer using native byte ordering. The rest is up to you are to how you do it.
A library which might help is Javolution's Struct library which helps you map C structs onto ByteBuffers. This can help with C's various padding rules i.e.the exact layout might not be obvious.