Develop Reference

Passing a collection of JNA structures to native method

The Problem
I am attempting to pass a collection of JNA structures to a native method but it's proving very fiddly:
Let's say we have a structure:
class MyStructure extends Structure {
// fields...
}
and a method in a JNA interface:
void pass(MyStructure[] data);
which maps to the native method:
void pass(const MYStructure* data);
Now the complication comes from the fact that the application is building a collection of these structures dynamically, i.e. we are NOT dealing with a static array but something like this:
class Builder {
private final Collection<MyStructure> list = new ArrayList<>();
// Add some data
public void add(MyStructure entry) {
list.add(entry);
}
// Pass the data to the native library
public void pass() {
// TODO
}
}
A naive implementation of the pass() method could be:
MyStructure[] array = list.toArray(MyStucture[]::new);
api.pass(array);
(where lib is the JNA library interface).
Of course this doesn't work because the array is not a contiguous block of memory - fair enough.
Rubbish Solution #1
One solution is to allocate a JNA array from a structure instance and populate it field-by-field:
MYStructure[] array = (MyStructure[]) new MyStructure().toArray(size);
for(int n = 0; n < array.length; ++n) {
array[n].field = list.get(n).field;
// other fields...
}
This guarantees the array consist of contiguous memory. But we have had to implement a field-by-field copy of the data (which we've already populated in the list) - this is OK for a simple structure, but some of the data I am dealing with has dozens of fields, structures that point to further nested arrays, etc. Basically this approach is just not viable.
Rubbish Solution #2
Another alternative is to convert the collection of data to a simple JNA pointer, something along these lines:
MyStructure[] array = list.toArray(MyStructure[]::new);
int size = array[0].size();
Memory mem = new Memory(array.length * size);
for(int n = 0; n < array.length; ++n) {
if(array[n] != null) {
array[n].write();
byte[] bytes = array[n].getPointer().getByteArray(0, size);
mem.write(n * size, bytes, 0, bytes.length);
}
}
This solution is generic so we can apply it to other structure as well. But we have to change the method signatures to be Pointer instead of MyStructure[] which makes the code more obtuse, less self-documenting and harder to test. Also we could be using a third-party library where this might not even be an option.
(Note I asked a similar question a while ago here but didn't get a satisfactory answer, thought I'd try again and I'll delete the old one / answer both).
Summary
Basically I was expecting/hoping to have something like this:
MyStructure[] array = MyStructure.magicContiguousMemoryBlock(list.toArray());
similar to how the JNA helper class provides StringArray for an array-of-string:
StringArray array = new StringArray(new String[]{...});
But no such 'magic' exists as far as I can tell. Is there another, simpler and more 'JNA' way of doing it? It seems really dumb (and probably incorrect) to have to allocate a byte-by-byte copy of the data that we essentially already have!
Do I have any other options? Any pointers (pun intended) gratefully accepted.

As the author of the previous answer, I realize a lot of the confusion was approaching it one way before realizing a better solution that we discussed primarily in comments to your answer. I will try to answer this additional clarification with an actual demonstration of my suggestion on that answer which I think is the best approach. Simply, if you have a non-contiguous structure and need a contiguous structure, you must either bring the contiguous memory to the structure, or copy the structure to the contiguous memory. I'll outline both approaches below.
Is there another, simpler and more 'JNA' way of doing it? It seems really dumb (and probably incorrect) to have to allocate a byte-by-byte copy of the data that we essentially already have!
I did mention in my answer on the other question that you could use useMemory() in this situation. It is a protected method but if you are already extending a Structure you have access to that method from the subclass (your structure), in much the same way (and for precisely the same purpose) as you would extend the Pointer constructor of a subclass.
You could therefore take an existing structure in your collection and change its native backing memory to be the contiguous memory. Here is a working example:
public class Test {
#FieldOrder({ "a", "b" })
public static class Foo extends Structure {
public int a;
public int b;
// You can either override or create a separate helper method
#Override
public void useMemory(Pointer m) {
super.useMemory(m);
}
}
public static void main(String[] args) {
List<Foo> list = new ArrayList<>();
for (int i = 1; i < 6; i += 2) {
Foo x = new Foo();
x.a = i;
x.b = i + 1;
list.add(x);
}
Foo[] array = (Foo[]) list.get(0).toArray(list.size());
// Index 0 copied on toArray()
System.out.println(array[0].toString());
// but we still need to change backing memory for it to the copy
list.get(0).useMemory(array[0].getPointer());
// iterate to change backing and write the rest
for (int i = 1; i < array.length; i++) {
list.get(i).useMemory(array[i].getPointer());
list.get(i).write();
// Since sending the structure array as an argument will auto-write,
// it's necessary to sync it here.
array[1].read();
}
// At this point you could send the contiguous structure array to native.
// Both list.get(n) and array[n] point to the same memory, for example:
System.out.println(list.get(1).toString());
System.out.println(array[1].toString());
}
Output (note the contiguous allocation). The second two outputs are the same, from either the list or the array.
Test$Foo(allocated#0x7fb687f0d550 (8 bytes) (shared from auto-allocated#0x7fb687f0d550 (24 bytes))) {
int a#0x0=0x0001
int b#0x4=0x0002
}
Test$Foo(allocated#0x7fb687f0d558 (8 bytes) (shared from allocated#0x7fb687f0d558 (8 bytes) (shared from allocated#0x7fb687f0d558 (8 bytes) (shared from allocated#0x7fb687f0d550 (8 bytes) (shared from auto-allocated#0x7fb687f0d550 (24 bytes)))))) {
int a#0x0=0x0003
int b#0x4=0x0004
}
Test$Foo(allocated#0x7fb687f0d558 (8 bytes) (shared from allocated#0x7fb687f0d558 (8 bytes) (shared from allocated#0x7fb687f0d550 (8 bytes) (shared from auto-allocated#0x7fb687f0d550 (24 bytes))))) {
int a#0x0=0x0003
int b#0x4=0x0004
}
If you don't want to put useMemory in every one of your structure definitions you can still put it in an intermediate class that extends Structure and then extend that intermediate class instead of Structure.
If you don't want to override useMemory() in your structure definitions (or a superclass of them), you can still do it "simply" in code with a little bit of inefficiency by copying over the memory.
In order to "get" that memory to write it elsewhere, you have to either read it from the Java-side memory (via reflection, which is what JNA does to convert the structure to the native memory block), or read it from Native-side memory (which requires writing it there, even if all you want to do is read it). Under-the-hood, JNA is writing the native bytes field-by-field, all hidden under a simple write() call in the API.
Your "Rubbish Solution #2" seems close to what's desired in this case. Here are the constraints that we have to deal with, with whatever solution:
In the existing list or array of Structure, the native memory is not contiguous (unless you pre-allocate contiguous memory yourself, and use that memory in a controlled manner, or override useMemory() as demonstrated above), and the size is variable.
The native function taking an array argument expects a block of contiguous memory.
Here are the "JNA ways" of dealing with structures and memory:
Structures have native-allocated memory at a pointer value accessible via Structure.getPointer() with a size of (at least) Structure.size().
Structure native memory can be read in bulk using Structure.getByteArray().
Structures can be constructed from a pointer to native memory using the new Structure(Pointer p) constructor.
The Structure.toArray() method creates an array of structures backed by a large, contiguous block of native memory.
I think your solution #2 is a rather efficient way of doing it, but your question indicates you'd like more type safety, or at least self-documenting code, in which case I'd point out a more "JNA way" of modifying #2 with two steps:
Replace the new Memory(array.length * size) native allocation with the Structure.toArray() allocation from your solution #1.
You still have a length * size block of contiguous native memory and a pointer to it (array[0].getPointer()).
You additionally have pointers to the offsets, so you could replace mem.write(n * size, ... ) with array[n].getPointer().write(0, ... ).
There is no getting around the memory copying, but having two well-commented lines which call getByteArray() and immediately write() that byte array seem clear enough to me.
You could even one-line it... write(0, getByteArray(0, size), 0, size), although one might argue if that's more or less clear.
So, adapting your method #2, I'd suggest:
// Make your collection an array as you do, but you could just keep it in the list
// using `size()` and `list.get(n)` rather than `length` and `array[n]`.
MyStructure[] array = list.toArray(MyStructure[]::new);
// Allocate a contiguous block of memory of the needed size
// This actually writes the native memory for index 0,
// so you can start the below iteration from 1
MyStructure[] structureArray = (MyStructure[]) array[0].toArray(array.length);
// Iterate the contiguous memory and copy over bytes from the array/list
int size = array[0].size();
for(int n = 1; n < array.length; ++n) {
if(array[n] != null) {
// sync local structure to native (using reflection on fields)
array[n].write();
// read bytes from the non-contiguous native memory
byte[] bytes = array[n].getPointer().getByteArray(0, size);
// write bytes into the contiguous native memory
structureArray[n].getPointer().write(0, bytes, 0, bytes.length);
// sync native to local (using reflection on fields)
structureArray[n].read();
}
}
From a "clean code" standpoint I think this rather effectively accomplishes your goal. The one "ugly" part of the above method is that JNA doesn't provide an easy way to copy fields between Structures without writing them to native memory in the process. Unfortunately that's the "JNA way" of "serializing" and "deserializing" objects, and it's not designed with any "magic" for your use case. Strings include built-in methods to convert to bytes, making such "magic" methods easier.
It is also possible to avoid writing the structure to native memory just to read it back again if you do the field-by-field copy as you implied in your Method #1. However, you could use JNA's field accessors to make it a lot easier to access the reflection under the hood. The field methods are protected so you'd have to extend Structure to do this -- which if you're doing that, the useMemory() approach is probably better! But you could then pull this iteration out of write():
for (StructField sf : fields().values()) {
// do stuff with sf
}
My initial thought would be to iterate over the non-contiguous Structure fields using the above loop, storing a Field.copy() in a HashMap with sf.name as the key. Then, perform that same iteration on the other (contiguous) Structure object's fields, reading from the HashMap and setting their values.

If you able to create a continues block of memory, why don't you simply de-serialize your list into it.
I.e. something like:
MyStructure[] array = list.get(0).toArray(list.size());
list.toArray(array);
pass(array);
In any case you'd better not to store Structure in your List or any another collection. It is better idea to hold a POJO inside, and then remap it to array of structures directly using a bean mapping library or manually.
With MapStruct bean mapping library it may looks like:
#Mapper
public interface FooStructMapper {
FooStructMapper INSTANCE = Mappers.getMapper( FooStructMapper.class );
void update(FooBean src, #MappingTarget MyStruct dst);
}
MyStrucure[] block = new MyStructure().toArray(list.size());
for(int i=0; i < block.length; i++) {
FooStructMapper.INSTANCE.update(list.get(i), block[i]);
}
What the point - Structure constructor allocates memory block using Memory, it is really slow operation. As well as memory allocated outside of java heap space. It is always better to avoid this allocate whenever you can.

The solutions offered by Daniel Widdis will solve this 'problem' if one really needs to perform a byte-by-byte copy of a JNA structure.
However I have come round to the way of thinking expressed by some of the other posters - JNA structures are intended purely for marshalling to/from the native layer and should not really be used as 'data'. We should be defining domain POJOs and transforming those to JNA structures as required - a bit more work but deal with I guess.
EDIT: Here is the solution that I eventually implemented using a custom stream collector:
public class StructureCollector <T, R extends Structure> implements Collector<T, List<T>, R[]> {
/**
* Helper - Converts the given collection to a contiguous array referenced by the <b>first</b> element.
* #param <T> Data type
* #param <R> Resultant JNA structure type
* #param data Data
* #param identity Identity constructor
* #param populate Population function
* #return <b>First</b> element of the array
*/
public static <T, R extends Structure> R toArray(Collection<T> data, Supplier<R> identity, BiConsumer<T, R> populate) {
final R[] array = data.stream().collect(new StructureCollector<>(identity, populate));
if(array == null) {
return null;
}
else {
return array[0];
}
}
private final Supplier<R> identity;
private final BiConsumer<T, R> populate;
private final Set<Characteristics> chars;
/**
* Constructor.
* #param identity Identity structure
* #param populate Population function
* #param chars Stream characteristics
*/
public StructureCollector(Supplier<R> identity, BiConsumer<T, R> populate, Characteristics... chars) {
this.identity = notNull(identity);
this.populate = notNull(populate);
this.chars = Set.copyOf(Arrays.asList(chars));
}
#Override
public Supplier<List<T>> supplier() {
return ArrayList::new;
}
#Override
public BiConsumer<List<T>, T> accumulator() {
return List::add;
}
#Override
public BinaryOperator<List<T>> combiner() {
return (left, right) -> {
left.addAll(right);
return left;
};
}
#Override
public Function<List<T>, R[]> finisher() {
return this::finish;
}
#SuppressWarnings("unchecked")
private R[] finish(List<T> list) {
// Check for empty data
if(list.isEmpty()) {
return null;
}
// Allocate contiguous array
final R[] array = (R[]) identity.get().toArray(list.size());
// Populate array
final Iterator<T> itr = list.iterator();
for(final R element : array) {
populate.accept(itr.next(), element);
}
assert !itr.hasNext();
return array;
}
#Override
public Set<Characteristics> characteristics() {
return chars;
}
}
This nicely wraps up the code that allocates and populates a contiguous array, example usage:
class SomeDomainObject {
private void populate(SomeStructure struct) {
...
}
}
class SomeStructure extends Structure {
...
}
Collection<SomeDomainObject> collection = ...
SomeStructure[] array = collection
.stream()
.collect(new StructureCollector<>(SomeStructure::new, SomeStructure::populate));
Hopefully this might help anyone that's doing something similar.

Memory usage/ reference in java

I am going to ask a basic question about Java memory usage.
Imagine we have an array List and it is large enough and we don't like to use more memory. Now if I want to pass this array to another methods in this class, or other classes through their constructor or method, do I need additional memory/is there additional memory usage for this array?
If yes, could I just make this array package level, and therefore the other classes in this package could access it directly, without any memory need.
Thank you in advance.

No, no additional memory is necessary. The parameter of a function is passed by copy of the reference. It means that for any kind of object only 4 additional bytes are used.
If you pass an array as parameter and you modify it in the body of the method the changes will be exported outside of method.
Instead if you reassign the array variable, the difference is not visible externally.
This happens because the parameters are passed as copy of the reference and not by reference.
public void vsibleModification(int[] a) {
for (int i = 0; i < a.length; i++) {
// This change is visible outside of method because I change
// the content of a, not the reference
a[i] = a[i] + 1;
}
}
public void nonVisibleModification(int[] a) {
// Non visible modification because a is reassigned to a new value (reference modification)
a = new int[2];
a[0] = 1;
a[1] = 2;
}

JNA array of structures uninitialized after being passed from Java to C

I've been using JNA with relative success to make native function calls from Java to a small C library that I wrote. Passing structures or pointers from one to the other works great once you've worked out the tricks of structure mapping, memory management and passing by reference.
I am now trying to pass, from Java to C, an array of structures. Here is the C code for the structure:
typedef struct key {
int length;
void *data;
} key_t;
I have the matching definition in Java:
public class Key extends Structure {
public int length;
public Pointer data;
public Key() {
this.setFieldOrder(new String[] {"length", "data"});
}
public void setAsLong(long value) {
this.length = 8;
this.data = new Memory(this.length);
this.data.setLong(0, value);
}
public long longValue() {
return this.data != null ? this.data.getLong(0) : Long.MIN_VALUE;
}
};
If I understood the documentation and what I read online, I need to create my array as a contiguous memory section by doing the following on the Java side:
Key[] keys = new Key().toArray(2);
for (int i=0; i<2; i++) {
k.setAsLong(42+i);
}
So far so good. If I dump the content of each Key structure in Java using Structure.toString(), everything is in here as expected. Note that the code about setting as a long value, allocating memory for the key's content, etc, work fine when I pass a single Key structure from Java to C. So here I pass my array to my native function by using the pointer to the first element of the array:
instance.foo(keys[0].getPointer(), keys.length);
My C function is of course defined like this:
void foo(key_t *keys, size_t count) {
...;
}
The array gets there correctly: the keys pointer on the C side has the same address as keys[0].getPointer() in Java, but unfortunately the members of each structure in the array are 0/NULL, as pointed out by GDB:
(gdb) print keys
$1 = (key_t *) 0x7fd7e82389e0
(gdb) print keys[0]
$2 = {length = 0, data = 0x0}
At this point I honestly have no clue what's going on. As I said, if I pass just one structure, it works fine, but here no way. The only difference I can see is the Java native method signature which uses Pointer instead of Key[] but when I use the array I get:
IllegalArgumentException: [Lfoo.bar.Key; is not a supported argument type (in method foo ...
Thanks

If you pass a Pointer value, JNA has no idea that you're actually passing a Structure or an array of them, and it's up to you to ensure you call Structure.write() before and Structure.read() after the native call.
If you pass either a Structure or Structure[], then JNA will take care of the synchronization automagically. In the case of Structure, JNA uses internal bookkeeping to determine whether the structure you're passing is at the head of an array of structures.

Java function parameter value is being changed

I've written a function which. Problem is, the parameters I'm sending, is being manipulated in the main program, though it is not my intention. I just want to have the value inside the function, but while operating, the actual value in the main program is also being changed.
How can I prevent this?
Here is my code:
Tiles[][] MoveRight(Tiles[][] tilesArray) {
Tiles[][] tempTilesArray = new Tiles[3][3];
Tiles[][] tempTilesArrayToSend = new Tiles[3][3];
tempTilesArrayToSend = CopyTilesArrays(tilesArray, tempTilesArrayToSend);
ArrayIndex zeroPos = FindZero(tilesArray);
Tiles zeroTile = GetTile(zeroPos, tilesArray);
if (zeroPos.column != 2) {
ArrayIndex otherPos = new ArrayIndex(zeroPos.row,
zeroPos.column + 1);
tempTilesArray = SwapTilesPositions(zeroTile, GetTile(otherPos,
tilesArray), tempTilesArrayToSend);
}
return tempTilesArray;
}
The array I'm sending inside the SwapPositionFunction is actually modifying the tilesArray itself. Though I've made a new instance of tiles array and then sent it.

Without seeing what is done in
CopyTilesArrays (tilesArray, tempTilesArrayToSend);
we can not say much.
Note, that in Java, there is no pass-by-value or pass-by-reference, but a copy of the reference is passed to the methods. This copy of a reference will - in case of objects and Arrays - point to the same, original object, so if you change the underlying/embedded object, the original object is affected, but if you change the reference, the original object is not affected.
IF you want to pass an independent copy of your array, you have to perform a deep ocpy. Maybe that is, what CopyTilesArrays is supposed to do, but without seeing it, we don't know.
Note too, that there are, or better: that there can be several layers of objects, with different reasons to stay on the surface, to go to the core, or to stay somewhere in between.
For example, to make a deep copy from the Array of Array of Tiles, you could do something like this:
public class TilesCopy {
Tiles[][] copyTilesArrays (Tiles[][] from, int outer, int inner) {
Tiles[][] to = new Tiles[outer][inner];
int o = 0;
for (Tiles [] tiles: from) {
Tiles[] fresh = new Tiles [inner];
int i = 0;
for (Tiles t : tiles)
{
fresh[i] = t.deepCopy ();
i++;
}
to [o] = fresh;
o++;
}
return to;
}
}
Note, that in the innermost loop, the elements aren't just referenced with fresh[i] = t;, but with a deep copy, to keep the objects in the original Array unaffected.
You could copy an array of arrays of Tiles in multiple other ways. For example, you could rearrange the outer array. If the Tiles were
[[A][B][C]]
[[D][E][F]]
[[G][H][I]]
you could copy them, and modify the target to be:
[[G][H][I]]
[[D][E][F]]
[[A][B][C]]
with just copying the outer arrays, and rearranging them. And you could copy the inner arrays, to be:
[[C][B][A]]
[[F][E][D]]
[[I][H][G]]
If you now modify the A to a, the original A will be affected too, without a deep copy:
[[C][B][a]]
[[F][E][D]]
[[I][H][G]]
[[a][B][C]]
[[D][E][F]]
[[G][H][I]]

Simulating pass by reference for an array reference (i.e. a reference to a reference) in Java

I was wondering, in java, is it possible to in anyway, simulate pass by reference for an array? Yes, I know the language doesn't support it, but is there anyway I can do it. Say, for example, I want to create a method that reverses the order of all the elements in an array. (I know that this code snippet isn't the best example, as there is a better algorithms to do this, but this is a good example of the type of thing I want to do for more complex problems).
Currently, I need to make a class like this:
public static void reverse(Object[] arr) {
Object[] tmpArr = new Object[arr.length];
count = arr.length - 1;
for(Object i : arr)
tmpArr[count--] = i;
// I would like to do arr = tmpArr, but that will only make the shallow
// reference tmpArr, I would like to actually change the pointer they passed in
// Not just the values in the array, so I have to do this:
for(Object i : tmpArr)
arr[count++] = i;
return;
}
Yes, I know that I could just swap the values until I get to the middle, and it would be much more efficient, but for other, more complex purposes, is there anyway that I can manipulate the actual pointer?
Again, thank you.

is there anyway that I can manipulate the actual pointer?
Java does not pass by reference, so you can't directly manipulate the original pointer. As you've found out, Java passes everything by value. You can't pass a reference to an array object, and expect a method to modify the original reference to point to another array object.
You can, of course:
Modify elements of the referred array object (ala java.util.Arrays.sort)
Pass a reference to an object with a settable field (e.g. Throwable has a setStackTrace)
return the new reference instead (ala java.util.Arrays.copyOf)

Well, you can explicitly pass an object that contains a reference. java.util.concurrent.atomic.AtomicReference is ready out of the box, although it does come with volatile semantics that you probably don't want. Some people use single element arrays to returns values from anonymous inner classes (although that doesn't seem a great idea to me).

This method reverses the Array's elements in place. The caller sees the changes. (In Java everything is passed by value, including object references.)
public static void reverse(Object[] arr) {
for ( int i = 0, j = arr.length - 1; i < j; i++, j-- ) {
Object temp = arr[i];
arr[i] = arr[j];
arr[j] = temp;
}
}

In Java Object reference is passed by value.
So if you looking for something like
function referenceCheck()
{
int[] array = new int[]{10, 20, 30};
reassignArray(&array);
//Now array should contain 1,2,3,4,5
}
function reassignArray(int **array)
{
int *array = new int[] { 1, 2, 3, 4, 5};
}
Then its not possible in Java by any direct means.
If we need to change only the values stored in an array, then we can do it since object reference is passed by value.

You want to pass a reference to the array reference. In that case you just have to either create a class to hold the reference and pass a reference to that class or just pass a 1-element array of the type being passed. Then you'd be passing either an object holding the array or an array whose only element contains the array you want to operate on.