JNA: How to specify a variable-length (0+) array in a Structure?

user_5860073 0
Viewed 339

Example structure, where count is the number of bytes in the array, which could be 0. I want to allocate new instances in Java, and also read native-allocated instances.

    public class VarArray extends Structure {
        public byte dummy0;
        public short dummy1;
        public int count;
        public byte[] array;
    }

Using array = new byte[0] is not permitted in Structure.
Declaring default array = new byte[1] will read from a non-allocated address if the count is 0.

Removing the array field is ok for reading, as I can access bytes from pointer offset Structure.size() [Edit: incorrect, depends on padding]. However for allocating a new instance I need to manually determine field sizes and alignment padding in order to allocate the correct memory size.

I have a solution using two types - one without array for native-allocated and 0-count Java-allocated instances, and a sub-type with array for Java-allocated 1+ count instances. This seems rather bloated, especially with the required boiler-plate code.

Is there a better way?
Or perhaps a simple way to calculate the field size and alignment so that one type will suffice?

import java.util.Arrays;
import java.util.List;
import com.sun.jna.Pointer;
import com.sun.jna.Structure;

public class JnaStructTester {

    /**
     * For native-allocated, and 0-count JNA-allocated instances.
     */
    public static class VarArray extends Structure {
        public byte dummy0;
        public short dummy1;
        public int count;

        public VarArray() {}

        public VarArray(Pointer p) {
            super(p);
        }

        public byte[] getArray() {
            byte[] array = new byte[count];
            if (count > 0) {
                int offset = size();
                getPointer().read(offset, array, 0, count);
            }
            return array;
        }
        
        @Override
        protected List<String> getFieldOrder() {
            return List.of("dummy0", "dummy1", "count");
        }
    }
    
    /**
     * For 1+ count JNA-allocated instances.
     */
    public static class VarArrayX extends VarArray {
        public byte[] array;

        public VarArrayX() {}

        @Override
        public byte[] getArray() {
            return array;
        }
        
        @Override
        protected List<String> getFieldOrder() {
            return List.of("dummy0", "dummy1", "count", "array");
        }
    }
    
    public static void main(String[] args) {
        var va0 = new VarArrayX();
        va0.dummy0 = (byte) 0xef;
        va0.dummy1 = (short) 0xabcd;
        va0.count = 7;
        va0.array = new byte[] { 1, 2, 3, 4, 5, 6, 7 };
        va0.write();
        
        var va1 = new VarArray();
        va1.dummy0 = (byte) 0xab;
        va1.dummy1 = (short) 0xcdef;
        va1.write();
        
        print(new Pointer(Pointer.nativeValue(va0.getPointer())));
        print(new Pointer(Pointer.nativeValue(va1.getPointer())));
    }
    
    private static void print(Pointer p) {
        var va = new VarArray(p);
        va.read();
        System.out.println(va);
        System.out.println("byte[] array=" + Arrays.toString(va.getArray()));
        System.out.println();
    }
}

Output:

JnaStructTester$VarArray(native@0x7fb6835524b0) (8 bytes) {
  byte dummy0@0=ffffffef
  short dummy1@2=ffffabcd
  int count@4=7
}
byte[] array=[1, 2, 3, 4, 5, 6, 7]

JnaStructTester$VarArray(native@0x7fb683551210) (8 bytes) {
  byte dummy0@0=ffffffab
  short dummy1@2=ffffcdef
  int count@4=0
}
byte[] array=[]

(I am using the fairly old JNA version 4.2.2)

Update (2020-01-07)

Thanks to Daniel Widdis' suggestions, here is a nice solution.

Dynamically modifying the field list, based on whether the array is empty, is not possible. The layout is statically cached when the variable array field is not included, so a future instance with a non-empty array would fail. Instead:

  1. The array is adjusted in ensureAllocated(), to avoid empty array errors.
  2. writeField() only writes the field if the array is non-empty.
  3. readField() sets the array size from count, which has already been read, and skips reading the array if the count is 0.

By setting the array to the correct size in readField(), the full array is populated automatically, no need for manual creation.

import java.util.Arrays;
import java.util.List;
import com.sun.jna.Pointer;
import com.sun.jna.Structure;

public class JnaStructTester {

    public static class VarArray extends Structure {
        public short dummy0;
        public int dummy1;
        public byte count;
        public byte[] array = new byte[0];

        public VarArray() {}

        public VarArray(byte[] array) {
            this.count = (byte) array.length;
            this.array = array;
        }

        public VarArray(Pointer p) {
            super(p);
        }

        @Override
        protected void ensureAllocated() {
            if (count == 0) array = new byte[1];
            super.ensureAllocated();
            if (count == 0) array = new byte[0];
        }

        @Override
        protected void writeField(StructField structField) {
            if (structField.name.equals("array") && count == 0) return;
            super.writeField(structField);
        }

        @Override
        protected Object readField(StructField structField) {
            if (structField.name.equals("array")) {
                array = new byte[count];
                if (count == 0) return null;
            }
            return super.readField(structField);
        }

        @Override
        protected List<String> getFieldOrder() {
            return List.of("dummy0", "dummy1", "count", "array");
        }
    }

    public static void main(String[] args) {
        var va0 = new VarArray(new byte[] { 1, 2, 3, 4, 5, 6, 7 });
        va0.dummy0 = 0x4321;
        va0.dummy1 = 0xabcdef;
        va0.write();

        var va1 = new VarArray();
        va1.dummy0 = 0x4321;
        va1.dummy1 = 0xabcdef;
        va1.write();
        
        print(new Pointer(Pointer.nativeValue(va0.getPointer())));
        print(new Pointer(Pointer.nativeValue(va1.getPointer())));
    }

    private static void print(Pointer p) {
        var va = new VarArray(p);
        va.read();
        System.out.println(va);
        System.out.println("byte[] array=" + Arrays.toString(va.array));
        System.out.println();
    }
}

Output:

JnaStructTester$VarArray(native@0x7fd85cf1ffb0) (12 bytes) {
  short dummy0@0=4321
  int dummy1@4=abcdef
  byte count@8=7
  byte array[7]@9=[B@4f2410ac
}
byte[] array=[1, 2, 3, 4, 5, 6, 7]

JnaStructTester$VarArray(native@0x7fd85cf20690) (12 bytes) {
  short dummy0@0=4321
  int dummy1@4=abcdef
  byte count@8=0
  byte array[0]@9=[B@722c41f4
}
byte[] array=[]

I have only tested this in my narrow use-case, it may not work if other methods are called on Structure.

jna structure arrays java
1 Answers

I do think a solution with two structures, with the array version extending the other but simply adding the new field, is a reasonable solution. Your concern about "boilerplate bloat" is greatly reduced with JNA 5.X which has a @FieldOrder annotation, which significantly reduces that boilerplate. Your structures would be rather simple:

@FieldOrder({"dummy0", "dummy1", "count"})
public class VarArray extends Structure {
    public byte dummy0;
    public short dummy1;
    public int count;
}

@FieldOrder({"dummy0", "dummy1", "count", "array"})
public class VarArrayX extends VarArray  {
    public byte[] array = new byte[1];

    public VarArrayX(int arraySize) {
        array = new byte[arraySize];
        super.count = arraySize;
        allocateMemory();
    }
}

Other than adding constructors if you wanted to initialize with a pointer, this should be sufficient.

You could do a single-structure version, however, by doing the same alteration of the FieldOrder. JNA's memory allocation is dependent on the field order, defined using the getFieldList() and getFieldOrder() methods (which the new annotation removes boilerplate requirements for).

You can keep the array allocation in the structure, but alter the getFieldOrder() override to skip the field that defines the array in the case that it's zero sized, doing the same for getFieldList(). I had to deal with exactly this sort of situation when writing the Sysinfo structure in JNA's Linux LibC mapping. The main structure includes this field:

public byte[] _f = new byte[PADDING_SIZE];

But the getFieldList() override includes:

if (PADDING_SIZE == 0) {
    Iterator<Field> fieldIterator = fields.iterator();
    while (fieldIterator.hasNext()) {
        Field field = fieldIterator.next();
        if ("_f".equals(field.getName())) {
            fieldIterator.remove();
        }
    }
}

and getFieldOrder() includes:

if (PADDING_SIZE == 0) {
    fieldOrder.remove("_f");
}

Similar conditionals remove the _f_unused field in the Statvfs structure in the same file.

In the case of your structure, as long as you know the count before instantiating the structure, something like this should work (untested):

public static class VarArray extends Structure {
    public byte dummy0;
    public short dummy1;
    public int count;
    public byte[] array = new byte[1];

    public VarArray(int arraySize) {
        array = new byte[arraySize];
        count = arraySize;
        allocateMemory();
    }

    @Override
    protected List<String> getFieldOrder() {
        if (count == 0) {
            return List.of("dummy0", "dummy1", "count");
        } else {
            return List.of("dummy0", "dummy1", "count", "array");
        }
    }

    // do the same for getFieldList()
}
user_11614840
Related