problem with multidimensional uint8_t arrays

[cozzyd]

In ROOT 6.32.08 , which uses cppyy 3.1.2, an issue appeared that didn't happen before.

Basically we store a multimensional uint8_t array, and are having trouble converting it to a numpy array.

Here is a reproducer in the same spirit showing the problem.

import cppyy
import cppyy.ll
import numpy

cppyy.cppdef("""
 struct Foo {uint8_t bar[24][2] = {};};
 Foo foo;
 """)

arr = cppyy.gbl.foo.bar
a = numpy.frombuffer(cppyy.ll.cast['uint8_t*'](arr), dtype='uint8', count=24*2).reshape((24,2))
print(a)

but now this no longer works, presumably related to the changes for making uint8_t work in other cases? The cast was necessary to make it work before (and it still doesn't work without the cast, with numpy complains now complaining that it's not C contiguous, though I think it was a different problem earlier, like wrong size or something, I can check).

The LowLevelView itself has the right shape and format, and I can access elements fine from the LowLevelView. If I create a memoryview from the LowLevelView it indeed claims it's not C contiguous and also claims that that the itemsize is 8?

[wlav]

There have been a couple of recent changes with handling uint8_t and with how recent numpy handles copying. Which version are you using? This works with HEAD:

a = numpy.frombuffer(arr, dtype='uint8', count=4*2).reshape((4,2))

(The cast fails b/c the array is kept a uint8_t (as desired), but the cast apparently still resolves to unsigned char.)

The memoryview probably fails b/c none of the flags are set (it also claims not to be f_contiguous), so that needs fixing. (The numpy conversion works through __array__, not the buffer interface.)

[cozzyd]

Unfortunately, the direct conversion doesn't seem to work with cppyy 3.1.2 (in ROOT 6.32.08):

Traceback (most recent call last):
  File [redacted]/reproducer.py", line 31, in <module>
    a = numpy.frombuffer(arr, dtype='uint8',count=24*2).reshape((24,2))
BufferError: memoryview: underlying buffer is not C-contiguous

Trying

arr.__array__()

outputs the same error on this cppyy version.

However, I have found a workaround that works in ROOT 6.32.08 (and is presumably backwards compatible as well):

cppyy.cppdef(" uint8_t* uint8_t_ptr_cast(void *x) { return (uint8_t*) x; } ")
uint8_t_ptr_cast = cppyy.gbl.uint8_t_ptr_cast
a = numpy.frombuffer(uint8_t_ptr_cast(arr), dtype='uint8', count=24*2).reshape((24,2))

[wlav]

The cast fails b/c internally, the LowLevelView is represented as the 1D array of pointers to 1D arrays of uint8_t, rather than as a flat array (this is only about the view; it doesn't say anything about the actual underlying array). The upshot is that the check to the item size fails as the expected is 1 byte (sizeof(uint8_t)), but it gets 8 bytes (sizeof(uint8_t)).

What the checking code should do, is make sure the array is flat and c_contiguous and then it can go to inner view to check the item size.

That said, with the just released 3.5.0, both passing arr directly (no cast) and using __array__() work fine.