LH5iterator behaviour if buffer size doesnt divide table size

[tdixon97]

When reading a LGDO.Table with lh5.LH5Iterator if the buffer size does not divide the size of the table some data is read twice in the last iteration.

For example:

from lgdo import lh5
import lgdo
print("LGDO version = ",lgdo.__version__)
from lgdo import Array, Table
import numpy as np

# create input data
arr = Array([1,2,3,4,5,6,7])
tab = Table(size=7)
tab.add_field("arr",arr)
print(f"data = {arr} length = {len(arr)}")
lh5.write(tab,"tab","test.lh5",wo_mode="of")

# read with Lh5Iterator
for lh5_obj,_,_ in lh5.LH5Iterator("test.lh5", "tab/arr", buffer_len=5):
    print(f"data = {lh5_obj.view_as('np')},    length = {len(lh5_obj.view_as('np'))}")

Which results in:

LGDO version =  1.9.0
data = [1 2 3 4 5 6 7] length = 7
data = [1 2 3 4 5],    length = 5
data = [6 7 3 4 5],    length = 5

So for the last iteration the buffer is not cleared for the rows which are not needed. This could mean that some rows of the Table are read twice...
What do you think @iguinn ?

[ManuelHu]

you always have to use n_rows_read returned by the iterator, otherwise you will get duplicate rows. This is - at the moment - expected and documented(?) behaviour.

also see related #107 (issue) and #109 (PR)

[tdixon97]

Ok makes sense, but I do not think its well documented (@iguinn ?) and potentially a bit confusing can lead to errors.

[iguinn]

Hi Toby, I agree that it is confusing, and I am hoping we can remove the need for n_rows_read altogether, as Manual said.