[Feature]: ROS3 HDF5 driver support

[yarikoptic]

What would you like to see added to MatNWB?

Analogous to pynwb/h5py support of ROS3 HDF5 driver mode https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html#streaming-method-1-ros3

I remember some folks talking about it but failed to find any issue, so decided to report

Is your feature request related to a problem?

No response

What solution would you like?

A solution similar to pynwb

Do you have any interest in helping implement the feature?

No.

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you ensured this change was not already requested?

[bendichter]

Thanks for opening this issue! In fact, this is possible to do, and you are right this should be a tutorial in MatNWB.

MATLAB has their own version of this. See the convo on the DANDI helpdesk here: dandi/helpdesk#59.

You need to make sure your credentials as set up correctly, and then you can just pass an s3 path to nwbRead. The problem is that this is quite slow. Last time I ran this on my local computer reading a file took 938 seconds. It will probably depend a lot on the file though and will probably be faster on DANDI Hub. I just tried this out but MATLAB isn't working on the hub for me: dandi/dandi-hub#42

[bendichter]

Just tested it locally.

>> s3 = 's3://dandiarchive/blobs/284/eb3/284eb346-0bc5-42a6-9b33-268e6b0b0bde'
>> tic; nwbRead(s3); toc;
Elapsed time is 20.215386 seconds.

I think > 20 seconds to open a file is too long to really be practical. Just opening it without any construction of the NWB objects is much more reasonable:

>> tic; H5F.open(s3); toc;
Elapsed time is 0.247859 seconds.

which makes me think that matnwb on DANDI Hub might be someone reasonable. I'll give that a shot once it's fixed