Loading time takes a while after io.read()

[GaelleChapuis]

Hello,

Together with Olivier @oliche we noticed that it takes quite a while to load a dataset into memory, even after having read the file via io.read.
Below is the code we both ran to evaluate the time each code line takes (the comments contain our timing in seconds).

You can first notice it takes a variable amount of time across users (probably depending on the network quality - mine is extremely fast), but the trend remains the same: charging the units via units = nwb.units.to_dataframe() is the longest part of the process.

Is this normal ? or are we charging the data wrongly?

I would be happy to show you the full tutorial where this came from, if it helps.
Thank you for your help!

## Import
import time

# DANDI
import fsspec
import h5py
from fsspec.implementations.cached import CachingFileSystem
from pynwb import NWBHDF5IO
from dandi.dandiapi import DandiAPIClient
DANDISET_ID = '000409'

# Get S3 URL for NWB file on DANDI
filepath = 'sub-KS042/sub-KS042_ses-07dc4b76-5b93-4a03-82a0-b3d9cc73f412_behavior+ecephys+image.nwb'

cache_dir = "/tmp/fsspec_cache"  # Local folder for the cache

with DandiAPIClient() as client:
    asset = client.get_dandiset(DANDISET_ID, 'draft').get_asset_by_path(filepath)
    s3_url = asset.get_content_url(follow_redirects=1, strip_query=True)

# configure CachingFileSystem for streaming
cfs = CachingFileSystem(
  fs=fsspec.filesystem("http"),
  cache_storage=cache_dir,
)
file_system = cfs.open(s3_url, "rb")
file = h5py.File(file_system)

# Get io
t = time.time()
io = NWBHDF5IO(file=file, load_namespaces=True)
print(time.time() - t)  # this takes in second: 180 OW / 821 CCU / 26 GC

# Read in the NWB file
t = time.time()
nwb = io.read()
print(time.time() - t)  # this takes in second: 25 OW / 0.8 GC

# Load spike data
t = time.time()
units = nwb.units.to_dataframe()
print(time.time() - t)   # this takes in second: 2783 OW / 60 GC

[satra]

@GaelleChapuis - can you try this remote access wrapper created by @magland? https://github.com/magland/remfile

some of the variability is due to speeds of s3 changing over time depending on demand. unfortunately, there is nothing we can do about that bit.

[CodyCBakerPhD]

@GaelleChapuis @oliche

Also, on top of remfile, you should see vastly improved read speed of this type of data if you paginate the processed HDF5 files with a large (~10 MB) page buffer size

A part of the reason it's likely feeling slow is because the Units table data is split into several small pieces (each column is a different dataset with a different byte range requiring several small GET requests) whereas pagination would theoretically keep many of those bundled together in a single (or a few) larger requests