import pynwb is slow

[ahwillia]

Importing pynwb is surprisingly slow on my machine. Is this typical for others?

Python 3.9.9 (main, Jan 13 2022, 00:56:30) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: %time import pynwb
CPU times: user 2.74 s, sys: 1.7 s, total: 4.44 s
Wall time: 3.52 s

I think all my other packages are loading reasonably fast. For example, h5py loads in ~400 ms.

Python 3.9.9 (main, Jan 13 2022, 00:56:30) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: %time import h5py
CPU times: user 598 ms, sys: 1.47 s, total: 2.06 s
Wall time: 393 ms

[oruebel]

@mavaylon1 it would be helpful to run this through a profiler to see where the time is being spent. I assume the issue is likely that HDMF has to construct the TypeMap from the schema on import, but it will be good as a first step to get a clear understanding of where time is being spent so we can see what may be feasible to do to improve time for importing pynwb.

[ahwillia]

Yes I think importing HDMF accounts for a good chunk of this. Thanks for looking into it.

[mavaylon1]

Running this through a profiler, hdmf accounts for roughly 46-50% of the time for importing pynwb. Within the time spent for hdmf, a very small portion is spent on backend, i.e objectmapper, builders, and manager.

Using a more detailed profiler seems to point to loading the namespaces as a large factor in cumulative time spent.

[rly]

Related to #1050 and new issue #1255 . Initially, the problem with the linked approach of pickling the TypeMap is that it was not compatible with loading NWB extensions from a file. But I think this is not a completely blocking -- I think we can still pickle the TypeMap for the core namespace and load NWB extensions from a file. @mavaylon1 can you look into this?