close analysis nwb files #1193
close analysis nwb files #1193
Conversation
|
Leaving the files open by default gives performance improvements when reaccessing data in a python process If you hit this error, you can run this to clear up you open files instead: from spyglass.utils.nwb_helper_fn import close_nwb_files
close_nwb_files() |
src/spyglass/utils/nwb_helper_fn.py
Outdated
| @@ -96,6 +96,11 @@ def get_nwb_file(nwb_file_path): | |||
| nwbfile = io.read() | |||
| __open_nwb_files[nwb_file_path] = (io, nwbfile) | |||
|
|
|||
| __open_nwb_files.clear() | |||
There was a problem hiding this comment.
Will this clear the cache every time?
There was a problem hiding this comment.
Yes. I agree this doesn't seem ideal and have removed the line.
| @@ -294,7 +296,7 @@ def fetch_nwb(query_expression, nwb_master, *attrs, **kwargs): | |||
| get_nwb_file(file_path) | |||
|
|
|||
| query_table = query_expression.join( | |||
| tbl.proj(nwb2load_filepath=attr_name), log_export=False | |||
There was a problem hiding this comment.
#1192 Has a fix for the log_export for non-SpyglassMixin tables that will still let the export work. Would prefer waiting for that to merge than pushing this change
There was a problem hiding this comment.
Sounds good, thanks.
There was a problem hiding this comment.
Can you revert the removals of log_export?
There was a problem hiding this comment.
I think the current failure of tests is because its not being logged
There was a problem hiding this comment.
done! Hopefully this works.
| @@ -237,7 +237,7 @@ def get_nwb_table(query_expression, tbl, attr_name, *attrs, **kwargs): | |||
| # TODO: check that the query_expression restricts tbl - CBroz | |||
| nwb_files = ( | |||
| query_expression.join( | |||
| tbl.proj(nwb2load_filepath=attr_name), log_export=False | |||
There was a problem hiding this comment.
see comment refering #1192 below
|
Hi @jguides, do you happen to have a snippet of code that creates the problem so we can evaluate the solution? |
Hi @edeno, this was happening with population of some custom tables that rely on multiple data entries. I think this would recreate it: |
This pull request has changes to avoid the error:
OSError: [Errno 24] Too many open files: '/home/jguidera/Src/jguides_2024/src/jguides_2024'
This error results from analysis nwb files being left open after being fetched.