Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very very slow loading of Spike data in current dev branch #360

Closed
KatharineShapcott opened this issue Oct 7, 2022 · 1 comment · Fixed by #362
Closed

Very very slow loading of Spike data in current dev branch #360

KatharineShapcott opened this issue Oct 7, 2022 · 1 comment · Fixed by #362
Assignees
Labels
Critical Bug A critical error that either complete breaks (parts of) the package or causes erroneous calculation

Comments

@KatharineShapcott
Copy link
Contributor

Describe the bug

Went from 180 ms to 9.43 seconds!

image

This seems to be specific to spike data, here I'm looping through all the data types in a folder and most times are similar:

0.3.dev187
ADC
CPU times: user 7.22 ms, sys: 4.03 ms, total: 11.2 ms
Wall time: 25.6 ms
events
CPU times: user 264 ms, sys: 3.99 ms, total: 268 ms
Wall time: 271 ms
lfp
CPU times: user 2.99 ms, sys: 1.02 ms, total: 4.01 ms
Wall time: 6.57 ms
raw
CPU times: user 4.83 ms, sys: 4 µs, total: 4.84 ms
Wall time: 16.9 ms
spikes
CPU times: user 9.21 s, sys: 214 ms, total: 9.42 s
Wall time: 9.46 s
2022.8.1a0
ADC
CPU times: user 5.93 ms, sys: 0 ns, total: 5.93 ms
Wall time: 7.25 ms
events
CPU times: user 4.68 ms, sys: 713 µs, total: 5.4 ms
Wall time: 5.39 ms
lfp
CPU times: user 2.35 ms, sys: 1.95 ms, total: 4.3 ms
Wall time: 4.17 ms
raw
CPU times: user 3.35 ms, sys: 2.06 ms, total: 5.41 ms
Wall time: 4.88 ms
spikes
CPU times: user 160 ms, sys: 19.9 ms, total: 180 ms
Wall time: 181 ms

Files sizes looks like this:
image

To Reproduce
Switch to the current dev branch and load in spike data.

Expected behavior
Loading time stays the same as previously.

System Profile:

  • OS: ESI cluster

  • Please paste the output of the following command here

    import syncopy as spy; spy.AnalogData().log
     >>> SyNCopy v. 0.3.dev187 <<< 
    

Created: Fri Oct 7 18:29:01 2022

System Profile:
3.8.13 (default, Mar 28 2022, 11:38:47)
[GCC 7.5.0]
ACME: 0.21
Dask: 2021.10.0
NumPy: 1.21.2
SciPy: 1.7.3

--- LOG ---

|=== shapcottk@esi-svhpc23: Fri Oct 7 18:29:02 2022 ===|

__init__: created AnalogData object

Additional Information
I don't notice any change in the memory usage and it's not specific to loading with tags.

@tensionhead tensionhead self-assigned this Oct 8, 2022
@tensionhead tensionhead added the Critical Bug A critical error that either complete breaks (parts of) the package or causes erroneous calculation label Oct 8, 2022
@tensionhead tensionhead linked a pull request Oct 14, 2022 that will close this issue
@tensionhead
Copy link
Contributor

tensionhead commented Oct 14, 2022

So the problem at hand got solved, however we found unexpected (nTrials^2 + nTrials) scaling of that expensive, and with the fix avoided again, equality check here
We are also wondering why for a simple init of AnalogData the __eq__ operator gets into action in the first place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Critical Bug A critical error that either complete breaks (parts of) the package or causes erroneous calculation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants