Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deal with unsigned int to int conversion #1707

Merged
merged 10 commits into from
Jun 27, 2023

Conversation

alejoe91
Copy link
Member

@alejoe91 alejoe91 commented Jun 7, 2023

Fixes #1702 so you can do the following for a recording of typoe unsigned:

rec_int16 = rec_uint16.astype("int16")

Internally, it takes care of upcasting and removing the 2**(nbits-1) offset before adding the sign (to avoid overflow errors).

Also added some tests ;)

@alejoe91 alejoe91 added core Changes to core module preprocessing Related to preprocessing module labels Jun 7, 2023
@alejoe91 alejoe91 requested review from samuelgarcia and cwindolf June 7, 2023 11:28
@cwindolf
Copy link
Collaborator

cwindolf commented Jun 7, 2023

Cool! This one would be mildly surprising to me to me since the behavior is different from np.astype, but I have never worked with unsigned recordings and maybe this is what people in this domain would expect

@alejoe91 alejoe91 added this to the 0.98.0 milestone Jun 7, 2023
Copy link
Collaborator

@h-mayorquin h-mayorquin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not seen the other issue but I think that astype preprocessing should behave in exactly the same way as numpy.astype. Just because that is the behavior that our users will probably expect. Is this the case?

Edit: after thinking about it I am less sure. I will need to think.



def test_astype():
traces = (np.random.rand(10000, 4) * 100).astype("float32")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these tests will work better with a user defined traces

Just write something like [1.0 ,3.0, 5.0, ...] or at least let's make them non-deterministic.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's ok to leave randomness here. It should work in any case since it's only casting the dtypes

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is unlikely to create a problem here but I also don't see why the randomness is required. If it is meant to work as a test then it should be reproducible so you can debug it if it fails. But here, the failure is unlikely to come from randonmess so not a big concern ...

Anyway, not a big deal:
"A foolish consistency is the hobgoblin of little minds"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll set a seed ;)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@alejoe91 alejoe91 changed the title Deal with unsigned int in 'astype' Deal with unsigned int to int conversion Jun 27, 2023
@alejoe91
Copy link
Member Author

@cwindolf @h-mayorquin made a new preprocessor UnsignedToSignedRecording for the purpose! I agree that the astype should behave like numpy.

Ready to review!

Copy link
Collaborator

@h-mayorquin h-mayorquin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me:

  • Comment about docstrings which I think will be useful to have for us to remember and users for knowing what to expect. I think this will be good to add.
  • Style comments as suggestions / discussion. Feel free to ignore them.

Copy link
Collaborator

@h-mayorquin h-mayorquin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good ot me.

@alejoe91 alejoe91 merged commit 569b536 into SpikeInterface:main Jun 27, 2023
@samuelgarcia
Copy link
Member

I think we could have better implementation using numpy view. No one gave time to give feedback.

@alejoe91
Copy link
Member Author

@samuelgarcia it's been open for 3 weeks!

@h-mayorquin
Copy link
Collaborator

Wrote some thoughts on #1748
@alejoe91 @samuelgarcia

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Changes to core module preprocessing Related to preprocessing module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error : Compressing Maxwell hdf5 dataformat to Zarr using SpikeInterface
4 participants