Stateful tests with Dataset

[dcherian]

I was curious to see if the hypothesis stateful testing would catch an inconsistent sequence of index manipulation operations like #8646. Turns out rename_vars is basically broken? (filed #8659) :P

PS: this blog post is amazing.

E           state = DatasetStateMachine()
E           state.assert_invariants()
E           > ===
E
E            <xarray.Dataset>
E           Dimensions:  ()
E           Data variables:
E               *empty*
E           ===
E
E
E           > vars: ('1', '1_')
E           state.add_dim_coord(var=<xarray.Variable (1: 1)>
E           array([0], dtype=uint32))
E           state.assert_invariants()
E           > ===
E
E            <xarray.Dataset>
E           Dimensions:  (1: 1)
E           Coordinates:
E             * 1        (1) uint32 0
E           Data variables:
E               1_       (1) uint32 0
E           ===
E
E
E           > renaming 1 to 0
E           state.rename_vars(newname='0')
E           state.assert_invariants()
E           > ===
E
E            <xarray.Dataset>
E           Dimensions:  (1: 1)
E           Coordinates:
E             * 0        (1) uint32 0
E           Dimensions without coordinates: 1
E           Data variables:
E               1_       (1) uint32 0
E           ===
E
E
E           state.teardown()

[TomNicholas]

Oh this is a really cool idea!

Also yeah I've read that blog post before and my mind was also blown haha

[max-sixty]

Great idea! Really cool.

I even wonder to what extent this could replace vast swathes of our tests — anything where the output is calculated from the input.

We'll always want some unit tests to code to when developing, but plausibly the unit tests are either a) examples or b) some saved cases that have failed these model-based tests before.

[benbovy]

That looks very cool indeed! (although I don't think that rename_vars is broken :) but instead some valid cases require skipping the default indexes invariant check).

[dcherian]

instead some valid cases require skipping the default indexes invariant check.

Can we write down rules for when these checks are needed?

[benbovy]

Yes although I think it is easier to write rules for when these checks aren't needed.

One example: skip the default indexes invariant test when the name of an existing dimension coordinate is passed as input kwarg or dict key to .rename_vars().

[dcherian]

Hah now our strategy tests are failing

[Zac-HD]

Obviously there are some remaining todo comments, but this looks great! I'd consider merging it more or less as-is, and then continuing work in a follow-up PR 🙂

[Zac-HD]

I think HypothesisWorks/hypothesis#3944 will enable this, though I wouldn't delay this PR to wait.

[Zac-HD]

It doesn't seem that surprising that you hit bugs related to empty-string or null-prefixed string? To most C code, those are identical!

[dcherian]

Thanks for the review @Zac-HD