Add test helper to explain why lists do not match.

[vingkan]

While pairing with @emmadiamon, we realized the error messages for lists of dataclasses can be really hard to read because of the large number of attributes.

Consider the example where a test is compared the actual and expected lists of Match objects and the only difference is that one has score=1 and the other has score=0.

Before

If the test uses a raw assertion:

assert actual == expected

Then the pytest assertion error will show all the attributes of the mismatched list items, which can be hard to read:

After

Now, if you use this new test helper function:

from pipeline.testing import equal_lists

assert equal_lists(actual, expected)

Then the assertion error will provide a brief explanation and display the attributes that do not match, for the first pair of items that do not match:

This takes advantage of the rich attribute comparison for dataclasses that pytest already supports, but it only works when comparing individual dataclass objects, not a list of dataclass objects: pytest-dev/pytest#3776

Logic

The equal_lists helper considers four cases:

1. Lists are equal -> PASS
2. Lists have different sizes -> FAIL with descriptive error message showing diff between lists
3. Lists have the same size and same items, but different orders -> FAIL with descriptive error message showing diff between first mismatched pair of items
4. Lists have the same size, but different items -> FAIL with descriptive error message showing diff between first mismatched pair of items

Caveats

• If you want to use equal_list you need to import it into your test.
• Your list values must be sortable to use equal_list, otherwise it will raise an error when checking if the lists have the same values, but different orders.
  • You can set @dataclass(order=True) on a dataclass schema to enable this.
• This pull request extends pytest assertion rewriting into the pipeline.testing.assertions module so that equal_lists can pick up the pytest assertion message.
  • The pytest assertion message can change at runtime based on the pytest settings, so I have mocked it out of the unit tests for equal_lists.

[gitpod-io]

[ghost]

👇 Click on the image for a new way to code review

• Make big changes easier — review code in small groups of related files

• Know where to start — see the whole change at a glance

• Take a code tour — explore the change with an interactive tour

• Make comments and review — all fully sync’ed with github

  Try it now!

Legend