pytest --pdb breaks Django test cases

[adamchainz]

Since #1890, when running with --pdb, Django test cases don't get per-test transactions. This is similar to #1932; Django has an equivalent _pre_setup function that needs calling per test to set up transactions (source - later in file for transaction support under TestCase). Since the --pdb change, the test case isn't called, so _pre_setup isn't called, so tests don't get transactional isolation.

A developer who always ran tests with --pdb noticed this first after our upgrade to 3.0.2. It took a long time to find that it was just the --pdb flag stopping database rollback, and not some weird behaviour in our app causing the inter-test failures 😢

[nicoddemus]

Thanks for the report! Sorry to hear this change brought you problems! 😞

I'm starting to think we probably should add a flag that is opted in to avoid this type of regression... apparently it is common for frameworks to override the TestCase.run method but not TestCase.debug.

@mbyt what do you think? Perhaps a --disable-unittest-teardown flag?

[adamchainz]

The other option is that it gets fixed upstream in Django, and then a piece of magic is added to pytest-django to fix it for the older Django versions.

[mbyt]

My conclusion is still the same as for #1932: If people overwrite unittest.TestCase __call__ or run and still want the post-mortem debugging feature, they need to overwrite debug in the same way (this is also true for standard unittest). So similar to adamchainz I suggests it would be anyway good if this gets fixed in Django upstream.

However for this case _post_teardown is empty, so there would not be any problem :.

What is unfortunate is that quite some people seem to overwriting run and thus now lose time when running into this issue. It would be really good, to at least somehow more pro-actively communicate this (for some cases backward incompatible) change.

[nicoddemus]

It would be really good, to at least somehow more pro-actively communicate this (for some cases backward incompatible) change.

I agree. I regret we didn't release this in 3.1.0, but I don't think we could have foreseen that it could cause this much trouble. 😞

[The-Compiler]

I need to say this: I knew why I mentioned (after the merge though...) that it should go to features 😉

[nicoddemus]

@The-Compiler if only. 😢

[adamchainz]

_post_teardown is not empty in all of Django's test case classes, only in SimpleTestCase. If you're using the database in anyway you have to use TransactionTestCase or TestCase which roll back the database to the pre-test setup - see https://github.com/django/django/blob/master/django/test/testcases.py#L909 . So that's how --pdb here breaks things - the first test doesn't reset the database and the second one gets an unclean state which is normally enough to break it...

I'll report this to Django and try find a fix to add to pytest-django.

[nicoddemus]

@adamchainz thanks a lot!

Please keep this thread updated. 😁

I'm closing this for now, but we should re-open if we decide to take another route in the future.

[blueyed]

@adamchainz
We have a fix for pytest-django in pytest-dev/pytest-django#406.
Have you reported this to Django already? Please followup on the pytest-django PR.