Writing a plugin to auto-add test dependencies

[ydirson]

Looking for a way to use pytest-dependency marks to auto-collect dependencies of tests requested on command-line, I originally found a plugin in RKrahl/pytest-dependency#56, with which I've been experimenting, and which turns out to have issues, so I'm trying to understand what a more viable solution could look like.

First thing is, pytest_collection_modifyitems is not advertised for adding new items, only filter or re-order the items, and adding really looks like more part of a collector's job - but maybe there is a proper way to do that? Are there fundamental problems with adding new items from there (apart from them not being counted as collected, which might have to do with how that plugin is implemented)?

I also do not see how we could write a collector that would be able to decide in one single pass if every test it discovers is a (possibly recursive) dependency of the requested tests - would it be possible to run an additional collector as a second pass?

From what I understand of pytest_collection it should be possible to override it with a 2-pass collection process by overriding the while process, but it is not necessarily obvious how the default collection phase outlined in the API reference maps to the session.perform_collect() actual implementation:

It looks like the perform_collect() implementation takes steps in the following order, which does not fully match the doc:

• steps 1.1 to 1.3 are performed by collect_one_node()
• step 1.5 pytest_collectreport()
• step 1.4 is session.genitems()

I'm also a bit worried that going for a modified implementation of perform_collect() as pytest_collection hook could be problematic with upcoming pytest versions (relying on implementation details).

Looking at perform_collect() it looks like the pytest_collection_modifyitems is indeed not badly placed for that 2nd pass, but at first sight:

• that 2nd pass would need to perform a collection of self.config.args instead of the command-line args, and store collected items separately (would items for the same test be equal?)
• we'd just iterate on dependencies as in the original plugin, and try to locate them in that extra item set instead of trying to forge items for them
• as with a full fork of perform_collect(), it looks like we'd be accessing and duplicating implementation details (but less of them)

Any thoughts?