Giving file_or_dir twice collects tests twice

[htgoebel]

When passing the same directory twice to py.test, all tests are run twice. The same is true if passing a sub-directory and a parent-directory:

$ python3 -m py.test --collect-only tests/ | grep collected
collecting ... collected 338 items
$ python3 -m py.test --collect-only tests/ tests/ | grep collected
collecting ... collected 676 items
$ python3 -m py.test --collect-only tests/unit/ tests/ | grep collected
collecting ... collected 418 items

platform linux -- Python 3.4.3, pytest-2.8.3, py-1.4.30, pluggy-0.3.1 -- /usr/bin/python3

Rationale: I want to run the unit-test (which are in a certain directory) prior to he other ones. Thus I put the tests/unit directory in front, but this gave me duplicate tests.

[nicoddemus]

Thanks for the report @htgoebel!

I had used this behavior of collecting tests more than once from the command-line in order to run the same test multiple times to track down some flaky behavior, but I think my use-case is a little contrived and yours makes more sense IMO.

Any more thoughts, @hpk42 @RonnyPfannschmidt @The-Compiler?

[The-Compiler]

I usually do something like while (($? == 0)); do py.test -k 'test_foo'; done in that case... I guess I should write a pytest-repeat plugin 😆

I don't really have an opinion on this, I can find arguments for and against both behaviours, and I'd be fine with both.

[hpk42]

don't have a strong opinion either but if in doubt i'd rather not change the default and better introduce a --drop-duplicate option or so. Maybe it's even better to write a pytest-drop-duplicates plugin which, when installed, will drop duplicates? It's easy to implement through the pytest_collect_modifyitems hook, basically a plugin with 20 lines of code or so.

[RonnyPfannschmidt]

I'm for the plugin solution

[htgoebel]

Plugin sounds complicated. So I'm for a plugin keep-duplicates.

[RonnyPfannschmidt]

a propperly separate keep-duplicates plugin would be impossible to do cleanly since the duplicates would be gone by the time it gets to the plugin

[nicoddemus]

Making a plugin is actually really easy using the cookiecutter-pytest-plugin, in fact just made one myself to quickly see it in action.

@htgoebel, could you try it out? If all is well I will publish it to PyPI, it still needs more tests and README. To install it use:

pip install git+https://github.com/nicoddemus/pytest-drop-dup-tests.git

No other configuration is necessary.

[RonnyPfannschmidt]

i'm closing this as working as intended given that @nicoddemus already finished a plugin

[nicoddemus]

Btw kudos to @hackebrot, it took me 10 minutes to have that plugin working, well done! cookiecutter-pytest-plugin makes it really simple to create new plugins, covering a lot of the boiler plate of setting up a new repository, setup.py, C.I., etc etc. Highly recommended. 👍

[htgoebel]

@nicoddemus The plugin does not show any effect here. I still get duplicates.

[nicoddemus]

Can you show me the output of a sample session?

[htgoebel]

It is the same as in my very fist comment here. I used --traceconfig to verified the plugin is loaded:

$ rm -rf /tmp/pytest-* ; python3 -m py.test --collect-only --traceconfig tests/unit/ tests/ | grep -P 'collected|-dup-'
  pytest-drop-dup-tests-0.1.0 at /home/xxx/.local/lib/python3.4/site-packages/pytest_drop_dup_tests.py
    drop-dup-tests      : /home/xxx/.local/lib/python3.4/site-packages/pytest_drop_dup_tests.py
plugins: drop-dup-tests-0.1.0
collected 418 items

[hackebrot]

Thank you @nicoddemus! 🙇 Happy to hear that 😸

[nicoddemus]

@htgoebel hmm that's strange. I will investigate this later (most probably tomorrow). I moved over this discussion over to nicoddemus/pytest-drop-dup-tests#2

[foxx]

Just came up against this problem, I'm confused why this functionality has to be placed into a separate plugin, why can't it be submitted as a pytest patch? Surely collecting tests twice, when given a directory to look for tests, would be considered a bug?

[nicoddemus]

Mostly because it is a behavior change and no one seemed to have a strong opinion for it... the change is trivial, as the plugin demonstrates. If more people are in favor of changing the behavior, I would gladly merge it into the core. 😄

[foxx]

Ah okay. I'm struggling to think of a scenario where allowing duplicates would be considered an advantage. Can someone give me an example of where this behaviour might be desired? If not, I'd vote for re-opening this for merge into the core.

PS) It turns out my duplication issue was caused by different bug, which I've raised in another issue.

[The-Compiler]

@foxx Check the history of this discussion - e.g. to run the same test multiple times to track down some flaky behaviour.

[foxx]

I had used this behavior of collecting tests more than once from the command-line in order to run the same test multiple times to track down some flaky behavior, but I think my use-case is a little contrived and yours makes more sense IMO.

I might have misinterpreted the use case, but I'd argue that if a test suite is producing different results on each run, with no actual change to any of the code, then the tests are broken.

That being said, most of the GNU core utils do not have any sort of de-duplication, for example if you run cat /etc/issue /etc/issue the file will be outputted twice. The same goes for find tests/ tests/, and ls /etc/passwd /etc/passwd.

In fact it could be argued that changing the behaviour to be different than the standard of most other utilities, would be the wrong thing to do.

In my situation, the problem was caused by a bug with setup.cfg handling (see #1311), and de-duplication would actually have hidden the problem, rather than fix it.

On that basis, I'm changing my vote to -1

[The-Compiler]

@nicoddemus FYI, for your usecase: I just wanted to write a pytest-repeat and then noticed it already exists 😆

[nicoddemus]

😆 thanks

[LewisGaul]

FWIW I'd like to see this implemented in core pytest, so just giving my +1 here.

See nicoddemus/pytest-drop-dup-tests#5 for my use-case and issue with the plugin approach (although thanks to niccodemus for making the plugin usable for my case!).