Excluding Tests for Verify Types

[hmc-cs-mdrissi]

Following typed library documentation and using pyright --verifytype, I learned that for one of my libraries many of the errors came from tests folders. The suggestion there is,

If your package includes tests or sample code, consider removing them from the distribution. If there is good reason to include them, consider placing them in a directory that begins with an underscore so they are not considered part of your library’s interface.

and that does work to rename all tests -> _tests. It is some churn and I think folders named test/tests are fairly common practice. Could rule either automatically exclude those or could rule be configurable with list of exclusions for verifytypes?

There is an exclude setting, but while I don't want tests treated as public api, I do still expect to CI check that there's no type errors in them. Maybe non_public/exclude_library setting?

[erictraut]

Why would you would want to include a tests directory as part of a library's distribution? Why force all consumers of your library to download the tests when installing the library?

While you could exclude a directory called tests when running --verifytypes, that masks the real problem. If you ship a tests folder as part of your library, tools like pylance will assume that tests is part of your public interface for your library and will include completion suggestions for it. That's probably not what you want.

So I recommend that you remove tests from the installed library. If there's a good reason to include it, then renaming it to _tests is the way to tell tools "this isn't part of my public interface".

[hmc-cs-mdrissi]

This is fairly common though for various reasons. The best reason is it helps downstream distributors of library be able to test library. It also allows users to run unit tests directly as a way to check install worked. Contents of unit test itself may not be part of api, but being able to run pytest some_lib/tests may be expected to work. My personal reason is more niche and is workaround for pytest issue. I can rename libraries I work on to _tests and probably will if needed.

There’s a long debate in python packaging here on whether package should include or remove tests, https://discuss.python.org/t/should-sdists-include-docs-and-tests/14578 As an example numpy, pyarrow, and scipy all include tests inside installed package. There are other packages that don’t but it’s one common convention that is debated back and forth.