Improve tests samples

[15r10nk]

I would like to improve the quality of the test samples, because I had some issues with them during the implementation of #31. I would like to get some feedback on this Idea.

The problems

• They take long to execute and are therefore only executed with EXECUTING_SLOW_TESTS=1 enabled.
• They cover by far not all kinds of ast/bytecode python could generate
• It is not useful to add more to them because they would run even longer
• They are also a major slowdown for the build pipeline.

I tested the #31 branch with ~400000 python files which I collected in different ways from github/pypi and found a lot of issues there which where not covered by any existing test.
I got the idea, at the end of #31, that it might be useful to add minimized versions of this files which contain only the problematic code to the samples. However this idea came to late and I finished #31 without adding tests to the samples.

But there is maybe another way:

• Lets say we use a new sample folder ... lets call it tests/issues/ for now which is initial empty.

• It should be possible to use mutmut to generate a code change inside executing/_position_node_finder.py which produces no problem with an code sample in tests/issues/ (this the case for the first run and gets more interesting later).

• We can run then over a much bigger set of files (400000 if it is necessary) to find a file which contains some code which raises an error for the change mutmut did in executing.

• This file can be minimized to the exact code which leads to the problem. Adding this minimized file to tests/issues would show mutmut that this issue is covered. mutmut would search for another issue in the next run.

This process could be completely automated and generate a new set of small and diverse sample files.
Every file would contain some special case which should be handled by executing.

It should be almost impossible to change the implementation of executing in a bad way (this is what mutmut tries really hard all the time), because there is a new issue file for almost every case.

I know that this will probably not work perfect, but I think it is worth to try it.

Some questions ...

I have started to work on the minimizer for the python source, because I could not find any thing I could use, but does anyone knows a tool I could use?
It should take a python source file as input and remove code from it as long as some condition holds.
In our case as long as the source file produces the issue.

I would like to know about any ideas or questions to this approach.

[alexmojaki]

If you think this will make the tests more thorough and faster at the same time, and you'll have fun doing it, then go for it!

For minimising a python source file, I imagine you'd want to take a 'binary search' approach and repeatedly remove half of a list of AST nodes wherever you can.