denylist: support being able to denylist a subtest

[dustymabe]

We currently have a specific sub-test that is failing consistently on certain platforms. It would be nice if we could denylist/snooze just the subtest.

With:

- pattern: podman.base/resources                                                                                                             
  tracker: https://github.com/coreos/fedora-coreos-tracker/issues/966                                                                        
  snooze: 2021-10-15                                                                                                                         
  platforms:                                                                                                                                 
    - aws                                                                                                                                    
    - gcp                                                                                                                                    
    - qemu-unpriv

I currently see:

kola -p qemu-unpriv --build 35.20210924.dev.1 --output-dir tmp/kola run podman.base
⚠️  Skipping kola test pattern "fcos.internet":
  👉 https://github.com/coreos/coreos-assembler/pull/1478
⚠️  Skipping kola test pattern "podman.workflow":
  👉 https://github.com/coreos/coreos-assembler/pull/1478
🕒 Snoozing kola test pattern "podman.base/resources" until Oct 15 2021:
  👉 https://github.com/coreos/fedora-coreos-tracker/issues/966
=== RUN   podman.base
=== RUN   podman.base/info
=== RUN   podman.base/resources
--- PASS: podman.base (28.07s)
    --- PASS: podman.base/info (0.27s)
    --- PASS: podman.base/resources (3.76s)
PASS, output in tmp/kola

Which is not correct.

[saqibali-2k]

Sounds like a useful addition to the current functionality. Currently, kola only becomes aware of most sub tests (with the exception of non-exclusive tests) at test runtime (ie they are not registered). The current denylist logic is handled before tests are started, so sub tests are overlooked.

Two approaches come to mind:

• Add additional denylist checking for only subtests in harness.tRunner or harness.Run. This is straightforward
  but we are scattering the denylist logic.
• Move ALL the denylist logic to harness. At runtime, we check if any test - not just a subtest - is denylisted.

[jlebon]

While subtests are often used to separate truly independent subtests, in practice they're also used as "checkpoints" of test code that build on top of each other. So I don't think one can simply denylist a single subtest, without having it also imply denylisting all the subtests that follow.

Maybe a better avenue is to enhance the exclusive tests work to support subgroups, and then split out instances of the first kind of subtests above into separate tests sharing the same provisioned resources?

[dustymabe]

While subtests are often used to separate truly independent subtests, in practice they're also used as "checkpoints" of test code that build on top of each other. So I don't think one can simply denylist a single subtest, without having it also imply denylisting all the subtests that follow.

Good to know. Though I think in practice when the user went to denylist the test and then did a subsequent run, they'd discover the dependency and realize denylisting the individual subtest won't work and they'll need to denylist the top level test.

Maybe a better avenue is to enhance the exclusive tests work to support subgroups, and then split out instances of the first kind of subtests above into separate tests sharing the same provisioned resources?

Sounds a bit complicated (maybe?) for what I was going for here. If this feature request isn't somewhat easy to do we should probably skip it.