Add a plan of plans feature

[sopos]

This is an extension on plan import. The goal is to be able to reference multiple plans with just one link (fmf id).

Imagine following situation:

an SST-A repo:

/plans/Tier1:
    tag:
      - tier
    discover:
      - how: fmf
        filter: tag:tier1
      - how: fmf
        url: https://remote.url
        filter: tag:tier1
    execute:
      - how: tmt
/plans/Tier2:
    tag:
      - tier
    discover:
      - how: fmf
        filter: tag:tier2
      - how: fmf
        url: https://remote.url
        filter: tag:tier2
    execute:
      - how: tmt

Now, I want to run all the Tier plans from the SST-A's repo. To include whole sub-tree of plans or filtered plans my plan should look like this:

/SST-A:
    plan:
        include:
            url: https://sstA.repo.url
            name: /plans
            filter: tag:tier

It could be easily internally converted to the known problem:

/SST-A:
    /plans/Tier1:
        import:
            url: https://sstA.repo.url
            name: /plans/Tier1
    /plans/Tier2:
        import:
            url: https://sstA.repo.url
            name: /plans/Tier2

Which in turn would effectively be:

/SST-A:
    /plans/Tier1:
        tag:
          - tier
        discover:
          - how: fmf
            filter: tag:tier1
          - how: fmf
            url: https://remote.url
            filter: tag:tier1
        execute:
          - how: tmt
    /plans/Tier2:
        tag:
          - tier
        discover:
          - how: fmf
            filter: tag:tier2
          - how: fmf
            url: https://remote.url
            filter: tag:tier2
        execute:
          - how: tmt

+ correctly cloned remote repository for the local discovery to work.

[martinpitt]

"Me too". In cockpit-project/cockpit#19117 I am exploring how to do effective reverse-dependency testing between upstream tests. For example, we'd like to run cockpit-podman's tests in all https://github.com/containers/podman PRs, or cockpit's tests in all systemd or fedora-selinux PRs. It's not possible via packit, and it's a bit awkward with tmt right now as all these projects (selinux, etc.) have to duplicate the plan structure and thus they are hard to change or add.

This seems to be true for both importing plans (as name: is required) and also for the standard discover: url method.

So, some way of "run every plan that you find in url: http:/..." would be very useful. Thanks!

[happz]

I'm trying to understand the difference between what tmt already has - "take a plan from a repo and replace the current plan with it" - and what's the proposal here. Is it "just" about taking multiple plans from a repo, and sort of replacing the current plan with all of them? Sorting them under the current plan as its children, but effectively doing import of multiple plans instead of one?

[martinpitt]

@happz That sounds more or less what we need, but more like a "recursive import". E.g. we want to run cockpit-podman's three plans in podman. Currently podman's (and all other consumers, like SELinux has to keep a duplicate of cockpit's plan structure, and kept in sync. This doesn't scale well if the plans change over time, as that change has to be applied to all other projects which want to run these "foreign" plans.

A naive approach of just pointing to a foreign repo and not duplicating the plan structure results in a single plan which runs all of the repo's (intended) plans as tests -- i.e. serially in a single instance.

[idorax]

Hi @sopos and @martinpitt, here is a summary after discussion - add plan of plans.

• A single plan would expand into many plans, and parent-child relationship will be used instead of replacing the local plan.
• For the naming, we would like to concatenate local (referencing) plan name + remote (referenced) plan name. e.g.

  /plans/local/plans/remote
      /plans/something

• For selecting plans, it is very likely to support to select by a regular expression. By default, all plans would be imported when name is not provided.
• To trigger the new functionality, just omit the name or use a regular expression. And the implementation would be backward compatible with the sing one-one mapping.
• To support mixing local config with remote content, tmt should complain about such a combination:

  execute:
      how: tmt
      script: echo
  plan:
      import:
          url: here
          name: (optional) REGEX

• To avoid recursion, we would not pay efforts to implement it from the start. We can consider it
  in the future.

Hi @psss, @happz, @lukaszachy and @thrix, please help to correct the summary above if anything is incorrect, thanks very much!

[happz]

Hi @psss, @happz, @lukaszachy and @thrix, please help to correct the summary above if anything is incorrect, thanks very much!

+1.

One thing I'm not sure about is how the feature is enabled: not specifying the name is fine, but "use a regular expression" - /foo is also a regular expression, matches /foobar and many other strings, yet it contains no interesting regex-ish characters like dot. It's very hard to tell whether a string is not supposed to be a regular expression...

I think we should add a flag, another key to enable this feature, to make it clear that the user really wants us to import remote plans - even just a single plan! - under the local one. We do have the whole import mapping to play with.

plan:
    import:
        url: https://src.fedoraproject.org/tests/binutils
        # No name set, imports all plans.

    import:
        # Not a regular expression, but all plans in the repo start with `/plans`.
        name: /plans

    import:
        # Not a regular expression, yields just a single plan - shall we replace the local
        # one, or did the user want it to be put under the local plan?
        name: /plans/build-gating/common

    #
    # With a flag telling tmt to "import and slot under the local plan". Let's call it `integrate` for now.
    #

    import:
        url: https://src.fedoraproject.org/tests/binutils

        # No name set, imports all plans.
        # integrate: true -> nothing changes, remote plans slotted under the local plan.
        # integrate: false -> raise an error, tmt is not replacing a single plan with multiple remote ones.

    import:
        # Not a regular expression, but all plans in the repo start with `/plans`.
        name: /plans
        # integrate: true -> nothing changes, remote plans slotted under the local plan.
        # integrate: false -> raise an error, tmt is not replacing a single plan with multiple remote ones.

    import:
        # Not a regular expression, yields just a single plan - shall we replace the local
        # one, or did the user want it to be put under the local plan?
        name: /plans/build-gating/common

        # integrate: true -> slot the remote plan under the local one. This would be a new
        #                    behavior when 1:1 import happens, special case of "import and
        #                    slot under" for a single plan.
        # integrate: false -> replace the local plan with the remote one. This is the current
        #                     behavior of "import a plan".

[psss]

This seems to be an important issue for the kernel folks as they would like to import multiple plans from their dedicated plan repository, @bgoncalv can provide some more details. Let's try to implement this in the near future. Proposing for 1.35.

[mkluson]

I also have a use case for this feature, and I am volunteering to test it during its development.