Changes to registry stacks directory version names don't get picked up in delta test

[mike-hoang]

Which area this feature is related to?

/kind bug

Which area this bug is related to?

/area registry

Bug Summary

Describe the bug:

The Validate stacks github action for the registry repo will trigger on every open PR. It will find the change delta and only run the test suite against changed stacks.

However, if the stack contains multiple versions (ie. has a stack.yaml in the stack root) and the directory's version name has been changed, the delta will not be correctly picked up.

The delta is created with get_stacks.sh and will use a git diff to find the files changed. In the case where an existing version directory name is changed, the diff will look like this:

...
stacks/go/{1.0.2 => 1.1.0}/devfile.yaml           | 25 ++++++++++++++++++++++++-
stacks/go/{2.0.0 => 2.1.0}/devfile.yaml           | 25 ++++++++++++++++++++++++-
...

and the existing regex (stacks\/(.*)\/devfile\.yaml) won't match correctly.

To Reproduce:

Change a version directory in an existing multi version stack and open a PR to trigger the test.

Expected behavior

The changed version stack should be picked up in the get_stacks.sh

Additional context

Any workaround?

Force TEST_DELTA to be false to turn off delta testing.

Suggestion on how to fix the bug

Update the way changed_stacks are found in get_stacks.sh

[kadel]

This is still broken even after #176
For example, odo v3 tests on devfile/registry#177 should be failing. But the tests were not executed against added stack - https://github.com/devfile/registry/actions/runs/4669480312/jobs/8267987204?pr=177#step:9:80

[mike-hoang]

For example, odo v3 tests on devfile/registry#177 should be failing. But the tests were not executed against added stack - https://github.com/devfile/registry/actions/runs/4669480312/jobs/8267987204?pr=177#step:9:80

Yeah it looks like the stack did get picked up stackDirs=udi (which was the main issue with multi version stacks) but then it still says 0 tests were ran so it might be another problem in the tests.

I also tried running a full build with the delta off and it still results in a passing odo v3 test:
https://github.com/mike-hoang/registry/actions/runs/4680567043/jobs/8292073695#step:9:81

[mike-hoang]

Opened #1103 to track the problem with odo v3 testing