Running tmt in GitHub Actions

[LecrisUT]

Had to play around with how to setup tmt to be executed within GitHub Actions and I wanted to share this. LecrisUT/CMakeExtraUtils#18. Basically the usecase here is to go beyond packaging tests and add some of the main project tests using tmt. For example in this project, testing against multiple versions of CMake

tmt limitations

• Currently using the default ubuntu-latest is not possible because of missing package-manager and potentially beakerlib. Some dependencies could be factored out and installed manually, but because of framework = beakerlib I don't think it is feasible at the moment. Works now with Extract "package manager" functionality into plugins #2557. Still requires the installation of beakerlib for many tests though
• No access to report data. This is unfortunate because /var/tmp/tmt is not that easy to upload as an artifact
  Workaround: Use report.how = junit, and one of the Github Junit test reporters. So far EnricoMi/publish-unit-test-result-action is the most compatible. Example run
• JUnit reporters are generally built to report the line of the source-code that triggered the error, but something like beakerlib would not have that metadata. It could instead point to the shell script code line instead. For other things like running pytest --cov that could be useful to forward the reports.

Next steps

Create dedicated GitHub actions to:

• setup-tmt: This one is pretty straightforward, and ideally it would not be tied to Fedora packaging
• run-tmt: Execute tmt run with appropriate steps injected:
  • provision.how = local: For more complex usage, the user should use testing-farm instead
  • context.initiator = GithubCI: Should be equivalent to human since the environment is very similar to running locally
  • report.how = junit: So far there is no other report that could give a navigable report. The html report would be nice, but they link to specific files that are not bundled
• tmt: combine setup-tmt and run-tmt for a simpler interface

[LecrisUT]

Update: I have a repo up for this: https://github.com/LecrisUT/tmt-actions. It's almost functional, but I would need some help debugging and designing.

[LecrisUT]

Progress report, it WORKS! thanks to #2557