[CI] Enable packaging with Pipeline 2.0

[v1v]

What does this PR do?

Proposal to run the packaging in parallel on a PR basis.

See the follow ups section regarding what are the different areas that could be done in the future.

Why is it important?

Split the packaging per beat in order to validate them on a PR basis.

Tasks

• Test
• E2E disabled at the moment but already implemented
• publish packages and docker images

Follow ups

• Enable e2e
• ARM support if possible
• MAC support if possible
• Deprecate the existing packaging pipeline in favour of this approach

Issues

Requires #23960
Caused elastic/e2e-testing#705

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #23854 updated

• Start Time: 2021-02-10T18:11:18.191+0000

• Duration: 57 min 33 sec

• Commit: 024b732

Test stats 🧪

Test	Results
Failed	0
Passed	45794
Skipped	4762
Total	50556

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	45794
Skipped	4762
Total	50556

[v1v]

Let's see what we do with this

[v1v]

This is a new step, though there is no build in place yet but the packaging itself.

[urso]

Does this change mean the pr-merge will only be green if packaging and E2E tests did succeed? Especially E2E tests are quite red recently and flakyness in Beats tests has increased again.

With this change, will we run all E2E tests? E.g. if I modify filebeat, will metricbeat packaing or auditbeat E2E test be able to mark my PR as failed?

Regarding running packaging automatically. I would expect that packaging jobs are run when any magefile, Makfile, dev-tools or scripts directory is touched in a PR.

[v1v]

Does this change mean the pr-merge will only be green if packaging and E2E tests did succeed? Especially E2E tests are quite red recently and flakyness in Beats tests has increased again.

Exactly, if the concern is regarding the E2E flakiness, I think there were a couple of issues here:

• some regressions that were not detected since the master branch got broken. (See later on an example of this particular scenario)
• packages were not generated for x-pack for the last 3 days.

This particular approach will help to detect earlier any broken changes before any merges.

cc @mdelapenya as he can clarify the above.

With this change, will we run all E2E tests? E.g. if I modify filebeat, will metricbeat packaing or auditbeat E2E test be able to mark my PR as failed?

No, E2E will only run for those beats that got E2E to run and only if the changes are related to that particular beat. In other words, if you modify filebeat then only the packaging for filebeat will be executed, and if there are E2E for filebeat then it will be triggered afterwards, if no E2E then nothing will happen but the packaging.

Regarding running packaging automatically. I would expect that packaging jobs are run when any magefile, Makfile, dev-tools or scripts directory is touched in a PR.

Both systems will coexist and eventually we will move to this approach. The more couple the build/test/packaging/e2e stages are the sooner we can find regressions/bugs on a PR basis.

The current approach for every commit in the master branch, or any, follow the below steps:

• Build/Test/IngTest all the beats.
  • If the above passed then, it runs the Packaging and E2E stage for each beat.
  • If it does not pass, then no Packaging and E2E will be executed

If no packaging and E2E are executed after a few commits, then it means the E2E might not detect regressions and the git bisect between the last successful Packaging and E2E and the latest commit might not be easy to do so.

In order to help with the analysis, we want to shift left all the validations that run by default in any branch. And ensure those validations are synchronous, and move away from the perception that merging on a broken branch does not produce slowness to everyone.

In a nutshell, this process does nothing but validating the build/test/cross-compile/ing-testing/packaging/e2e in parallel for the specific beat. Therefore it's similar what we got already in place but with the addition of the packaging, the e2e will only happen if there is something configured for that particular beat.

[urso]

The current approach for every commit in the master branch, or any, follow the below steps:

Build/Test/IngTest all the beats.

• If the above passed then, it runs the Packaging and E2E stage for each beat.
• If it does not pass, then no Packaging and E2E will be executed

If no packaging and E2E are executed after a few commits, then it means the E2E might not detect regressions and the git bisect between the last successful Packaging and E2E and the latest commit might not be easy to do so.

+1 on running packaging, E2E and tests in parallel. When testing branches, no tests should be excluded.

• some regressions that were not detected since the master branch got broken. (See later on an example of this particular scenario)
• packages were not generated for x-pack for the last 3 days.

This particular approach will help to detect earlier any broken changes before any merges.

Again +1 for always running all CI on master and release/development branches.

This also shows shortcomming in alerting. We should be able to build from master and 7.x at any time. If builds go red, the sirens should turn on. No packages for 3 days without anyone investigating for 3 workdays is unacceptable. We introduced a Build Manager Rotation last week having Beats developers help in analyzing and fixing tests. Are we missing discoverability/alerting?

No, E2E will only run for those beats that got E2E to run and only if the changes are related to that particular beat. In other words, if you modify filebeat then only the packaging for filebeat will be executed, and if there are E2E for filebeat then it will be triggered afterwards, if no E2E then nothing will happen but the packaging.

+1 on testing only based on related changes. If I modify filebeat, will x-pack/filebeat packaging and E2E tests be run as well?
Changing metricbeat internals might also require auditbeat testsuite to be run.

In general there are serious issues in the process that need to be addressed. Not just during PRs but regular alerting as well, and us getting active. I see how the approach can help, but given that CI is not as stable as it should be yet, I worry about running everything on each PR potentially affecting the developer experience negatively, as we will need to run more things in parallel and increase the amount of sometimes unrelated tests that must pass.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)