Root cause analysis for the recent failures in the Fleet agent

[mdelapenya]

Possible culprit: elastic/beats#23452

• Beats CI checks are ALL passing for the PR: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/PR-23452/5/pipeline
  • No packaging job was triggered in the PR, as it was always skipped: https://beats-ci.elastic.co/job/Beats/job/packaging/view/change-requests/job/PR-23452/
    • In consequence, no E2E tests were executed
• Beats CI checks are ALL passing for the merge-to-master commit: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/master/88/pipeline
  • Not possible to analyse if the packaging job was triggered by the merge commit, as the first build in Jenkins history is for master branch in the packaging job is for the 144 build, which was triggered by the master build 90. Will look up Jenkins Stats service, as tonight it's down or restarting.

Possible fix: elastic/beats#23609

• Beats CI checks are ALL passing for the PR: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/PR-23609/5/pipeline
  • Packaging job was triggered in the PR. @michalpristas used the /package command in the buildID 2: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fpackaging/detail/PR-23609/2/pipeline
    • E2E tests were executed because of that: https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/master/198/pipeline
      • There were 3 broken tests: need to investigate why the "E2E Tests" GH check is not present in the PR.
        1. Re-enrolling the centos agent. Seems a potential real error
        2. Re-enrolling the debian agent. Seems a potential real error
        3. Ceph integration (which is considered flaky and it's been skipped in chore: skip ceph tests #635 and Investigate why Ceph tests are flaky, as it seems to fail with quite a frequency #607)
      • The test code fetches the proper snapshot artifact for the PR:

[2021-01-21T11:22:21.702Z] time="2021-01-21T11:22:21Z" level=debug msg="Media link found for the object" bucket=beats-ci-artifacts elapsedTime=54.952319ms medialink="https://storage.googleapis.com/download/storage/v1/b/beats-ci-artifacts/o/pull-requests%2Fpr-23609%2Felastic-agent%2Felastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz?generation=1611227685877128&alt=media" object=elastic-agent/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz prefix=pull-requests/pr-23609 retries=1

• Beats CI checks are ALL passing for the merge-to-master commit: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/master/102/pipeline
  • Packaging job was triggered by the merge commit: https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fpackaging/detail/master/158/pipeline
    • E2E tests were executed_ https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/master/204/pipeline need to investigate why the "E2E Tests" GH check is not present in the PR.
      • Noticed that the test code did not take the proper snapshot artifact. Instead of fetching it from the GCP bucket, it took the released one, which could be not up-to-date. This is caused by the BEATS_USE_CI_SNAPSHOTS input parameter in the E2E test pipeline, which was not properly filled in the packaging job: it was set to true only for PRs. We have sent a PR to fix that. [CI] chore: always use CI snapshots beats#23633

cc/ @elastic/observablt-robots @EricDavisX @ph

[EricDavisX]

as of this afternoon all tests are passing in master, again. very nicely done Manu, Michal and all involved. Should we close this out or is there other review we want to do, @mdelapenya ?

[mdelapenya]

Closing, as we fixed the issue, thanks!