[8.13](backport #39126) Beats DRA pipeline #39198
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?s=60&v=4){kind=small}
* Packaging pipieline resource * Revetred agentbeat pipeline deletion * Cleanup * Test auditbeat packaging pipeline * Fix steps keys * Fix steps keys * Fix env vars * Fix env vars * Fix env vars * Unified artifacts dir * Implemented DRA steps * Test filebeat * Test matrix * Aligned artifacts directories * Aligned artifacts directories * Aligned artifacts directories * Aligned artifacts directories * Debug * Debug * Package all beats * Test DRA snapshot * Test DRA snapshot * Rename artifacts * fix dashboards artifacts * Cleanup * cleanup * No need to install msi tools * Apply suggestions from code review Co-authored-by: Dimitrios Liappis <[email protected]> * Extraced platforms variables * Cleanup * Added RUN_SNAPSHOT condition * Cleanup * Boolean dry-run * Boolean dry-run --------- Co-authored-by: Dimitrios Liappis <[email protected]> (cherry picked from commit c6444db) # Conflicts: # .buildkite/packaging.pipeline.yml
mergify
bot
added
backport
conflicts
|
Cherry-pick of c6444db has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
botelastic
bot
added
the
needs_team
|
This pull request doesn't have a |
💚 Build Succeeded
Expand to view the summary
Build stats
❕ Flaky test reportNo test was executed to be analysed. 🤖 GitHub commentsExpand to view the GitHub comments
To re-run your PR in the CI, just comment with:
|
This PR fixes a number of bugs from #39126
This commit fixes the error "interpolating remaining fields: Expected identifier to start with a letter, got /"
This commit fixes the release-manager error > # > java.io.FileNotFoundException: /artifacts/build/distributions/... (Permission denied) by adjusting the permissions of parent directories. Additionally there are a few logging enhancements to improve debugging, as well as the introduction of a new optional `DRA_BRANCH` environment variable, to help running and debugging via PRs. Relates elastic/ingest-dev#3095 --------- Signed-off-by: Alexandros Sapranidis <[email protected]> Co-authored-by: Dimitrios Liappis <[email protected]>
…39217) Added DEV flag to snapshot builds. Specified the workflow for prepare-release-manager.sh Relates: elastic/ingest-dev#3095
|
This pull request has not been merged yet. Could you please review and merge it @pazone? 🙏 |
As things are now we allow parallel builds on the packaging pipeline, which could result in out of order artifacts (depending on which one takes longer to finish). This commit implements two "queues" (snapshot/staging) to ensure ordered builds of DRA artifacts. Relates elastic/ingest-dev#3095
The DRA staging release is failing because the Buildkite step isn't capturing the right artifacts. This commit fixes the issue by adjusting the artifact_paths to match the other steps.
* Fix the annotating for snapshot and staging This commit adds the `--append` flag to the buildkite-annotate so that when it is called by the snapshot and staging steps, it will not overwrite the other annotation. Signed-off-by: Alexandros Sapranidis <[email protected]>
This was referenced May 1, 2024
PR #39293 introduced one concurrency queue per staging/snapshot but this slows down unnecessarily concurrent DRA builds for main and other release branches. This commit makes the concurrency gates (additionally) specific per branch.
dliappis
added a commit
to dliappis/beats
that referenced
this pull request
May 3, 2024
Now that DRA support via Buildkite has also been backported to 8.13[^1] this commit widens the whitelisted branches that can trigger Buildkite DRA builds to include all 8.x branches. Relates: elastic/ingest-dev#3095 [^1]: elastic#39198
dliappis
added a commit
that referenced
this pull request
May 3, 2024
Now that DRA support via Buildkite has also been backported to 8.13[^1] this commit widens the whitelisted branches that can trigger Buildkite DRA builds to include all 8.x branches. Relates: elastic/ingest-dev#3095 [^1]: #39198
mergify bot
pushed a commit
that referenced
this pull request
May 3, 2024
Now that DRA support via Buildkite has also been backported to 8.13[^1] this commit widens the whitelisted branches that can trigger Buildkite DRA builds to include all 8.x branches. Relates: https://github.com/elastic/ingest-dev/issues/3095 [^1]: #39198 (cherry picked from commit 7e203ad)
mergify bot
pushed a commit
that referenced
this pull request
May 3, 2024
Now that DRA support via Buildkite has also been backported to 8.13[^1] this commit widens the whitelisted branches that can trigger Buildkite DRA builds to include all 8.x branches. Relates: https://github.com/elastic/ingest-dev/issues/3095 [^1]: #39198 (cherry picked from commit 7e203ad)
mergify bot
pushed a commit
that referenced
this pull request
May 3, 2024
Now that DRA support via Buildkite has also been backported to 8.13[^1] this commit widens the whitelisted branches that can trigger Buildkite DRA builds to include all 8.x branches. Relates: https://github.com/elastic/ingest-dev/issues/3095 [^1]: #39198 (cherry picked from commit 7e203ad)
dliappis
added a commit
that referenced
this pull request
May 3, 2024
Now that DRA support via Buildkite has also been backported to 8.13[^1] this commit widens the whitelisted branches that can trigger Buildkite DRA builds to include all 8.x branches. Relates: https://github.com/elastic/ingest-dev/issues/3095 [^1]: #39198 (cherry picked from commit 7e203ad) Co-authored-by: Dimitrios Liappis <[email protected]>
dliappis
added a commit
that referenced
this pull request
May 3, 2024
Now that DRA support via Buildkite has also been backported to 8.13[^1] this commit widens the whitelisted branches that can trigger Buildkite DRA builds to include all 8.x branches. Relates: https://github.com/elastic/ingest-dev/issues/3095 [^1]: #39198 (cherry picked from commit 7e203ad) Co-authored-by: Dimitrios Liappis <[email protected]>
dliappis
added a commit
that referenced
this pull request
May 3, 2024
Now that DRA support via Buildkite has also been backported to 8.13[^1] this commit widens the whitelisted branches that can trigger Buildkite DRA builds to include all 8.x branches. Relates: https://github.com/elastic/ingest-dev/issues/3095 [^1]: #39198 (cherry picked from commit 7e203ad) Co-authored-by: Dimitrios Liappis <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Beats Packaging pipeline
Migrates packaging.groovy to buildkite.
Buildkite packaging pipeline is used to build and publish the packages for the Beats. The pipeline is triggered by a commit to the
mainor release branches.The pipeline definition is located in the
.buildkite/packaging.pipeline.ymlTriggers
Staging packaging DRA is triggered for the
mainand release branches.Snapshot can be triggered for any branch by the
/packagecomment in the PR. The release-manager dry-run will be used for PR builds.Pipeline steps
Beats dashboards
Generates
build/distributions/dependencies.csvand adds it to thebeats-dashboardsartifact. Thedependencies.csvis required by the release-manager configurationPackaging snapshot/staging
mage package, mage ironbank)build/distributions/<beat>/directory and adds it as an artifact. Where<beat>is the name of the beatbuild/distributions/<beat>/directory, where<beat>is the name of the beat. For example,auditbeat, notx-pack/auditbeatDRA publish
Downloads the artifacts from the
packaging snapshot/stagingstep and publishes them to the Elastic DRA registry.Snapshot publishing example (Dry run)
Beats-tester and e2e tests are not triggered
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs
This is an automatic backport of pull request #39126 done by [Mergify](https://mergify.com).