Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a mage target that packages elastic-agent using elastic-agent-core DRA #4403

Merged
merged 15 commits into from
Apr 10, 2024

Conversation

pchila
Copy link
Member

@pchila pchila commented Mar 12, 2024

What does this PR do?

This PR adds 2 new mage targets:

  • FetchLatestAgentCoreStagingDRA : fetches the latest staging artifact for agent core for a given branch passed as argument
  • PackageUsingDRA: start the packaging process but instead of compiling the agent binaries using Crossbuild, downloads the elastic-agent-core DRAs specified in the manifest passed using ManifestURL environment variable.
    This target assumes that downloadManifest has been called with the same manifest

The algorithm is pretty straightforward: packageAgent now receives an mg.F parameter to run as dependency to obtain the agent binaries:

  • in the existing Package target we pass GolangCrossbuild as the dependency to run
  • in the new PackageUsingDRA we pass mg.F(useDRAAgentBinaryForPackage, manifestUrl) to retrieve the binaries from the elastic-agent-core DRA packages specified in the manifest

There is also some refactoring to break up the packageAgent function in more manageable pieces: collectPackageDependencies and flattenDependencies have been extracted from the original function body.

Why is it important?

When repackaging agent for early releases we want to specify what agent binaries to use via the manifest instead of rebuilding from the HEAD of a give release branch.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • [ ] I have added an integration test or an E2E test

Author's Checklist

  • [ ]

How to test this PR locally

To test the packaging functionality we can run mage packageUsingDRA specifiying a manifest and a version, for example:

mage clean && AGENT_DROP_PATH="build/elastic-agent-drop" DRA_VERSION="8.13.0+build202403190000" AGENT_PACKAGE_VERSION="8.13.0+build202403190000" DRA_WORKFLOW="staging" ManifestURL="https://staging.elastic.co/independent-agent/8.13.0+build202403190101/agent-artifacts-8.13.0+build202403190101.json"  mage -v  downloadManifest packageUsingDRA

The dependencies and agent binaries will be downloaded according to what is specified in the manifest and packages will be created with the correct package version in build/distributions

To just fetch the agent-core binaries (useful for debugging scenarios) we call FetchLatestAgentCoreStagingDRA with a release branch as argument (we can have a look at what is available in the DRA report):

mage FetchLatestAgentCoreStagingDRA 8.13

Related issues

Use cases

Screenshots

Logs

Questions to ask yourself

  • How are we going to support this in production?
  • How are we going to measure its adoption?
  • How are we going to debug this?
  • What are the metrics I should take care of?
  • ...

@mergify mergify bot assigned pchila Mar 12, 2024
Copy link
Contributor

mergify bot commented Mar 12, 2024

This pull request does not have a backport label. Could you fix it @pchila? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@pchila pchila force-pushed the add-mage-target-packageDRA branch 7 times, most recently from 62ecfea to 137f9c6 Compare March 19, 2024 07:08
@pchila pchila added enhancement New feature or request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Team:Elastic-Agent Label for the Agent team labels Mar 19, 2024
@pchila pchila force-pushed the add-mage-target-packageDRA branch from 2731879 to ca8aa74 Compare March 19, 2024 15:55
@pchila pchila force-pushed the add-mage-target-packageDRA branch 3 times, most recently from 2ecfcc0 to 354e2d7 Compare March 20, 2024 14:54
@pchila pchila marked this pull request as ready for review March 20, 2024 14:54
@pchila pchila requested a review from a team as a code owner March 20, 2024 14:54
@pchila pchila requested a review from AndersonQ March 20, 2024 14:54
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@pchila pchila requested a review from blakerouse March 20, 2024 14:55
Copy link
Collaborator

@dwhyrock dwhyrock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Love all the added verbose logging, btw.

@pchila
Copy link
Member Author

pchila commented Mar 21, 2024

buildkite package this

1 similar comment
@pchila
Copy link
Member Author

pchila commented Mar 21, 2024

buildkite package this

@pchila pchila force-pushed the add-mage-target-packageDRA branch from a518031 to 2e294bb Compare March 21, 2024 15:09
@pchila pchila force-pushed the add-mage-target-packageDRA branch from 3230d4a to 3ff4616 Compare April 8, 2024 17:55
@pchila pchila requested a review from AndersonQ April 8, 2024 17:56
@cmacknz
Copy link
Member

cmacknz commented Apr 8, 2024

Verified the MacOS artifact works now. Thanks!

When we merge this, it is going to make it so that snapshots and build candidates depend on the output of the elastic-agent-binary-dra buildkite pipeline that produces the elastic-agent-core artifact.

Looking at the pipeline https://buildkite.com/elastic/elastic-agent-binary-dra I don't think it runs often enough. Main seems to be on a daily schedule, and I'm not sure what triggers 8.13 as it is now several days behind the head of the 8.13 branch.

It looks like this job only takes 5m most of the time, as part of this PR can we change this agent core DRA step to execute for every commit? It is cheap and that will avoid the situation where someone forgets to kick it manually for a build candidate.

Looking at the buildkite pipeline definition it isn't obvious to me why it isn't already working this way for these branches. https://github.com/elastic/elastic-agent/blob/8.13/.buildkite/pipeline.elastic-agent-binary-dra.yml

@pchila
Copy link
Member Author

pchila commented Apr 9, 2024

Verified the MacOS artifact works now. Thanks!

When we merge this, it is going to make it so that snapshots and build candidates depend on the output of the elastic-agent-binary-dra buildkite pipeline that produces the elastic-agent-core artifact.

Looking at the pipeline https://buildkite.com/elastic/elastic-agent-binary-dra I don't think it runs often enough. Main seems to be on a daily schedule, and I'm not sure what triggers 8.13 as it is now several days behind the head of the 8.13 branch.

It looks like this job only takes 5m most of the time, as part of this PR can we change this agent core DRA step to execute for every commit? It is cheap and that will avoid the situation where someone forgets to kick it manually for a build candidate.

Looking at the buildkite pipeline definition it isn't obvious to me why it isn't already working this way for these branches. https://github.com/elastic/elastic-agent/blob/8.13/.buildkite/pipeline.elastic-agent-binary-dra.yml

There's nothing in the pipeline definition that would limit the elastic-agent-binary-dra from running on any release branch or main (we just have a regex identifiying \d+.\d+ branches or exact match with main).
You are right about the schedule as we have a daily on main in the buildkite settings directly.
image

I think that the reason why we don't get a build out of every commit is that the option has not been selected on the pipeline Github configuration Build branches option in the pic below
image

If we want to build on every commit on main and release branches I think we need to enable that option (not sure if we need to keep the schedule ?)

I would like a second opinion from our eng-prod team before making any changes though...

Edit: had an OOB chat with @alexsapran, in the end the most sensible thing seems to be having a scheduled buildkite job for each of the release branches. I will raise that modification in a separate PR and crossreference with this one.

Edit2: switched to building branches, PR: #4550

Copy link

@pchila pchila merged commit 87336db into elastic:main Apr 10, 2024
9 checks passed
@pchila pchila added backport-v8.13.0 Automated backport with mergify and removed backport-skip labels Apr 17, 2024
mergify bot pushed a commit that referenced this pull request Apr 17, 2024
…e DRA (#4403)

* Extract dependencies download to its own function

* Add PackageUsingDRA mage target

* filter elastic-agent-core artifacts by platform

* update buildkite package script with packageUsingDRA target

* Add sha512 validation for downloaded elastic-agent-core packages

* Override the elastic agent commit hash when packaging using DRA

* change Manifest URL env variable name

(cherry picked from commit 87336db)

# Conflicts:
#	dev-tools/mage/settings.go
#	magefile.go
pchila added a commit that referenced this pull request Apr 29, 2024
…e DRA (#4403)

* Extract dependencies download to its own function

* Add PackageUsingDRA mage target

* filter elastic-agent-core artifacts by platform

* update buildkite package script with packageUsingDRA target

* Add sha512 validation for downloaded elastic-agent-core packages

* Override the elastic agent commit hash when packaging using DRA

* change Manifest URL env variable name
pchila added a commit that referenced this pull request Apr 29, 2024
…e DRA (#4403)

* Extract dependencies download to its own function

* Add PackageUsingDRA mage target

* filter elastic-agent-core artifacts by platform

* update buildkite package script with packageUsingDRA target

* Add sha512 validation for downloaded elastic-agent-core packages

* Override the elastic agent commit hash when packaging using DRA

* change Manifest URL env variable name
pchila added a commit that referenced this pull request Apr 29, 2024
pchila added a commit that referenced this pull request Apr 29, 2024
…using elastic-agent-core DRA (#4589)

* Add a mage target that packages elastic-agent using elastic-agent-core DRA (#4403)

* Extract dependencies download to its own function

* Add PackageUsingDRA mage target

* filter elastic-agent-core artifacts by platform

* update buildkite package script with packageUsingDRA target

* Add sha512 validation for downloaded elastic-agent-core packages

* Override the elastic agent commit hash when packaging using DRA

* change Manifest URL env variable name

---------

Co-authored-by: Paolo Chilà <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v8.13.0 Automated backport with mergify enhancement New feature or request skip-changelog Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a mage target that packages the elastic-agent-core package instead of recompiling the agent
6 participants