Allow upgrade action to signal retry #1887

michel-laterman · 2022-09-20T22:12:23Z

What is the problem this PR solves?

Allow agent to retry upgrade operations that have failed.

How does this PR solve the problem?

Allow the ack of an upgrade action to set the upgrade status to retrying.
The event associated with the upgrade action should have a non-empty error attribute, and a payload attribute that is a json object with retry: bool and retry_attempt: int attributes.

How to test this PR locally

TODO

Checklist

I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Relates Auto-retry upgrades on the edge elastic-agent#778
Agent PR: [v2] Add upgrade action retry elastic-agent#1219

Allow the ack of an upgrade action to set the upgrade status to retrying.

elasticmachine · 2022-09-20T22:16:34Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-10-24T16:23:54.716+0000
Duration: 12 min 31 sec

Test stats 🧪

Test	Results
Failed	0
Passed	364
Skipped	1
Total	365

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

internal/pkg/api/handleAck.go

mergify · 2022-09-29T12:34:43Z

This pull request is now in conflicts. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b upgrade-retry upstream/upgrade-retry
git merge upstream/main
git push upstream upgrade-retry

juliaElastic

Code LGTM

mergify · 2022-10-17T19:15:43Z

This pull request is now in conflicts. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b upgrade-retry upstream/upgrade-retry
git merge upstream/main
git push upstream upgrade-retry

michalpristas · 2022-10-18T06:20:08Z

internal/pkg/api/handleAck.go

+			zlog.Info().Int("retry_attempt", pl.Attempt).Msg("marking agent upgrade as retrying")
+			doc[dl.FieldUpgradeStatus] = "retrying" // TODO should we also change FieldUpgradedAt and FieldUpgradeStated at?
+		} else {
+			zlog.Info().Int("retry_attempt", pl.Attempt).Msg("marking agent upgrade as failed")


how will we handle stale agent (older versions without retry in ack)? is it expected for them not to retry?
i'd say yes, just want to double check with your intentions.

FieldUpgradeStartedAt and FieldUpgradedAt is used in Fleet UI side to set status Upgrading
i wonder if we should mark FieldUpgradedAt in case of failed retry as it was not upgraded so far.

Yeah, a pre v8.6 agent that reports an error will set FieldUpgradeStatus to failed. However I'm not sure if it is able to send that ack as I don't think it could actually ack a failure.

If it sends a successful upgrade ack FieldUpgradeStatus will be set to nil and FieldUpgradedAt is set to the current time (keeping the current behaviour)

@juliaElastic should we be setting FieldUpgradedAt and clearing FieldUpgradeStartedAt in failure cases, or should we keep these attributes to their original values?

Currently there is no Failed agent status. So the agent can either remain in Updating or move back to Healthy status. The error should be captured in action results similar to other action failures.
So I think the agent can move back to Healthy state if the upgrade failed (supposing nothing changed for the agent) - this means setting FieldUpgradeStartedAt to nil and leaving FieldUpgradedAt at the previous value.

juliaElastic · 2022-10-24T15:04:09Z

internal/pkg/api/handleAck.go

-		dl.FieldUpgradeStatus:    nil,
-		dl.FieldUpgradedAt:       now,
-	}
+	doc := bulk.UpdateFields{}


what does an empty UpdateFields do?

I think it's a nop

should we remove it then?

internal/pkg/api/handleAck.go

michalpristas

Lgtm now

* [Automation] Update elastic stack version to 8.5.0-6b9f92c0 for testing (#1756) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-0616acda for testing (#1760) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-dd6f2bb0 for testing (#1765) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-feb644de for testing (#1768) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-7783a03c for testing (#1776) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-17b8a62d for testing (#1780) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-9aed3b11 for testing (#1784) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-440e0896 for testing (#1788) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-fedc3e60 for testing (#1791) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-b5001a6d for testing (#1795) Co-authored-by: apmmachine <[email protected]> * ci: move to fleet-ci (#1199) * Fic path to the packaging (#1806) * Fix gcs credentials for packaging (#1807) * [Automation] Update elastic stack version to 8.5.0-de69302b for testing (#1822) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-1bd77fc1 for testing (#1826) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-167dfc80 for testing (#1831) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-6b7dda2d for testing (#1835) Co-authored-by: apmmachine <[email protected]> * Allow multiple ES outputs as long as they are the same ES (#1684) * add 'outputs' field to the ES agent schema to store the API key data and permission hash for each ES output * add output name to API key metadata * add v8.5 migration to migration.go * add migration docs and improve logging * group migration functions per version * [Automation] Update elastic stack version to 8.5.0-4140365c for testing (#1837) Co-authored-by: apmmachine <[email protected]> * updating upgrade_status: completed (#1833) * updating upgrade_status: completed * updated schema.json and regenerated schema.go * updated license headers * Fix v8.5.0 migration painless script (#1839) * fix v8.5.0 migration painless script * [Automation] Update elastic stack version to 8.5.0-8e906f9f for testing (#1843) Co-authored-by: apmmachine <[email protected]> * ci: rename dra staging for release dra release staging (#1840) * Remove events from agent checkin body. (#1842) Remove the events attribute from the agent checkin body. Note that removal of the attribute will not stop the server from issuing a 400 if the response body is too long. The removal is so that the checkin code on the fleet-server and agent remain comparable. Co-authored-by: Blake Rouse <[email protected]> * [Automation] Update elastic stack version to 8.5.0-589a4a10 for testing (#1852) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-37418cf3 for testing (#1855) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-fcf3d4c2 for testing (#1862) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-c7913db3 for testing (#1868) Co-authored-by: apmmachine <[email protected]> * Add error detail to catch-all HTTP response (#1854) * Make authc log debug and add cache hit field (#1870) * Document Go 1.18 certificate change in changelog. (#1871) * Revert "Fix v8.5.0 migration painless script" (#1878) * Revert "Fix v8.5.0 migration painless script (#1839)" This reverts commit de5d74b. * Revert "Allow multiple ES outputs as long as they are the same ES (#1684)" This reverts commit 63fdcbf. * [Automation] Update elastic stack version to 8.5.0-56d2c52d for testing (#1880) Co-authored-by: apmmachine <[email protected]> * Bulk API Keys update (#1779) Bulk API Keys update (#1779) * Fix and reintroduce "Allow multiple ES outputs as long as they are the same ES" (#1879) * Revert "Revert "Fix v8.5.0 migration painless script" (#1878)" This reverts commit ef9ca2b. * Revert "Revert "Allow multiple ES outputs as long as they are the same ES (#1684)"" This reverts commit bb696ac. * avoid new API keys being marked for invalidation Co-authored-by: Michal Pristas <[email protected]> He fixed the merge conflicts after Bulk API Keys update (#1779), commit 46ac14b, got merged * [Automation] Update elastic stack version to 8.5.0-7dc445a0 for testing (#1888) Co-authored-by: apmmachine <[email protected]> * Update pre-sets limits to avoid overlap. (#1891) Update file max limits and env_defaults_test.go running make defaults to generate the new one * [Release] add-backport-next (#1892) * Bump version to 8.6.0 (#1895) * Catch error in waitBulkAction. Add bulk.WithRetryOnConflict(3) in multiple places. (#1896) * Catch error in waitBulkAction. Add bulk.WithRetryOnConflict(3) in multiple places. * Add changelog entry. * Update CHANGELOG.next.asciidoc Co-authored-by: Craig MacKenzie <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> * Update apikey.cache_hit log field name to match convention (#1900) * [Automation] Update elastic stack version to 8.6.0-21651da3 for testing (#1908) Co-authored-by: apmmachine <[email protected]> * LoadLimits does not override existing values (#1912) Fleet-server will use any specified cache or server limit values over whatever is returned by the default/agent number loader. For example, if A max body size is specifically set to a value such as 5MB, and the default returned by the LoadLimits is 1MB, the 5MB value is used. * [Automation] Update elastic stack version to 8.6.0-326f84b0 for testing (#1916) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-df00693f for testing (#1925) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-a2f4f140 for testing (#1928) Co-authored-by: apmmachine <[email protected]> * Revert "updating upgrade_status: completed (#1833)" (#1920) * Revert "updating upgrade_status: completed (#1833)" This reverts commit 23be42a. * Leaving in upgrade_status field for retry functionality * Storing checkin message in last_checkin_message (#1932) * Storing checkin message in last_checkin_message * added changelog * fixed tests * Unique limiters for each API listener (#1904) * Unique limiters for each API listener Refactor the limit.Limiter so it can wrap the separate API httprouter endpoints. Limiter.WrapX() calls take the handler and stats incrementer for metrics/error counting. api.Run() replaced with Router.Run(), which will generate an httprouter for each listener in order to be able to associate the httprouter with a unique Limiter. * Add listener address labeled logs to limiter * Review feedback * Apply suggestions from code review Co-authored-by: Anderson Queiroz <[email protected]> * review feedback * fix import * Fix test Co-authored-by: Anderson Queiroz <[email protected]> * Cleanup cmd/fleet/main.go (#1886) * Replace cache.Config with config.Cache * Move server setup from cmd/fleet to new pkg/server * Move constants * Fix imports and integration tests * fix linter * [Automation] Update elastic stack version to 8.6.0-158a13db for testing (#1938) Co-authored-by: apmmachine <[email protected]> * [8.6](forwardport) Add extra protection against accessing null fields to 8.5 migration (#1921) (#1926) * [Automation] Update elastic stack version to 8.6.0-aea1c645 for testing (#1942) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-0fca2953 for testing (#1948) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-e4c15f15 for testing (#1954) Co-authored-by: apmmachine <[email protected]> * Conditional log level for api key read (#1946) Conditional log level for api key read (#1946) * Updated migration query to match items with deprecated field present (#1959) Co-authored-by: Anderson Queiroz <[email protected]> * Fix fleet.migration.total log key overlap (#1951) Co-authored-by: Anderson Queiroz <[email protected]> * [Automation] Update elastic stack version to 8.6.0-d939cfde for testing (#1964) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-7c9f25a9 for testing (#1969) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-c49fac70 for testing (#1976) Co-authored-by: apmmachine <[email protected]> * Update to Go 1.18.7. (#1978) * [Automation] Update elastic stack version to 8.6.0-5a8d757d for testing (#1981) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-40086bc7 for testing (#1987) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-233dc5d4 for testing (#1990) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-54a302f0 for testing (#1995) Co-authored-by: apmmachine <[email protected]> * Don't send POLICY_CHANGE actions retrieved from index to agent. (#1963) * Don't send POLICY_CHANGE actions retrieved from index to agent. The fleet-server should not send any policy change actions that are written to the actions index to an agent on checkin. The server will remove these actions in the convert method and emit a warning message. The ack token that is used is not altered in this case. Policy change actions are dynamically generated by the fleet-server when it detects that the agent is not running an up to date version of the policy. * move filtering to its own method * Fix linter, tests, fix file name * [Automation] Update elastic stack version to 8.6.0-cae815eb for testing (#2000) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-6545f2df for testing (#2005) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-055acc83 for testing (#2011) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-baf193e8 for testing (#2016) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-22d60ec9 for testing (#2020) Co-authored-by: apmmachine <[email protected]> * Allow upgrade action to signal retry (#1887) * Allow upgrade action to signal retry Allow the ack of an upgrade action to set the upgrade status to retrying. * fix tests set failed state * Fix broken test * nil upgrade status by default * Set agent to healthy in case of upgrade failure * fix upgrade fields * Fix tests * [Automation] Update elastic stack version to 8.6.0-b8b35931 for testing (#2024) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-a892f234 for testing (#2030) Co-authored-by: apmmachine <[email protected]> * [Automation] Add GH action to add issues to ingest board Issues in this repo labeled with `Team:Fleet` will be added to the ingest board automatically w/ the `Fleet Server` area. * Update add-issues-to-ingest-board.yml * [Automation] Update elastic stack version to 8.6.0-89d224d2 for testing (#2034) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-949a38d2 for testing (#2039) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-26dc1164 for testing (#2045) Co-authored-by: apmmachine <[email protected]> * Add active filter for enrollment key queries. (#2044) * Add active filter for enrollment key queries. Add an active: true filter to enrollment key queries. This allows fleet-server to handle cases where there may be 10+ inactive keys associated with a policy. * review feedback * fix linter * fix tests * Fix test cases * [Automation] Update elastic stack version to 8.6.0-4765d2b0 for testing (#2048) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-8a615646 for testing (#2050) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-3f5f98b7 for testing (#2051) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-f20b7179 for testing (#2056) Co-authored-by: apmmachine <[email protected]> * Run mod tidy. * Run make notice. * Fix intergration tests. * Run go mod tidy and make notice. * Fix path to fleet-server.yml in integration test. * Fix race condition. * Fix try 2. * Fix race. * Fix race try 2. Co-authored-by: apmmachine <[email protected]> Co-authored-by: apmmachine <[email protected]> Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Anderson Queiroz <[email protected]> Co-authored-by: Julia Bardi <[email protected]> Co-authored-by: Michel Laterman <[email protected]> Co-authored-by: Josh Dover <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> Co-authored-by: Michal Pristas <[email protected]> Co-authored-by: Julien Lind <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Anderson Queiroz <[email protected]> Co-authored-by: Kyle Pollich <[email protected]>

* Support for Elastic Agent V2 status (#1747) * Support for Elastic Agent V2 status * Make 'make check-ci' happy * Add a check that 'components' is valid array * Rename variable to better reflect it's meaning * [v2] Switch to Elastic Agent v2 control protocol (#1751) * Switch to new client.V2 for communication with Elastic Agent. * Fix tests. * Fix integration tests. * Update go.sum. * Fix some lint issues. * Fix panic with agentInfo. * Fix panic in logger reconfigure. * Fixes for switching units. * updated version (#2014) * Update the elastic-agent-client to latest version. (#2061) * [v2] Merge main as of Nov 7 (#2062) * [Automation] Update elastic stack version to 8.5.0-6b9f92c0 for testing (#1756) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-0616acda for testing (#1760) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-dd6f2bb0 for testing (#1765) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-feb644de for testing (#1768) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-7783a03c for testing (#1776) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-17b8a62d for testing (#1780) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-9aed3b11 for testing (#1784) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-440e0896 for testing (#1788) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-fedc3e60 for testing (#1791) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-b5001a6d for testing (#1795) Co-authored-by: apmmachine <[email protected]> * ci: move to fleet-ci (#1199) * Fic path to the packaging (#1806) * Fix gcs credentials for packaging (#1807) * [Automation] Update elastic stack version to 8.5.0-de69302b for testing (#1822) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-1bd77fc1 for testing (#1826) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-167dfc80 for testing (#1831) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-6b7dda2d for testing (#1835) Co-authored-by: apmmachine <[email protected]> * Allow multiple ES outputs as long as they are the same ES (#1684) * add 'outputs' field to the ES agent schema to store the API key data and permission hash for each ES output * add output name to API key metadata * add v8.5 migration to migration.go * add migration docs and improve logging * group migration functions per version * [Automation] Update elastic stack version to 8.5.0-4140365c for testing (#1837) Co-authored-by: apmmachine <[email protected]> * updating upgrade_status: completed (#1833) * updating upgrade_status: completed * updated schema.json and regenerated schema.go * updated license headers * Fix v8.5.0 migration painless script (#1839) * fix v8.5.0 migration painless script * [Automation] Update elastic stack version to 8.5.0-8e906f9f for testing (#1843) Co-authored-by: apmmachine <[email protected]> * ci: rename dra staging for release dra release staging (#1840) * Remove events from agent checkin body. (#1842) Remove the events attribute from the agent checkin body. Note that removal of the attribute will not stop the server from issuing a 400 if the response body is too long. The removal is so that the checkin code on the fleet-server and agent remain comparable. Co-authored-by: Blake Rouse <[email protected]> * [Automation] Update elastic stack version to 8.5.0-589a4a10 for testing (#1852) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-37418cf3 for testing (#1855) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-fcf3d4c2 for testing (#1862) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.5.0-c7913db3 for testing (#1868) Co-authored-by: apmmachine <[email protected]> * Add error detail to catch-all HTTP response (#1854) * Make authc log debug and add cache hit field (#1870) * Document Go 1.18 certificate change in changelog. (#1871) * Revert "Fix v8.5.0 migration painless script" (#1878) * Revert "Fix v8.5.0 migration painless script (#1839)" This reverts commit de5d74b. * Revert "Allow multiple ES outputs as long as they are the same ES (#1684)" This reverts commit 63fdcbf. * [Automation] Update elastic stack version to 8.5.0-56d2c52d for testing (#1880) Co-authored-by: apmmachine <[email protected]> * Bulk API Keys update (#1779) Bulk API Keys update (#1779) * Fix and reintroduce "Allow multiple ES outputs as long as they are the same ES" (#1879) * Revert "Revert "Fix v8.5.0 migration painless script" (#1878)" This reverts commit ef9ca2b. * Revert "Revert "Allow multiple ES outputs as long as they are the same ES (#1684)"" This reverts commit bb696ac. * avoid new API keys being marked for invalidation Co-authored-by: Michal Pristas <[email protected]> He fixed the merge conflicts after Bulk API Keys update (#1779), commit 46ac14b, got merged * [Automation] Update elastic stack version to 8.5.0-7dc445a0 for testing (#1888) Co-authored-by: apmmachine <[email protected]> * Update pre-sets limits to avoid overlap. (#1891) Update file max limits and env_defaults_test.go running make defaults to generate the new one * [Release] add-backport-next (#1892) * Bump version to 8.6.0 (#1895) * Catch error in waitBulkAction. Add bulk.WithRetryOnConflict(3) in multiple places. (#1896) * Catch error in waitBulkAction. Add bulk.WithRetryOnConflict(3) in multiple places. * Add changelog entry. * Update CHANGELOG.next.asciidoc Co-authored-by: Craig MacKenzie <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> * Update apikey.cache_hit log field name to match convention (#1900) * [Automation] Update elastic stack version to 8.6.0-21651da3 for testing (#1908) Co-authored-by: apmmachine <[email protected]> * LoadLimits does not override existing values (#1912) Fleet-server will use any specified cache or server limit values over whatever is returned by the default/agent number loader. For example, if A max body size is specifically set to a value such as 5MB, and the default returned by the LoadLimits is 1MB, the 5MB value is used. * [Automation] Update elastic stack version to 8.6.0-326f84b0 for testing (#1916) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-df00693f for testing (#1925) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-a2f4f140 for testing (#1928) Co-authored-by: apmmachine <[email protected]> * Revert "updating upgrade_status: completed (#1833)" (#1920) * Revert "updating upgrade_status: completed (#1833)" This reverts commit 23be42a. * Leaving in upgrade_status field for retry functionality * Storing checkin message in last_checkin_message (#1932) * Storing checkin message in last_checkin_message * added changelog * fixed tests * Unique limiters for each API listener (#1904) * Unique limiters for each API listener Refactor the limit.Limiter so it can wrap the separate API httprouter endpoints. Limiter.WrapX() calls take the handler and stats incrementer for metrics/error counting. api.Run() replaced with Router.Run(), which will generate an httprouter for each listener in order to be able to associate the httprouter with a unique Limiter. * Add listener address labeled logs to limiter * Review feedback * Apply suggestions from code review Co-authored-by: Anderson Queiroz <[email protected]> * review feedback * fix import * Fix test Co-authored-by: Anderson Queiroz <[email protected]> * Cleanup cmd/fleet/main.go (#1886) * Replace cache.Config with config.Cache * Move server setup from cmd/fleet to new pkg/server * Move constants * Fix imports and integration tests * fix linter * [Automation] Update elastic stack version to 8.6.0-158a13db for testing (#1938) Co-authored-by: apmmachine <[email protected]> * [8.6](forwardport) Add extra protection against accessing null fields to 8.5 migration (#1921) (#1926) * [Automation] Update elastic stack version to 8.6.0-aea1c645 for testing (#1942) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-0fca2953 for testing (#1948) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-e4c15f15 for testing (#1954) Co-authored-by: apmmachine <[email protected]> * Conditional log level for api key read (#1946) Conditional log level for api key read (#1946) * Updated migration query to match items with deprecated field present (#1959) Co-authored-by: Anderson Queiroz <[email protected]> * Fix fleet.migration.total log key overlap (#1951) Co-authored-by: Anderson Queiroz <[email protected]> * [Automation] Update elastic stack version to 8.6.0-d939cfde for testing (#1964) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-7c9f25a9 for testing (#1969) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-c49fac70 for testing (#1976) Co-authored-by: apmmachine <[email protected]> * Update to Go 1.18.7. (#1978) * [Automation] Update elastic stack version to 8.6.0-5a8d757d for testing (#1981) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-40086bc7 for testing (#1987) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-233dc5d4 for testing (#1990) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-54a302f0 for testing (#1995) Co-authored-by: apmmachine <[email protected]> * Don't send POLICY_CHANGE actions retrieved from index to agent. (#1963) * Don't send POLICY_CHANGE actions retrieved from index to agent. The fleet-server should not send any policy change actions that are written to the actions index to an agent on checkin. The server will remove these actions in the convert method and emit a warning message. The ack token that is used is not altered in this case. Policy change actions are dynamically generated by the fleet-server when it detects that the agent is not running an up to date version of the policy. * move filtering to its own method * Fix linter, tests, fix file name * [Automation] Update elastic stack version to 8.6.0-cae815eb for testing (#2000) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-6545f2df for testing (#2005) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-055acc83 for testing (#2011) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-baf193e8 for testing (#2016) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-22d60ec9 for testing (#2020) Co-authored-by: apmmachine <[email protected]> * Allow upgrade action to signal retry (#1887) * Allow upgrade action to signal retry Allow the ack of an upgrade action to set the upgrade status to retrying. * fix tests set failed state * Fix broken test * nil upgrade status by default * Set agent to healthy in case of upgrade failure * fix upgrade fields * Fix tests * [Automation] Update elastic stack version to 8.6.0-b8b35931 for testing (#2024) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-a892f234 for testing (#2030) Co-authored-by: apmmachine <[email protected]> * [Automation] Add GH action to add issues to ingest board Issues in this repo labeled with `Team:Fleet` will be added to the ingest board automatically w/ the `Fleet Server` area. * Update add-issues-to-ingest-board.yml * [Automation] Update elastic stack version to 8.6.0-89d224d2 for testing (#2034) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-949a38d2 for testing (#2039) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-26dc1164 for testing (#2045) Co-authored-by: apmmachine <[email protected]> * Add active filter for enrollment key queries. (#2044) * Add active filter for enrollment key queries. Add an active: true filter to enrollment key queries. This allows fleet-server to handle cases where there may be 10+ inactive keys associated with a policy. * review feedback * fix linter * fix tests * Fix test cases * [Automation] Update elastic stack version to 8.6.0-4765d2b0 for testing (#2048) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-8a615646 for testing (#2050) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-3f5f98b7 for testing (#2051) Co-authored-by: apmmachine <[email protected]> * [Automation] Update elastic stack version to 8.6.0-f20b7179 for testing (#2056) Co-authored-by: apmmachine <[email protected]> * Run mod tidy. * Run make notice. * Fix intergration tests. * Run go mod tidy and make notice. * Fix path to fleet-server.yml in integration test. * Fix race condition. * Fix try 2. * Fix race. * Fix race try 2. Co-authored-by: apmmachine <[email protected]> Co-authored-by: apmmachine <[email protected]> Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Anderson Queiroz <[email protected]> Co-authored-by: Julia Bardi <[email protected]> Co-authored-by: Michel Laterman <[email protected]> Co-authored-by: Josh Dover <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> Co-authored-by: Michal Pristas <[email protected]> Co-authored-by: Julien Lind <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Anderson Queiroz <[email protected]> Co-authored-by: Kyle Pollich <[email protected]> Co-authored-by: Aleksandr Maus <[email protected]> Co-authored-by: Michal Pristas <[email protected]> Co-authored-by: apmmachine <[email protected]> Co-authored-by: apmmachine <[email protected]> Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Anderson Queiroz <[email protected]> Co-authored-by: Julia Bardi <[email protected]> Co-authored-by: Michel Laterman <[email protected]> Co-authored-by: Josh Dover <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> Co-authored-by: Julien Lind <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Anderson Queiroz <[email protected]> Co-authored-by: Kyle Pollich <[email protected]>

Allow upgrade action to signal retry

1bb5966

Allow the ack of an upgrade action to set the upgrade status to retrying.

michel-laterman added enhancement New feature or request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Sep 20, 2022

michel-laterman requested review from michalpristas, blakerouse and juliaElastic September 20, 2022 22:12

michel-laterman requested a review from a team as a code owner September 20, 2022 22:12

juliaElastic reviewed Sep 21, 2022

View reviewed changes

internal/pkg/api/handleAck.go Outdated Show resolved Hide resolved

juliaElastic mentioned this pull request Sep 27, 2022

Revert "updating upgrade_status: completed (#1833)" #1920

Merged

5 tasks

michel-laterman added 4 commits October 5, 2022 10:35

Merge branch 'main' into upgrade-retry

f0af07d

fix tests set failed state

f29d495

Merge remote-tracking branch 'origin/main' into upgrade-retry

f21a7e1

Fix broken test

890362d

michel-laterman requested a review from juliaElastic October 12, 2022 16:39

juliaElastic approved these changes Oct 13, 2022

View reviewed changes

michel-laterman and others added 2 commits October 14, 2022 12:09

Merge branch 'main' into upgrade-retry

d521281

nil upgrade status by default

f34221a

Merge branch 'main' into upgrade-retry

7b0e161

michalpristas reviewed Oct 18, 2022

View reviewed changes

Set agent to healthy in case of upgrade failure

63e4904

juliaElastic reviewed Oct 24, 2022

View reviewed changes

internal/pkg/api/handleAck.go Outdated Show resolved Hide resolved

juliaElastic reviewed Oct 24, 2022

View reviewed changes

internal/pkg/api/handleAck.go Outdated Show resolved Hide resolved

michel-laterman added 2 commits October 24, 2022 08:35

fix upgrade fields

f386789

Fix tests

567c5d2

michalpristas approved these changes Oct 24, 2022

View reviewed changes

Merge branch 'main' into upgrade-retry

18204bc

michel-laterman merged commit 17d3681 into elastic:main Oct 24, 2022

michel-laterman deleted the upgrade-retry branch October 24, 2022 22:13

juliaElastic mentioned this pull request Apr 18, 2023

[Fleet] Agent gets stuck in the updating state if the upgrade action fails elastic/elastic-agent#2508

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow upgrade action to signal retry #1887

Allow upgrade action to signal retry #1887

michel-laterman commented Sep 20, 2022

elasticmachine commented Sep 20, 2022 •

edited

Loading

Build stats

Test stats 🧪

mergify bot commented Sep 29, 2022

juliaElastic left a comment

mergify bot commented Oct 17, 2022

michalpristas Oct 18, 2022

michalpristas Oct 18, 2022 •

edited

Loading

michel-laterman Oct 18, 2022

juliaElastic Oct 24, 2022

juliaElastic Oct 24, 2022

michel-laterman Oct 24, 2022

juliaElastic Oct 25, 2022 •

edited

Loading

michalpristas left a comment

Allow upgrade action to signal retry #1887

Allow upgrade action to signal retry #1887

Conversation

michel-laterman commented Sep 20, 2022

What is the problem this PR solves?

How does this PR solve the problem?

How to test this PR locally

Checklist

Related issues

elasticmachine commented Sep 20, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Sep 29, 2022

juliaElastic left a comment

Choose a reason for hiding this comment

mergify bot commented Oct 17, 2022

michalpristas Oct 18, 2022

Choose a reason for hiding this comment

michalpristas Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

michel-laterman Oct 18, 2022

Choose a reason for hiding this comment

juliaElastic Oct 24, 2022

Choose a reason for hiding this comment

juliaElastic Oct 24, 2022

Choose a reason for hiding this comment

michel-laterman Oct 24, 2022

Choose a reason for hiding this comment

juliaElastic Oct 25, 2022 • edited Loading

Choose a reason for hiding this comment

michalpristas left a comment

Choose a reason for hiding this comment

elasticmachine commented Sep 20, 2022 •

edited

Loading

michalpristas Oct 18, 2022 •

edited

Loading

juliaElastic Oct 25, 2022 •

edited

Loading