[Fleet] Migration of saved objects do not trigger a policy update #193352
Assignees
Labels
bug
Fixes for quality problems that affect the customer experience
Team:Fleet
Team label for Observability Data Collection Fleet team
Comments
jen-huang
added
bug
|
Pinging @elastic/fleet (Team:Fleet) |
|
We have a mechanism to bump agent policies by increasing the FLEET_AGENT_POLICIES_SCHEMA_VERSION. We could potentially do something similar on package policy level. |
1 task
gergoabraham
added a commit
that referenced
this issue
Oct 18, 2024
…eployment issue mitigation (#196708) ## Summary closes elastic/security-team#10851 > [!note] >⚠️ needs to be included in v8.16 >⚠️ needs to be merged this week to avoid releasing #195797 on Serverless As backfilled package policies are not automatically redeployed (see #193352), this PR's goal is to provide quick mitigation in the following matters: - update default values in the descriptions of advanced options added in #195797, to harmonize with latest Endpoint changes (elastic/endpoint-dev#15109) - remove backfill/migration of those default values: - we should be _able_ to safely remove the backfills, as they have not yet been released to serverless. and, - we _should_ remove them to make sure that when we update the defaults in the future and apply the backfill, there will be a data change that could trigger policy re-deployment, in case data change is what the trigger will be in #193352. - example scenario of what could go wrong: - if we'd apply backfill now, the package won't be redeployed. - if the user does not touch it until the next release - no redeploy. - if #193352 is implemented and uses data comparison when running migrations - again, no redeploy because we already backfilled the data months before. - cc @ferullo @nfritts - hide banner describing event volume reduction (added in #195177, already released to serverless, but it is what it is) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 18, 2024
…eployment issue mitigation (elastic#196708) ## Summary closes elastic/security-team#10851 > [!note] >⚠️ needs to be included in v8.16 >⚠️ needs to be merged this week to avoid releasing elastic#195797 on Serverless As backfilled package policies are not automatically redeployed (see elastic#193352), this PR's goal is to provide quick mitigation in the following matters: - update default values in the descriptions of advanced options added in elastic#195797, to harmonize with latest Endpoint changes (elastic/endpoint-dev#15109) - remove backfill/migration of those default values: - we should be _able_ to safely remove the backfills, as they have not yet been released to serverless. and, - we _should_ remove them to make sure that when we update the defaults in the future and apply the backfill, there will be a data change that could trigger policy re-deployment, in case data change is what the trigger will be in elastic#193352. - example scenario of what could go wrong: - if we'd apply backfill now, the package won't be redeployed. - if the user does not touch it until the next release - no redeploy. - if elastic#193352 is implemented and uses data comparison when running migrations - again, no redeploy because we already backfilled the data months before. - cc @ferullo @nfritts - hide banner describing event volume reduction (added in elastic#195177, already released to serverless, but it is what it is) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 1429979)
gergoabraham
added a commit
to gergoabraham/kibana
that referenced
this issue
Oct 18, 2024
…eployment issue mitigation (elastic#196708) ## Summary closes elastic/security-team#10851 > [!note] >⚠️ needs to be included in v8.16 >⚠️ needs to be merged this week to avoid releasing elastic#195797 on Serverless As backfilled package policies are not automatically redeployed (see elastic#193352), this PR's goal is to provide quick mitigation in the following matters: - update default values in the descriptions of advanced options added in elastic#195797, to harmonize with latest Endpoint changes (elastic/endpoint-dev#15109) - remove backfill/migration of those default values: - we should be _able_ to safely remove the backfills, as they have not yet been released to serverless. and, - we _should_ remove them to make sure that when we update the defaults in the future and apply the backfill, there will be a data change that could trigger policy re-deployment, in case data change is what the trigger will be in elastic#193352. - example scenario of what could go wrong: - if we'd apply backfill now, the package won't be redeployed. - if the user does not touch it until the next release - no redeploy. - if elastic#193352 is implemented and uses data comparison when running migrations - again, no redeploy because we already backfilled the data months before. - cc @ferullo @nfritts - hide banner describing event volume reduction (added in elastic#195177, already released to serverless, but it is what it is) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 1429979)
kibanamachine
added a commit
that referenced
this issue
Oct 18, 2024
…cy re-deployment issue mitigation (#196708) (#196835) # Backport This will backport the following commits from `main` to `8.16`: - [[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)](#196708) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gergő Ábrahám","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-18T10:15:00Z","message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v9.0.0","Team:Defend Workflows","v8.16.0","backport:version"],"title":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation","number":196708,"url":"https://github.com/elastic/kibana/pull/196708","mergeCommit":{"message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0"}},"sourceBranch":"main","suggestedTargetBranches":["8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196708","number":196708,"mergeCommit":{"message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Gergő Ábrahám <[email protected]>
gergoabraham
added a commit
that referenced
this issue
Oct 18, 2024
…y re-deployment issue mitigation (#196708) (#196843) # Backport This will backport the following commits from `main` to `8.x`: - [[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)](#196708) <!--- Backport version: 9.6.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Gergő Ábrahám","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-18T10:15:00Z","message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v9.0.0","Team:Defend Workflows","v8.16.0","backport:version","v8.17.0"],"title":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation","number":196708,"url":"https://github.com/elastic/kibana/pull/196708","mergeCommit":{"message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196708","number":196708,"mergeCommit":{"message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/196835","number":196835,"state":"OPEN"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
1 task
|
Fixing this issue is more complex than I initially expected, due to handling spaces and multiple saved object types see the comments here: https://github.com/elastic/kibana/pull/200536/files |
juliaElastic
added a commit
to juliaElastic/kibana
that referenced
this issue
Nov 25, 2024
…200536) Closes elastic#193352 Update: Using a new SO field `bump_agent_policy_revision` in package policy type to mark package policies for update, this will trigger an agent policy revision bump. The feature supports both legacy and new package policy SO types, and queries policies from all spaces. To test, add a model version change to the package policy type and save. After Fleet setup is run, the agent policies using the package policies should be bumped and deployed. The same effect can be achieved by manually updating a package policy SO and loading Fleet UI to trigger setup. ``` '2': { changes: [ { type: 'data_backfill', backfillFn: (doc) => { return { attributes: { ...doc.attributes, bump_agent_policy_revision: true } }; }, }, ], }, curl -sk -XPOST --user fleet_superuser:password -H 'content-type:application/json' \ -H'x-elastic-product-origin:fleet' \ http://localhost:9200/.kibana_ingest/_update_by_query -d ' { "query": { "match": { "type": "fleet-package-policies" } },"script": { "source": "ctx._source[\"fleet-package-policies\"].bump_agent_policy_revision = true", "lang": "painless" } }' ``` ``` [2024-11-20T14:40:30.064+01:00][INFO ][plugins.fleet] Found 1 package policies that need agent policy revision bump [2024-11-20T14:40:31.933+01:00][DEBUG][plugins.fleet] Updated 1 package policies in space space1 in 1869ms, bump 1 agent policies [2024-11-20T14:40:35.056+01:00][DEBUG][plugins.fleet] Deploying 1 policies [2024-11-20T14:40:35.493+01:00][DEBUG][plugins.fleet] Deploying policies: 7f108cf2-4cf0-4a11-8df4-fc69d00a3484:10 ``` TODO: - the same flag has to be added on agent policy and output types, and the task extended to update them - I plan to do this in another pr, so that this doesn't become too big - add integration test if possible Tested with 500 agent policies split to 2 spaces, 1 integration per policy and bumping the flag in a new saved object model version, the bump task took about 6s. The deploy policies step is async, took about 30s. ``` [2024-11-20T15:53:55.628+01:00][INFO ][plugins.fleet] Found 501 package policies that need agent policy revision bump [2024-11-20T15:53:57.881+01:00][DEBUG][plugins.fleet] Updated 250 package policies in space space1 in 2253ms, bump 250 agent policies [2024-11-20T15:53:59.926+01:00][DEBUG][plugins.fleet] Updated 251 package policies in space default in 4298ms, bump 251 agent policies [2024-11-20T15:54:01.186+01:00][DEBUG][plugins.fleet] Deploying 250 policies [2024-11-20T15:54:29.989+01:00][DEBUG][plugins.fleet] Deploying policies: test-policy-space1-1:4, ... [2024-11-20T15:54:33.538+01:00][DEBUG][plugins.fleet] Deploying policies: policy-elastic-agent-on-cloud:4, test-policy-default-1:4, ... ``` - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]>
paulinashakirova
pushed a commit
to paulinashakirova/kibana
that referenced
this issue
Nov 26, 2024
…200536) ## Summary Closes elastic#193352 Update: Using a new SO field `bump_agent_policy_revision` in package policy type to mark package policies for update, this will trigger an agent policy revision bump. The feature supports both legacy and new package policy SO types, and queries policies from all spaces. To test, add a model version change to the package policy type and save. After Fleet setup is run, the agent policies using the package policies should be bumped and deployed. The same effect can be achieved by manually updating a package policy SO and loading Fleet UI to trigger setup. ``` '2': { changes: [ { type: 'data_backfill', backfillFn: (doc) => { return { attributes: { ...doc.attributes, bump_agent_policy_revision: true } }; }, }, ], }, curl -sk -XPOST --user fleet_superuser:password -H 'content-type:application/json' \ -H'x-elastic-product-origin:fleet' \ http://localhost:9200/.kibana_ingest/_update_by_query -d ' { "query": { "match": { "type": "fleet-package-policies" } },"script": { "source": "ctx._source[\"fleet-package-policies\"].bump_agent_policy_revision = true", "lang": "painless" } }' ``` ``` [2024-11-20T14:40:30.064+01:00][INFO ][plugins.fleet] Found 1 package policies that need agent policy revision bump [2024-11-20T14:40:31.933+01:00][DEBUG][plugins.fleet] Updated 1 package policies in space space1 in 1869ms, bump 1 agent policies [2024-11-20T14:40:35.056+01:00][DEBUG][plugins.fleet] Deploying 1 policies [2024-11-20T14:40:35.493+01:00][DEBUG][plugins.fleet] Deploying policies: 7f108cf2-4cf0-4a11-8df4-fc69d00a3484:10 ``` TODO: - the same flag has to be added on agent policy and output types, and the task extended to update them - I plan to do this in another pr, so that this doesn't become too big - add integration test if possible ### Scale testing Tested with 500 agent policies split to 2 spaces, 1 integration per policy and bumping the flag in a new saved object model version, the bump task took about 6s. The deploy policies step is async, took about 30s. ``` [2024-11-20T15:53:55.628+01:00][INFO ][plugins.fleet] Found 501 package policies that need agent policy revision bump [2024-11-20T15:53:57.881+01:00][DEBUG][plugins.fleet] Updated 250 package policies in space space1 in 2253ms, bump 250 agent policies [2024-11-20T15:53:59.926+01:00][DEBUG][plugins.fleet] Updated 251 package policies in space default in 4298ms, bump 251 agent policies [2024-11-20T15:54:01.186+01:00][DEBUG][plugins.fleet] Deploying 250 policies [2024-11-20T15:54:29.989+01:00][DEBUG][plugins.fleet] Deploying policies: test-policy-space1-1:4, ... [2024-11-20T15:54:33.538+01:00][DEBUG][plugins.fleet] Deploying policies: policy-elastic-agent-on-cloud:4, test-policy-default-1:4, ... ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]>
juliaElastic
added a commit
that referenced
this issue
Nov 29, 2024
## Summary Resolves #193352 Documented the usage of `bump_agent_policy_revision` and using `revision` to bump on agent policy SO changes. I think it's not needed to introduce `bump_agent_policy_revision` on the agent policy SO type, enough to bump revision, and the agent policy will be deployed on Fleet setup when https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/agent_policies/deploy_agent_policies_task.ts is running. --------- Co-authored-by: Cristina Amico <[email protected]>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Dec 12, 2024
…200536) ## Summary Closes elastic#193352 Update: Using a new SO field `bump_agent_policy_revision` in package policy type to mark package policies for update, this will trigger an agent policy revision bump. The feature supports both legacy and new package policy SO types, and queries policies from all spaces. To test, add a model version change to the package policy type and save. After Fleet setup is run, the agent policies using the package policies should be bumped and deployed. The same effect can be achieved by manually updating a package policy SO and loading Fleet UI to trigger setup. ``` '2': { changes: [ { type: 'data_backfill', backfillFn: (doc) => { return { attributes: { ...doc.attributes, bump_agent_policy_revision: true } }; }, }, ], }, curl -sk -XPOST --user fleet_superuser:password -H 'content-type:application/json' \ -H'x-elastic-product-origin:fleet' \ http://localhost:9200/.kibana_ingest/_update_by_query -d ' { "query": { "match": { "type": "fleet-package-policies" } },"script": { "source": "ctx._source[\"fleet-package-policies\"].bump_agent_policy_revision = true", "lang": "painless" } }' ``` ``` [2024-11-20T14:40:30.064+01:00][INFO ][plugins.fleet] Found 1 package policies that need agent policy revision bump [2024-11-20T14:40:31.933+01:00][DEBUG][plugins.fleet] Updated 1 package policies in space space1 in 1869ms, bump 1 agent policies [2024-11-20T14:40:35.056+01:00][DEBUG][plugins.fleet] Deploying 1 policies [2024-11-20T14:40:35.493+01:00][DEBUG][plugins.fleet] Deploying policies: 7f108cf2-4cf0-4a11-8df4-fc69d00a3484:10 ``` TODO: - the same flag has to be added on agent policy and output types, and the task extended to update them - I plan to do this in another pr, so that this doesn't become too big - add integration test if possible ### Scale testing Tested with 500 agent policies split to 2 spaces, 1 integration per policy and bumping the flag in a new saved object model version, the bump task took about 6s. The deploy policies step is async, took about 30s. ``` [2024-11-20T15:53:55.628+01:00][INFO ][plugins.fleet] Found 501 package policies that need agent policy revision bump [2024-11-20T15:53:57.881+01:00][DEBUG][plugins.fleet] Updated 250 package policies in space space1 in 2253ms, bump 250 agent policies [2024-11-20T15:53:59.926+01:00][DEBUG][plugins.fleet] Updated 251 package policies in space default in 4298ms, bump 251 agent policies [2024-11-20T15:54:01.186+01:00][DEBUG][plugins.fleet] Deploying 250 policies [2024-11-20T15:54:29.989+01:00][DEBUG][plugins.fleet] Deploying policies: test-policy-space1-1:4, ... [2024-11-20T15:54:33.538+01:00][DEBUG][plugins.fleet] Deploying policies: policy-elastic-agent-on-cloud:4, test-policy-default-1:4, ... ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Dec 12, 2024
…ic#201742) ## Summary Resolves elastic#193352 Documented the usage of `bump_agent_policy_revision` and using `revision` to bump on agent policy SO changes. I think it's not needed to introduce `bump_agent_policy_revision` on the agent policy SO type, enough to bump revision, and the agent policy will be deployed on Fleet setup when https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/agent_policies/deploy_agent_policies_task.ts is running. --------- Co-authored-by: Cristina Amico <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When agent policy or package policy saved objects are migrated, they do not trigger a revision bump and thus the updated policies are never sent out to the agents. The policies are only bumped if they get updated again by another means.
We often migrate these saved objects so we need to implement some mechanism here to push the changes. This is frequently an issue for endpoint package policy migrations in particular.
We do not often migrate other objects that are used by policies such as proxies, outputs, etc but theoretically the same issue can occur when those are migrated too.
The text was updated successfully, but these errors were encountered: