[Fleet] Fix issue of agent sometimes not getting inputs using a new agent policy with system integration #177594
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
juliaElastic
commented
Feb 22, 2024
| @@ -1034,7 +1038,7 @@ class AgentPolicyService { | |||
|
|
|||
| const bulkResponse = await esClient.bulk({ | |||
| index: AGENT_POLICY_INDEX, | |||
| body: fleetServerPoliciesBulkBody, | |||
| operations: fleetServerPoliciesBulkBody, | |||
There was a problem hiding this comment.
replaced deprecated field
|
/ci |
|
I'm working on an integration test as well to check the correct |
botelastic
bot
added
the
Team:Fleet
|
Pinging @elastic/fleet (Team:Fleet) |
juliaElastic
added
the
backport:prev-minor
nchaulet
approved these changes
Feb 23, 2024
Yeah it's interesting, this logic was there for a few years. |
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Feb 23, 2024
…gent policy with system integration (elastic#177594) ## Summary Closes elastic#177372 When creating an agent policy with a package policy immediately (e.g. system integration), the `deployPolicy` logic was called once, creating a doc in `.fleet-policies` with `revision:1` without `inputs`, and then updating the doc with `inputs`, still on `revision:1`. This is causing an intermittent issue on the agents, if Fleet-server picks up the first document, and delivers to agent without `inputs`. As a fix, added an option to skip `deploPolicy` when called from the `createAgentPolicyWithPackages` function, as the policy will be deployed after creating the package policies. To verify: - create an agent policy with system monitoring (default option) - check that the created documents in `.fleet-policies` are correct: there should be one doc with `revision_idx:1` and `coordinator_idx:0` (created by Fleet API), and one doc with `revision_idx:1` and `coordinator_idx:1` (created by fleet-server) - verify that both documents have `data.inputs` field populated Used this query to verify: ``` POST .fleet-policies/_search { "query": { "bool": { "must": [ { "term": {"coordinator_idx": 0} } ], "filter": { "term": { "policy_id": "<agent policy id>" } } } }, "_source": [ "revision_idx","coordinator_idx", "policy_id", "@timestamp", "data.inputs" ], "sort": [ { "revision_idx": { "order": "desc" } } ] } ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 5f17b39)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Feb 23, 2024
…a new agent policy with system integration (#177594) (#177725) # Backport This will backport the following commits from `main` to `8.13`: - [[Fleet] Fix issue of agent sometimes not getting inputs using a new agent policy with system integration (#177594)](#177594) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Julia Bardi","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-02-23T14:46:48Z","message":"[Fleet] Fix issue of agent sometimes not getting inputs using a new agent policy with system integration (#177594)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/177372\r\n\r\nWhen creating an agent policy with a package policy immediately (e.g.\r\nsystem integration), the `deployPolicy` logic was called once, creating\r\na doc in `.fleet-policies` with `revision:1` without `inputs`, and then\r\nupdating the doc with `inputs`, still on `revision:1`.\r\nThis is causing an intermittent issue on the agents, if Fleet-server\r\npicks up the first document, and delivers to agent without `inputs`.\r\nAs a fix, added an option to skip `deploPolicy` when called from the\r\n`createAgentPolicyWithPackages` function, as the policy will be deployed\r\nafter creating the package policies.\r\n\r\nTo verify:\r\n- create an agent policy with system monitoring (default option)\r\n- check that the created documents in `.fleet-policies` are correct:\r\nthere should be one doc with `revision_idx:1` and `coordinator_idx:0`\r\n(created by Fleet API), and one doc with `revision_idx:1` and\r\n`coordinator_idx:1` (created by fleet-server)\r\n- verify that both documents have `data.inputs` field populated\r\n\r\nUsed this query to verify:\r\n```\r\nPOST .fleet-policies/_search\r\n{\r\n \"query\": {\r\n \"bool\": {\r\n \"must\": [\r\n {\r\n \"term\": {\"coordinator_idx\": 0}\r\n }\r\n ],\r\n \"filter\": {\r\n \"term\": {\r\n \"policy_id\": \"<agent policy id>\"\r\n }\r\n }\r\n }\r\n }, \r\n \"_source\": [\r\n \"revision_idx\",\"coordinator_idx\", \"policy_id\", \"@timestamp\", \"data.inputs\"\r\n ],\r\n \"sort\": [\r\n {\r\n \"revision_idx\": {\r\n \"order\": \"desc\"\r\n }\r\n }\r\n ]\r\n}\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"5f17b39a1d4aa326f8b75bc0d2375f620433e9be","branchLabelMapping":{"^v8.14.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:prev-minor","v8.14.0"],"title":"[Fleet] Fix issue of agent sometimes not getting inputs using a new agent policy with system integration","number":177594,"url":"https://github.com/elastic/kibana/pull/177594","mergeCommit":{"message":"[Fleet] Fix issue of agent sometimes not getting inputs using a new agent policy with system integration (#177594)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/177372\r\n\r\nWhen creating an agent policy with a package policy immediately (e.g.\r\nsystem integration), the `deployPolicy` logic was called once, creating\r\na doc in `.fleet-policies` with `revision:1` without `inputs`, and then\r\nupdating the doc with `inputs`, still on `revision:1`.\r\nThis is causing an intermittent issue on the agents, if Fleet-server\r\npicks up the first document, and delivers to agent without `inputs`.\r\nAs a fix, added an option to skip `deploPolicy` when called from the\r\n`createAgentPolicyWithPackages` function, as the policy will be deployed\r\nafter creating the package policies.\r\n\r\nTo verify:\r\n- create an agent policy with system monitoring (default option)\r\n- check that the created documents in `.fleet-policies` are correct:\r\nthere should be one doc with `revision_idx:1` and `coordinator_idx:0`\r\n(created by Fleet API), and one doc with `revision_idx:1` and\r\n`coordinator_idx:1` (created by fleet-server)\r\n- verify that both documents have `data.inputs` field populated\r\n\r\nUsed this query to verify:\r\n```\r\nPOST .fleet-policies/_search\r\n{\r\n \"query\": {\r\n \"bool\": {\r\n \"must\": [\r\n {\r\n \"term\": {\"coordinator_idx\": 0}\r\n }\r\n ],\r\n \"filter\": {\r\n \"term\": {\r\n \"policy_id\": \"<agent policy id>\"\r\n }\r\n }\r\n }\r\n }, \r\n \"_source\": [\r\n \"revision_idx\",\"coordinator_idx\", \"policy_id\", \"@timestamp\", \"data.inputs\"\r\n ],\r\n \"sort\": [\r\n {\r\n \"revision_idx\": {\r\n \"order\": \"desc\"\r\n }\r\n }\r\n ]\r\n}\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"5f17b39a1d4aa326f8b75bc0d2375f620433e9be"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.14.0","branchLabelMappingKey":"^v8.14.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/177594","number":177594,"mergeCommit":{"message":"[Fleet] Fix issue of agent sometimes not getting inputs using a new agent policy with system integration (#177594)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/kibana/issues/177372\r\n\r\nWhen creating an agent policy with a package policy immediately (e.g.\r\nsystem integration), the `deployPolicy` logic was called once, creating\r\na doc in `.fleet-policies` with `revision:1` without `inputs`, and then\r\nupdating the doc with `inputs`, still on `revision:1`.\r\nThis is causing an intermittent issue on the agents, if Fleet-server\r\npicks up the first document, and delivers to agent without `inputs`.\r\nAs a fix, added an option to skip `deploPolicy` when called from the\r\n`createAgentPolicyWithPackages` function, as the policy will be deployed\r\nafter creating the package policies.\r\n\r\nTo verify:\r\n- create an agent policy with system monitoring (default option)\r\n- check that the created documents in `.fleet-policies` are correct:\r\nthere should be one doc with `revision_idx:1` and `coordinator_idx:0`\r\n(created by Fleet API), and one doc with `revision_idx:1` and\r\n`coordinator_idx:1` (created by fleet-server)\r\n- verify that both documents have `data.inputs` field populated\r\n\r\nUsed this query to verify:\r\n```\r\nPOST .fleet-policies/_search\r\n{\r\n \"query\": {\r\n \"bool\": {\r\n \"must\": [\r\n {\r\n \"term\": {\"coordinator_idx\": 0}\r\n }\r\n ],\r\n \"filter\": {\r\n \"term\": {\r\n \"policy_id\": \"<agent policy id>\"\r\n }\r\n }\r\n }\r\n }, \r\n \"_source\": [\r\n \"revision_idx\",\"coordinator_idx\", \"policy_id\", \"@timestamp\", \"data.inputs\"\r\n ],\r\n \"sort\": [\r\n {\r\n \"revision_idx\": {\r\n \"order\": \"desc\"\r\n }\r\n }\r\n ]\r\n}\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"5f17b39a1d4aa326f8b75bc0d2375f620433e9be"}}]}] BACKPORT--> Co-authored-by: Julia Bardi <[email protected]>
fkanout
pushed a commit
to fkanout/kibana
that referenced
this pull request
Mar 4, 2024
…gent policy with system integration (elastic#177594) ## Summary Closes elastic#177372 When creating an agent policy with a package policy immediately (e.g. system integration), the `deployPolicy` logic was called once, creating a doc in `.fleet-policies` with `revision:1` without `inputs`, and then updating the doc with `inputs`, still on `revision:1`. This is causing an intermittent issue on the agents, if Fleet-server picks up the first document, and delivers to agent without `inputs`. As a fix, added an option to skip `deploPolicy` when called from the `createAgentPolicyWithPackages` function, as the policy will be deployed after creating the package policies. To verify: - create an agent policy with system monitoring (default option) - check that the created documents in `.fleet-policies` are correct: there should be one doc with `revision_idx:1` and `coordinator_idx:0` (created by Fleet API), and one doc with `revision_idx:1` and `coordinator_idx:1` (created by fleet-server) - verify that both documents have `data.inputs` field populated Used this query to verify: ``` POST .fleet-policies/_search { "query": { "bool": { "must": [ { "term": {"coordinator_idx": 0} } ], "filter": { "term": { "policy_id": "<agent policy id>" } } } }, "_source": [ "revision_idx","coordinator_idx", "policy_id", "@timestamp", "data.inputs" ], "sort": [ { "revision_idx": { "order": "desc" } } ] } ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Closes #177372
When creating an agent policy with a package policy immediately (e.g. system integration), the
deployPolicylogic was called once, creating a doc in.fleet-policieswithrevision:1withoutinputs, and then updating the doc withinputs, still onrevision:1.This is causing an intermittent issue on the agents, if Fleet-server picks up the first document, and delivers to agent without
inputs.As a fix, added an option to skip
deploPolicywhen called from thecreateAgentPolicyWithPackagesfunction, as the policy will be deployed after creating the package policies.To verify:
.fleet-policiesare correct: there should be one doc withrevision_idx:1andcoordinator_idx:0(created by Fleet API), and one doc withrevision_idx:1andcoordinator_idx:1(created by fleet-server)data.inputsfield populatedUsed this query to verify:
Checklist