[Fleet] RBAC - Make upgrade agent APIs space aware #190069

jillguyonnet · 2024-08-07T15:05:24Z

Summary

Relates to #185040

This PR makes the following Fleet agents API space aware (behind useSpaceAwareness feature flag):

POST /agents/{agentId}/reassign
POST /agents/{agentId}/upgrade
POST /agents/bulk_reassign
POST /agents/bulk_upgrade
POST /agents/{agentId}/actions/{actionId}/cancel

While working on that last endpoint, I noticed and fixed an error in the documentation.

Checklist

Unit or functional tests were updated or added to match the most common scenarios
Flaky Test Runner was used on any tests changed

obltmachine · 2024-08-07T15:05:37Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

jillguyonnet · 2024-08-07T15:07:04Z

/ci

elasticmachine · 2024-08-08T08:03:31Z

Pinging @elastic/fleet (Team:Fleet)

kibanamachine · 2024-08-08T10:16:04Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6716

[✅] x-pack/test/fleet_api_integration/config.space_awareness.ts: 200/200 tests passed.

see run history

nchaulet · 2024-08-08T13:23:25Z

x-pack/plugins/fleet/server/services/agents/upgrade_action_runner.ts

@@ -167,6 +169,8 @@ export async function upgradeBatch(

  const actionId = options.actionId ?? uuidv4();
  const total = options.total ?? givenAgents.length;
+  const currentNameSpace = getCurrentNamespace(soClient);


it looks like this code could also run in a background task in that case the soClient namespace will not be correct (will be default),
we may want to use the agent namespace instead to populate that field

Thanks for flagging this up! I've made a change (pass space id to batch function), let me know if it looks ok.

Thinking more about this: in the current state, it looks like spaceId might be undefined in case this runs as a background task. A solution could be to modify the action runner to take a space id.

I'm not sure about using the namespaces property of (one of the) agents: could there be a scenario where we have agents belonging to multiple spaces? e.g. agent1 in spaces A and B, agent 2 in spaces B and C. In that case, we couldn't use agents to assign a space to the action.

I think as the task will come from an API we will probably have a namespace in the API request that we could pass to the task and use as a parameter for the action runner, does that make sense to you?

I thought this would affect the case when an action runner is used, for example for bulk reassign when the number of agents exceeds the batch size:

kibana/x-pack/plugins/fleet/server/services/agents/reassign.ts

Lines 128 to 138 in 813025c

return await new ReassignActionRunner(

esClient,

soClient,

{

...options,

batchSize,

total: res.total,

newAgentPolicyId,

},

{ pitId: await openPointInTime(esClient) }

).runActionAsyncWithRetry();

I should test this properly, but I expect that in this case the current solution will not provide a namespace. Do you mean that the API request should pass it (or already is)?

I think something like this should work

{ ...options, spaceId: soClient.getCurrentNamespace(), batchSize, total: res.total, newAgentPolicyId, },

kilfoyle

LGTM for platform-docs team! 👍

x-pack/test/fleet_api_integration/apis/space_awareness/agents.ts

jillguyonnet · 2024-08-09T08:36:09Z

x-pack/plugins/fleet/server/services/agents/crud.ts

@@ -442,7 +440,7 @@ async function _filterAgents(
      runtime_mappings: runtimeFields,
      fields: Object.keys(runtimeFields),
      sort: [{ [sortField]: { order: sortOrder } }],
-      query: { bool: { filter: query } },
+      query: addNamespaceFilteringToQuery({ bool: { filter: [query] } }, currentNameSpace),


@nchaulet What do you think of this change? I realised that a lot of bulk endpoints were calling getAgentsById and then checking isAgentInNamespace on each agent. With this, getAgentsById directly adds a namespace filter in the ES query. The downside is we lose the more specific "agent not in namespace" error message.

it seems okay to me 👍

nchaulet

🚀

After discussion, the case of background job should be addressed and tested in this PR

…ac-agents-write-api-3

…-fix'

x-pack/plugins/fleet/server/services/agents/bulk_actions_resolver.ts

…ac-agents-write-api-3

jillguyonnet · 2024-08-19T08:33:31Z

@elasticmachine merge upstream

kibana-ci · 2024-08-19T09:56:30Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: ad0195e

Failed CI Steps

Metrics [docs]

✅ unchanged

History

💚 Build #228104 succeeded a69ea1b
💔 Build #228058 failed 1ebcc3e
💔 Build #227739 failed 5d3fe24
💔 Build #227700 failed 903eb64
💔 Build #227656 failed 74a772a
💛 Build #226748 was flaky d7ee85b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jillguyonnet

nchaulet

🚀

jillguyonnet · 2024-08-19T14:22:18Z

Merging as agreed offline since Julia is away this week.

## Summary Closes #185040 Followup to: #188507 #189519 #190069 This PR contains the last required changed for making Fleet agents write APIs space aware: * Implement space awareness in the following endpoints: * `POST /agents/{agentId}/unenroll` * `POST /agents/{agentId}/request_diagnostics` * `POST /agents/bulk_unenroll` * `POST /agents/bulk_request_diagnostics` * Fix a bug in `POST /agents/bulk_update_agent_tags` where the space id was not passed. * Simply the setting of the `namespaces` property when creating agent actions when the space id is known. * Rename `currentNamespace` to `currentSpaceId` where appropriate (see comment below). * Add API integration tests and consolidate existing ones. ~⚠️ At the time of writing, I would like there to be more tests covering bulk query processing in batches, which are currently lacking. I have experienced difficulties getting those tests to pass consistently.~ Filed [followup issue](#191643) for those. ### A note on terminology As pointed out in #191083 (comment), it seems that the terms "namespace" and "space id" are occasionally used interchangeably in some parts of the codebase to refer to a Kibana space. For instance, documents in Fleet indices (agents, agent policies, agent actions...) [possess a `namespaces` property](elastic/elasticsearch#108363) to track the spaces they belong to. The current space id is also returned using the Saved Object client's `getCurrentNamespace` function. However, "namespace" is also a datastream property. In the Agent policy settings UI, the "Spaces" property (which will be linked to the saved object's `namespaces` property) is above the "Default namespace" property, which relates to the integration's data streams: <img width="1916" alt="Screenshot 2024-08-26 at 14 51 10" src="https://github.com/user-attachments/assets/fe2a0948-3387-4a93-96dc-90fc5cf1a683"> This should not be a source of major issues, but is best clarified for future reference. In this PR, I've replaced some occurrences of `namespace` with `spaceId` where appropriate to try to maximise the use of the latter. ### Testing * This PR should be put through the Flaky Test Runner prior to merging (I will kick the job). * Manual testing should also be performed for the new endpoints mentioned above. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: Elastic Machine <[email protected]>

[Fleet] Make upgrade agent APIs space aware

0a70588

jillguyonnet self-assigned this Aug 7, 2024

jillguyonnet added the Team:Fleet Team label for Observability Data Collection Fleet team label Aug 7, 2024

jillguyonnet added the release_note:skip Skip the PR/issue when compiling release notes label Aug 7, 2024

jillguyonnet requested a review from nchaulet August 7, 2024 15:05

jillguyonnet marked this pull request as ready for review August 8, 2024 08:03

jillguyonnet requested review from a team as code owners August 8, 2024 08:03

Refactor and add reassign endpoints

d2cff5d

nchaulet reviewed Aug 8, 2024

View reviewed changes

kilfoyle approved these changes Aug 8, 2024

View reviewed changes

Fix space id in batch functions

0c26e88

juliaElastic reviewed Aug 8, 2024

View reviewed changes

x-pack/test/fleet_api_integration/apis/space_awareness/agents.ts Outdated Show resolved Hide resolved

jillguyonnet added 2 commits August 8, 2024 17:02

remove tmp comment

4f7c2e7

Fix tests

25636c4

jillguyonnet commented Aug 9, 2024

View reviewed changes

Add reassign action namespaces

d7ee85b

nchaulet previously approved these changes Aug 9, 2024

View reviewed changes

nchaulet added 3 commits August 13, 2024 13:13

Merge branch 'main' of github.com:elastic/kibana into fleet/185040-rb…

bb3ee1e

…ac-agents-write-api-3

Fix test batch

cc50db2

Fix tests

74a772a

nchaulet requested a review from juliaElastic August 14, 2024 15:00

kibanamachine and others added 2 commits August 14, 2024 15:47

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

903eb64

…-fix'

fix types

5d3fe24

juliaElastic reviewed Aug 15, 2024

View reviewed changes

x-pack/plugins/fleet/server/services/agents/bulk_actions_resolver.ts Outdated Show resolved Hide resolved

nchaulet added 3 commits August 15, 2024 19:01

Merge branch 'main' of github.com:elastic/kibana into fleet/185040-rb…

1ebcc3e

…ac-agents-write-api-3

fix after merging main

f4e6738

fix unit tests

a69ea1b

Merge branch 'main' into fleet/185040-rbac-agents-write-api-3

ad0195e

nchaulet requested a review from juliaElastic August 19, 2024 11:55

nchaulet approved these changes Aug 19, 2024

View reviewed changes

jillguyonnet merged commit 1e64b9e into elastic:main Aug 19, 2024
23 checks passed

jillguyonnet deleted the fleet/185040-rbac-agents-write-api-3 branch August 19, 2024 14:22

kibanamachine added v8.16.0 backport:skip This commit does not require backporting labels Aug 19, 2024

lcawl mentioned this pull request Aug 22, 2024

[OAS][DOCS] Add feedback links for Kibana APIs #191145

Merged

jillguyonnet mentioned this pull request Aug 26, 2024

[Fleet] RBAC - Make agents write APIs space aware - 4/4 #191277

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] RBAC - Make upgrade agent APIs space aware #190069

[Fleet] RBAC - Make upgrade agent APIs space aware #190069

jillguyonnet commented Aug 7, 2024 •

edited

Loading

obltmachine commented Aug 7, 2024

jillguyonnet commented Aug 7, 2024

elasticmachine commented Aug 8, 2024

kibanamachine commented Aug 8, 2024

nchaulet Aug 8, 2024

jillguyonnet Aug 8, 2024 •

edited

Loading

jillguyonnet Aug 9, 2024 •

edited

Loading

nchaulet Aug 9, 2024

jillguyonnet Aug 9, 2024

nchaulet Aug 9, 2024

kilfoyle left a comment

jillguyonnet Aug 9, 2024

nchaulet Aug 9, 2024

nchaulet left a comment

jillguyonnet commented Aug 19, 2024

kibana-ci commented Aug 19, 2024

nchaulet left a comment

jillguyonnet commented Aug 19, 2024

	return await new ReassignActionRunner(
	esClient,
	soClient,
	{
	...options,
	batchSize,
	total: res.total,
	newAgentPolicyId,
	},
	{ pitId: await openPointInTime(esClient) }
	).runActionAsyncWithRetry();

[Fleet] RBAC - Make upgrade agent APIs space aware #190069

[Fleet] RBAC - Make upgrade agent APIs space aware #190069

Conversation

jillguyonnet commented Aug 7, 2024 • edited Loading

Summary

Checklist

obltmachine commented Aug 7, 2024

🤖 GitHub comments

jillguyonnet commented Aug 7, 2024

elasticmachine commented Aug 8, 2024

kibanamachine commented Aug 8, 2024

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#6716

nchaulet Aug 8, 2024

Choose a reason for hiding this comment

jillguyonnet Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

jillguyonnet Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

nchaulet Aug 9, 2024

Choose a reason for hiding this comment

jillguyonnet Aug 9, 2024

Choose a reason for hiding this comment

nchaulet Aug 9, 2024

Choose a reason for hiding this comment

kilfoyle left a comment

Choose a reason for hiding this comment

jillguyonnet Aug 9, 2024

Choose a reason for hiding this comment

nchaulet Aug 9, 2024

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

jillguyonnet commented Aug 19, 2024

kibana-ci commented Aug 19, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

nchaulet left a comment

Choose a reason for hiding this comment

jillguyonnet commented Aug 19, 2024

jillguyonnet commented Aug 7, 2024 •

edited

Loading

jillguyonnet Aug 8, 2024 •

edited

Loading

jillguyonnet Aug 9, 2024 •

edited

Loading