[Fleet] Add scoped AgentService #119017

joshdover · 2021-11-18T12:23:19Z

Summary

This updates the agent service currently exposed on Fleet's server-side plugin API to now supply a "scoped" and "internal user" interface of the same client to allow plugins and Fleet's internal implementation code to get authz out-of-the-box on these actions and support the switch to system indices.

In this first iteration, I have only included the changes to support external usages in other plugins of the 4 currently supported APIs. In the next PR, I'll add the additional APIs we use internally in Fleet and switch over our implementation to use this service internally.

Example usage, from a HTTP request handler

const handler = async (context, req, res) => {
  // Will automatically throw a `FleetUnauthorizedError` if user does not have required access.
  // Consumers should catch `FleetUnauthorizedError` errors and return unauthorized HTTP responses
  const agents = await context.fleet.agentClient.asCurrentUser.listAgents();
}

From a server-side background process, for example in telemetry collectors:

const agentClient = startDeps.fleet.agentService.asInternalUser; // uses kibana_system user and never throws authz errors
const agents = await agentClient.listAgents();

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

x-pack/plugins/security_solution/server/endpoint/routes/metadata/handlers.ts

x-pack/plugins/fleet/server/plugin.ts

x-pack/plugins/fleet/server/services/index.ts

nchaulet · 2021-11-18T15:17:02Z

Love that approach it could works well with Fleet superuser privilege requirement removal, and it's seems easy enough to consume.

joshdover · 2021-11-19T16:37:51Z

@paul-tavares Would you be able to take over fixing the security solution issues in this PR?

paul-tavares · 2021-11-19T19:42:23Z

@joshdover yes, I can do that. Will start now, but may need to go into Monday

paul-tavares · 2021-11-19T20:36:25Z

Hi @joshdover ,
I just noticed that AgentService now requires us to pass in the KibanaRequest in order to get back a scoped AgentClient (the method asScoped) which makes it a little harder for us to share services outside of a HTTP Handler. Any way we can change that? I'm still looking at all the changes, so I don't have any good ideas just yet, but figure I would ask.

I'm also going to see if we (endpoint) can maybe wrap this. Maybe I'll change our services interface to instead receive in an AgentClient on input to the methods that needed it. that will make it easier to move off of asScoped in the future if find the need to.

…pointAppContextServices to use it

paul-tavares · 2021-11-24T17:39:05Z

x-pack/plugins/security_solution/server/endpoint/services/endpoint_fleet_services.ts

+  ) {}
+
+  asScoped(req: KibanaRequest): EndpointScopedFleetServicesInterface {
+    const {


FYI: Endpoint team:
I would like at some point to maybe use this to also scope the the other services from fleet. Each of those normally takes in either an ES scoped client or a scoped SO client, so we could adjust the other services here so that we no longer have to pass in these client explicitly. Example for the packagePolicy.list() method:

{ list: packagePolicy.list.bind(packagePolicy, soScopedClient) }

and then we could just packagePolicy.list(options) - no more need to pass in the decencies (in this case the SO Client)

joshdover · 2021-11-29T14:15:19Z

I just noticed that AgentService now requires us to pass in the KibanaRequest in order to get back a scoped AgentClient (the method asScoped) which makes it a little harder for us to share services outside of a HTTP Handler. Any way we can change that? I'm still looking at all the changes, so I don't have any good ideas just yet, but figure I would ask.

This is the purpose of the asInternalUser API which returns a client that will use the internal user without any explicit authz checks. Does that solve the use case?

joshdover · 2021-11-29T14:15:54Z

I'm also going to see if we (endpoint) can maybe wrap this. Maybe I'll change our services interface to instead receive in an AgentClient on input to the methods that needed it. that will make it easier to move off of asScoped in the future if find the need to.

++ that was what I was thinking as well. Switch over to using AgentClient and no longer rely on the AgentService interface.

…coped-agent-service # Conflicts: # x-pack/plugins/fleet/server/mocks/index.ts # x-pack/plugins/fleet/server/plugin.ts # x-pack/plugins/fleet/server/routes/setup/handlers.test.ts # x-pack/plugins/fleet/server/types/request_context.ts # x-pack/plugins/security_solution/server/endpoint/routes/metadata/handlers.ts # x-pack/plugins/security_solution/server/endpoint/services/metadata/endpoint_metadata_service.ts

…coped-agent-service

joshdover · 2021-11-30T12:31:08Z

@paul-tavares I noticed that endpoint has a few other code paths that query .fleet-actions and .fleet-action-results indices that are not using Fleet services (probably because we don't provide an API for these).

I plan to add a follow up PR that introduces APIs for querying these and I think it will be necessary to switch over those usages as well in order to support these indices becoming system indices (so the end user cannot access them out of the box, not even superuser).

paul-tavares · 2021-11-30T13:40:01Z

Re: usage of .fleet-actions-* indexes

I'm not too familiar with discussions around those indexes, but I know that @pzl worked closely with the OSQuery team when we started to use them (I think OSQuery was the first). you mentioned creating an API, but I wonder if you mean providing a server-side service that we can consume - is that what you meant?

cc/ @ashokaditya FYI

paul-tavares · 2021-11-30T13:48:52Z

@joshdover
FYI - we also interact with the .fleet-artifacts index, but do so using a Fleet provided server side service - this one:

https://github.com/elastic/kibana/blob/7ebffc3728a04fa806a29e39f5fd0eb87f514368/x-pack/plugins/fleet/server/services/artifacts/client.ts

elasticmachine · 2021-11-30T14:51:47Z

Pinging @elastic/fleet (Team:Fleet)

patrykkopycinski

Asset management LGTM
Thank you @joshdover

paul-tavares

🚢 it 😄

x-pack/plugins/fleet/server/mocks/index.ts

x-pack/plugins/fleet/server/plugin.ts

x-pack/plugins/fleet/server/services/agents/agent_service.ts

nchaulet

🚀

kibana-ci · 2021-12-01T16:12:51Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1145	1143	-2

Unknown metric groups

API count

id	before	after	diff
`fleet`	1248	1249	+1

History

💔 Build #10331 failed b9502fb
💔 Build #10314 failed be29907
💔 Build #10008 failed 40ae1a1
💔 Build #9777 failed 562a2d4
💔 Build #9736 failed 940c840
💔 Build #9156 failed 8ea2553

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @joshdover @paul-tavares

pjhampton · 2021-12-01T17:30:38Z

x-pack/plugins/security_solution/server/lib/telemetry/receiver.ts

@@ -32,7 +32,7 @@ import {

 export class TelemetryReceiver {
  private readonly logger: Logger;
-  private agentService?: AgentService;
+  private agentClient?: AgentClient;


Security Telemetry change looks good!

pjhampton

🌔 🚀 ✨ LGTM ✨ 🚀 🌔

Approving minor import change on security telemetry instrumentation.

Co-authored-by: Paul Tavares <[email protected]>

kibanamachine · 2021-12-01T17:34:27Z

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.

Co-authored-by: Paul Tavares <[email protected]> Co-authored-by: Josh Dover <[email protected]> Co-authored-by: Paul Tavares <[email protected]>

Co-authored-by: Paul Tavares <[email protected]>

refs elastic/pull/119017

joshdover commented Nov 18, 2021

View reviewed changes

x-pack/plugins/security_solution/server/endpoint/routes/metadata/handlers.ts Outdated Show resolved Hide resolved

joshdover mentioned this pull request Nov 18, 2021

[Fleet] Exposed service to manipulate fleet-* should use internal ES client #116182

Closed

joshdover commented Nov 18, 2021

View reviewed changes

x-pack/plugins/fleet/server/plugin.ts Outdated Show resolved Hide resolved

joshdover commented Nov 18, 2021

View reviewed changes

x-pack/plugins/fleet/server/services/index.ts Outdated Show resolved Hide resolved

Add scoped AgentService

d27c1df

joshdover force-pushed the fleet/116182 branch from cd7799c to d27c1df Compare November 19, 2021 14:30

joshdover mentioned this pull request Nov 19, 2021

[Fleet] Remove superuser requirement to access Fleet #108252

Closed

29 tasks

paul-tavares self-assigned this Nov 22, 2021

paul-tavares added the OLM Sprint label Nov 22, 2021

paul-tavares added 2 commits November 24, 2021 11:12

endpoint: added new EndpointFleetServices factory + new method in End…

bb2253f

…pointAppContextServices to use it

Endpoint Metadata Service updated to use EndpointFleetServicesInterface

6c87f01

paul-tavares reviewed Nov 24, 2021

View reviewed changes

paul-tavares added 3 commits November 24, 2021 13:21

fix tests for endpoing metadatadata

5293413

Fix metadata handlers and tests

40b09af

Fix up hosts search strategy when calling metadata services

8ea2553

paul-tavares and others added 6 commits November 29, 2021 09:48

Fix ini of EndpointMetadataService from plugin

6a33fb9

some adjustments following merge from upstream/main

940c840

Fix mock in fleet server

4e0ab46

Merge remote-tracking branch 'upstream/main' into josh/fleet-116182-s…

562a2d4

…coped-agent-service

Merge remote-tracking branch 'upstream/main' into fleet/116182

fbb2b9b

joshdover requested review from a team as code owners November 30, 2021 14:51

joshdover requested review from joeypoon and parkiino November 30, 2021 14:51

joshdover changed the title ~~[WIP] Add scoped AgentService~~ Add scoped AgentService Nov 30, 2021

joshdover changed the title ~~Add scoped AgentService~~ [Fleet] Add scoped AgentService Nov 30, 2021

patrykkopycinski approved these changes Nov 30, 2021

View reviewed changes

paul-tavares approved these changes Nov 30, 2021

View reviewed changes

x-pack/plugins/fleet/server/mocks/index.ts Outdated Show resolved Hide resolved

x-pack/plugins/fleet/server/plugin.ts Outdated Show resolved Hide resolved

x-pack/plugins/fleet/server/services/agents/agent_service.ts Show resolved Hide resolved

joshdover added 2 commits December 1, 2021 13:20

Merge remote-tracking branch 'upstream/main' into fleet/116182

b31d58c

PR nits

be29907

joshdover enabled auto-merge (squash) December 1, 2021 12:27

joshdover mentioned this pull request Dec 1, 2021

[Meta] Use kibana_system to write to .fleet-* indices #120090

Closed

16 tasks

joshdover added 2 commits December 1, 2021 14:10

Fix mock

b9502fb

Fix security mock

9469691

nchaulet approved these changes Dec 1, 2021

View reviewed changes

pjhampton reviewed Dec 1, 2021

View reviewed changes

pjhampton approved these changes Dec 1, 2021

View reviewed changes

joshdover merged commit ad5afe1 into elastic:main Dec 1, 2021

kibanamachine mentioned this pull request Dec 1, 2021

[8.0] [Fleet] Add scoped AgentService (#119017) #120140

Merged

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Dec 1, 2021

[Fleet] Add scoped AgentService (elastic#119017)

193f0f9

Co-authored-by: Paul Tavares <[email protected]>

kibanamachine added a commit that referenced this pull request Dec 1, 2021

[Fleet] Add scoped AgentService (#119017) (#120140)

bb960f1

Co-authored-by: Paul Tavares <[email protected]> Co-authored-by: Josh Dover <[email protected]> Co-authored-by: Paul Tavares <[email protected]>

TinLe pushed a commit to TinLe/kibana that referenced this pull request Dec 22, 2021

[Fleet] Add scoped AgentService (elastic#119017)

f4c337e

Co-authored-by: Paul Tavares <[email protected]>

juliaElastic mentioned this pull request Apr 29, 2022

[Fleet] Expose Fleet services used by Osquery with authorization check #131240

Open

juliaElastic mentioned this pull request Jun 2, 2022

[Fleet] Collect Fleet services that should be exposed as scoped services #133406

Open

ashokaditya added a commit to ashokaditya/kibana that referenced this pull request Dec 22, 2022

correctly name types and update class methods

0dba4a0

refs elastic/pull/119017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Add scoped AgentService #119017

[Fleet] Add scoped AgentService #119017

joshdover commented Nov 18, 2021 •

edited

Loading

nchaulet commented Nov 18, 2021

joshdover commented Nov 19, 2021

paul-tavares commented Nov 19, 2021

paul-tavares commented Nov 19, 2021

paul-tavares Nov 24, 2021

joshdover commented Nov 29, 2021

joshdover commented Nov 29, 2021

joshdover commented Nov 30, 2021

paul-tavares commented Nov 30, 2021

paul-tavares commented Nov 30, 2021

elasticmachine commented Nov 30, 2021

patrykkopycinski left a comment

paul-tavares left a comment

nchaulet left a comment

kibana-ci commented Dec 1, 2021

API count

pjhampton Dec 1, 2021

pjhampton left a comment

kibanamachine commented Dec 1, 2021

[Fleet] Add scoped AgentService #119017

[Fleet] Add scoped AgentService #119017

Conversation

joshdover commented Nov 18, 2021 • edited Loading

Summary

Checklist

For maintainers

nchaulet commented Nov 18, 2021

joshdover commented Nov 19, 2021

paul-tavares commented Nov 19, 2021

paul-tavares commented Nov 19, 2021

paul-tavares Nov 24, 2021

Choose a reason for hiding this comment

joshdover commented Nov 29, 2021

joshdover commented Nov 29, 2021

joshdover commented Nov 30, 2021

paul-tavares commented Nov 30, 2021

paul-tavares commented Nov 30, 2021

elasticmachine commented Nov 30, 2021

patrykkopycinski left a comment

Choose a reason for hiding this comment

paul-tavares left a comment

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

kibana-ci commented Dec 1, 2021

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

History

pjhampton Dec 1, 2021

Choose a reason for hiding this comment

pjhampton left a comment

Choose a reason for hiding this comment

kibanamachine commented Dec 1, 2021

💚 Backport successful

joshdover commented Nov 18, 2021 •

edited

Loading