[EPM] Ingest - Policy and Datasources API

[mattapperson]

The routes & shapes are still TBD, but we know fka Ingest will need HTTP & JS APIs for:

• CRUD operations for Policies
• CRUD operations for Datasources
• Links between the two (policy id -> datasources; datasource id -> policy)

The initial draft of this is at jfsiii/kibana/ingest-api-strawman

See the route names & response types in a OpenAPI viewer like ReDoc

The TS types for the responses are in types.ts

[mattapperson]

A few comments on https://github.com/jfsiii/kibana/commit/14b21ca9f9e727cb93e3f2a23158be5ae61c5818:

1. The Policy type lacks
   1. the retry config
   2. the asset base path
   3. the monitoring settings (will this be a data source now? If so, how will agent know it needs a sidecar for it?)
2. We need a programatic API for getPolicyByID
3. How will things work if a user deletes a policy? Do all agents get unenrolled I assume? How will you notify fleet that this has happened?
4. Shouldn't inputs have an ID? So if there is an issue with one on the agent we can link back to the source of the error?
5. For beats that have a high cardinality of inputs, we need a delta API for the agent to only get what has been updated after a given timestamp (needed for heartbeat, perhaps not needed for phase 1?)
6. We also will need an event API to know when a policy has changed. This will be needed for the following as example use-cases:
   1. To be able to unenroll agents formerly assigned to a removed policy, and/or later unassigned them. (MVP)
   2. If/When we implement caching of policy on the fleet level for load handling (Not MVP)
   3. To push alerts out to agents once we have a real-time connection (Not MVP but high priority to support endgame use-case)

[jfsiii]

@mattapperson I deleted two comments and combined them into one with numbers so we can refer to them and discuss them

@roncohen @ruflin @clintongormley Can you take a look at these above items and weigh in on things like

• what falls under Ingest vs Fleet/Agent (e.g. the questions in point 1 above)
• what's in MVP vs subsequent phases?

[roncohen]

1. The Policy type lacks
   1. the retry config

@mattapperson could sum up the details following todays session? (check-in interval makes sense in the policy, asset download retry settings do not)

the asset base path

according to your comment elsewhere, it should just be a toggle. I think it makes sense to have the URL in the MVP. It could be a simple input field under advanced that is empty by default. Agent will download from elastic.co and then try to download from Kibana if that doesn't work. If this field is set it will try to download from there instead.

the monitoring settings (will this be a data source now? If so, how will agent know it needs a sidecar for it?)

I don't know what this setting does. Do you mind elaborating @mattapperson ?

We need a programatic API for getPolicyByID

that makes sense to me

How will things work if a user deletes a policy? Do all agents get unenrolled I assume? How will you notify fleet that this has happened?

I seem to recall you intend to let people move agents between in the MVP? Would be good to get explicit confirmation or rejection in the MVP doc. We keep going back and forth on it.

how about if a policy is deleted, getPolicyByID will just return nothing?

Shouldn't inputs have an ID? So if there is an issue with one on the agent we can link back to the source of the error?

Probably fine to skip for the MVP? Datasources already have IDs because we need to tag the data with ID.

For beats that have a high cardinality of inputs, we need a delta API for the agent to only get what has been updated after a given timestamp (needed for heartbeat, perhaps not needed for phase 1?)

You said it, let's skip for phase 1. 👍

1. We also will need an event API to know when a policy has changed. This will be needed for the following as example use-cases:
   1. To be able to unenroll agents formerly assigned to a removed policy, and/or later unassigned them. (MVP)
   1. If/When we implement caching of policy on the fleet level for load handling (Not MVP)
   1. To push alerts out to agents once we have a real-time connection (Not MVP but high priority to support endgame use-case)

Sounds like the MVP can be covered just by calling getPolicyByID when fleet receives a poll from the agent?

[jfsiii]

@mattapperson re: 1 After our "discuss package asset installs and Matts RFC if Fleet had done it" meeting, I understood that as long as the policy creation API returned the data structure the ingest branch originally did, the UI would have everything it needed. i.e.

kibana/x-pack/legacy/plugins/ingest/server/libs/policy.ts

Lines 34 to 47 in edb24f0

	const newPolicy: NewPolicyFile = {
	name,
	description: description || '',
	status: 'active',
	monitoring_enabled: true,
	shared_id: `shared-${uuidv4()}`,
	version: 0,
	agent_version: info.kibana.version,
	data_sources: [],
	created_on: new Date().toISOString(),
	created_by: withUser.username,
	updated_on: new Date().toISOString(),
	updated_by: withUser.username,
	};

The current Policy type is

export interface Policy {
  datasources?: Datasource[];
  description?: string;
  id: string;
  name?: string;
  status: Status;
}

Which I take to mean the differences between them are

• monitoring_enabled: I'm looking for confirmation that this is an Ingest concern
• The multiples ids, which I understood were for phased rollouts and not in MVP
• The created_* & updated_* audit fields. These seem like a good idea and might make it in. I'm not sure if they're required for MVP

I don't see any reference to the retry or asset base path concerns in the NewPolicyFile data structure

[mattapperson]

@jfsiii and @roncohen To avoid nesting quotes too deep ill start over and just try to cover all the points by referencing the number/letter in my orig comment.

1. Correct our API is our ideal, but even it required a few additions that were added after it was created. That said, the meeting today split things out a bit so not all the additions will exist on a policy.

   • the polling frequency config. @ph is working on a sample of what that will look like.
   • the the asset base path. This is **no longer ** will exist on the policy but will be a fleet config.
   • monitoring_enabled setting... is well... @roncohen do we want to support monitoring of agents/beats in MVP? -- cc @ph

2. If a policy gets removed, fleet but be notified.
   This touches on "will moving agents between policy be in MVP". The short answer is "I don't know". We are currently treating that as a stretch goal.

• If we do have move agent between policy, getPolicyByID will only let us know an agent is no longer assigned once it has checked in and gets null back. This will lead to a sporadicly growing list of unassigned agents with any temporary agents (laptops for example) taking days to be clearly marked as unassigned.
• If we dont have move between policy in MVP then this means agents will slowly become in an error state... same issue as above.

Regardless, not having an events API of some kind, IMHO would feel very very broken.

4. If agents don't have IDs, then fleet must remap inputs to have the ID of the data source they came from. Not a big deal but fleet ends up then source mapping errors back to policy and datasource thus not adding an ID does add more scope to Fleet. Not saying its right or wrong, but not having it does not remove work from an MVP, it just moves it.

5. see point 3. Comment is the same 😊

@jfsiii As for the spec for the policy, please see point 1 in this comment 😊

[mattapperson]

Summery from a zoom session:

• Fleet will expose a method of policyUpdated(policyId: string, type: "added" | "updated" | "removed", payload: Policy | null): Promise<{success: bool}>
• Fleet project will manage the transform from policy storage => agent optimized data structure, including reducing streams when needed. Thus fleet will be responsible to source map inputs <=> streams <=> data sources for error feedback.
• Followup with @ph included needs to happen for monitoring settings... does agent still sidecar beats for this?

[mattapperson]

@jfsiii @roncohen I forgot on our call, we also need a way to list policies in an API in order to populate the agent listing drop down to filter agents via the policy they are assigned too. We have this working today but only by policy ID. We need a way to get a list of policy names to make this more user friendly

[jfsiii]

@mattapperson /policies will do that (spec, explorer)

• GET /policies -> array of all policies
• GET /policies?policyId=xyz123 -> policy with id xyz123
• GET /policies/?datasource=abc -> array of policies which use datasource abc

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[mattapperson]

@jfsiii Could you please provide in your spec what paging would look like?

[jfsiii]

@mattapperson I hadn't considered it before now. A quick look around shows:

• the ES _search endpoint uses from & size
• the Saved Object _find endpoint uses page & per_page

Do you have a preference or ideal approach? You can comment here or on the PR I opened to merge the specs into our feature branch

[mattapperson]

We normally use page & per_page approach, but it's your call 😊