[Fleet] policy revision_idx source of truth

[michel-laterman]

The PR elastic/fleet-server#3131 to remove the fleet-server coordinator changes some migration logic to increment the revision_idx instead of the removed coordinator_idx in a .fleet-policy document in order to force fleet-server to reload the policy and generate a new outputs block.

However, fleet's current source of truth for this attribute is part of the agent policy saved object so we may run into a conflict as they differ.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

I think it make sense to have kibana to be the source of truth for revision, I am wondering if we can have another mechanism in fleet server.

[nchaulet]

@michel-laterman I do not think it should be possible for fleet-server to increment revision idx, we will not be able to know when a policy is out-of-date, and the new revision will not match our saved objects, I am wondering if it's better to include a new field? write to a new index fleet-server processed policies? (thinking loud here)

[michel-laterman]

We increment the coordinator_idx/revision_idx in order to signal the policy monitor to generate new api keys as part of output migration for v8.5.0: https://github.com/elastic/fleet-server/blob/main/internal/pkg/dl/migration.go#L187-L205.
If we changed the migration that updates the idx number to alter another field it would need to be one that the policy monitor is aware of.

Or we may be able to try to change the checkin handler on fleet-server to detect a migrated agent; i think this would be a document with outputs.default.api_key="" and outputs.default.permission_hash=$VALUE and force it to process the policy and prepare outputs for the agent. Does that seem correct to you?

@AndersonQ if you remember any details of the issue (elastic/fleet-server#1672) we would welcome your input

[AndersonQ]

@AndersonQ if you remember any details of the issue (elastic/fleet-server#1672) we would welcome your input

@michel-laterman, what I remember is pretty much what I commented in the code, the coordinator_idx is increased to force the api keys to be regenerated. As long as they are regenerated, it's fine.

[michel-laterman]

elastic/fleet-server#3131 has been updated so that fleet-server will no longer increment the revision_idx and the above migration strategy will be used