Allow conditions for all stack components' version

[ChrsMark]

At

package-spec/versions/1/manifest.spec.yml

Line 162 in 322a039

kibana:

we can define the version condition for Kibana. It would be useful to have a similar option for other stack's components like elastic-agent so as to ensure that a specific package's version we release defines properly the versions of Agent (and Beats under the hood) that is compatible with.

For reference I had tried something similar in elastic/integrations@c1a765d#diff-613f6ac342fad45946d18318537d166ce2b80989242310eb1dfb018b3002dc0d for prometheus package when I introduced types' functionality for remote_write, however most probably this condition is ignored.

cc: @mtojek @kaiyan-sheng

Update (@jsoriano): After some discussions we have decided that we will add a new agent.version constraint.

We considered adding a restriction for the Elasticsearch version, or rename the Kibana constraint to refer to the full stack, but we decided not to do it because the current check seems to be enough in that regard, and it could be risky to rename it.

Fleet will use this agent.version to display soft warnings user about possible problems if a policy is used with agents whose versions don't comply with the constraint. For packages that don't have an agent.version, it will be assumed that agents should satisfy the kibana.version constraint.

We will need to extend elastic-package so it is able to test with versions of the agent different to the version of the stack. The work on independent agents will help on this.

Tasks

Preview Give feedback

1. [package-spec] Add a new `agent.version condition in manifests
2. [kibana] Show a soft warning when using agents with versions not supported by all the packages in the policy
3. [elastic-package] Support testing with custom agent versions

[mtojek]

cc @ruflin @ycombinator

I assume we need to enable support for this in Kibana and Agent?

[ruflin]

In the early days of the package, we had a version for elastic agent but we remove it as there was no use for it back then. I wonder if this restriction will be on the package level or per data stream. Normally it is a config option of a data stream that is not supported I assume? Or an input not yet available? Should in this scenario just this data stream not be available for the older Agents?

[ChrsMark]

For the specific case of elastic/integrations#243 we have new configuration options for an already existent datastream/metricset. In this scenario only the datastream is affected but still this affects the version of the package too since we version on package level, no? In any case the point here is that it has to be a way to clearly define the compatibility matrix between package/datastream versions and Agent's versions.

[kaiyan-sheng]

I recently ran into this problem for AWS package as well. There was a bug fix in x-pack/libbeat for AWS credentials and it's backported into 7.13. We need a way for AWS package in integrations repo to be able to specify a version for elastic-agent so we can ensure the bug fix is applied.

[ruflin]

Started a related discussion in #334 but closing #334 in favor of this one. Lets continue here.

[eyalkraft]

This feature would be of interest for @elastic/cloud-security-posture team too.

The integration package we develop uses a new input we defined that belongs to our newly introduced cloudbeat.

Since constraints on the integration package only limit the Kibana version, and older agents can work with newer Kibana versions (support matrix), It's possible for users using old agent (pre 8.3, this is the first released version of cloudbeat) to install our integration.
In this case obviously cloudbeat wouldn't be installed on these agents and effectively our integration package won't work.

This also remains relevant to the future development of cloudbeat as we are constantly introducing more fetchers (that collect different resource types) to cloudbeat, and new compliance rules to our integrations, that are dependent on those new collected resources.

cc @uri-weisman

[joshdover]

In this case obviously cloudbeat wouldn't be installed on these agents and effectively our integration package won't work.

I wonder if solving this with health/status information (w/ recommendation to upgrade) after they've added the integration might be a better experience than preventing the user from moving forward with getting things working.

What's the downside of letting a user add an integration that isn't yet supported by the agent? Do the other integrations continue to run successfully?

[mtojek]

What's the downside of letting a user add an integration that isn't yet supported by the agent? Do the other integrations continue to run successfully?

AFAIK all integrations are operating correctly in that setup, but we don't perform any automated tests.

[ruflin]

What's the downside of letting a user add an integration that isn't yet supported by the agent? Do the other integrations continue to run successfully?

I don't think this is a great user experience. We allow users to set something up that "we" already know will not work.

[eyalkraft]

I wonder if solving this with health/status information (w/ recommendation to upgrade) after they've added the integration might be a better experience than preventing the user from moving forward with getting things working.

I think wether we decide this is the preferred UX or not, Adding this to the health/status information (w/ recommendation to upgrade) is a must since users will always be able to start a new agent (from an older version) connected to the agent policy where the integration is already installed.

Edit:
Or we don't allow older version agents to enroll but that could be a really bad UX for someone trying to deploy the agent.

[joshdover]

What's the downside of letting a user add an integration that isn't yet supported by the agent? Do the other integrations continue to run successfully?

I don't think this is a great user experience. We allow users to set something up that "we" already know will not work.

We need to think about scale here as well though. If I have an agent policy with 10k laptops enrolled and 5 of them are on an outdated agent version (b/c their users are on PTO and haven't opened their laptops), should I be blocked from adding an integration to the other 9995?

[eyalkraft]

@mukeshelastic @jlind23
I understand from @ruflin this issue is related to the Data Collection team roadmap.

Indicating users why our integration doesn't work (when it's related to their elastic-agent version) and how to fix it (update the agent) is pretty important for us (@elastic/cloud-security-posture) as we introduce our new kibana plugin and beat.

Do you think we can prioritize a solution for this problem to be delivered in one of the next releases (8.4/8.5)?
We are planning our first release (tech-preview) behind a feature flag for 8.3, and then an open (no feature flag) beta release for 8.4.

[jlind23]

@eyalkraft until we have a proper path to solve it, I will not take any commitment on it.
@joshdover could you please drive this conversation to a conclusion?

[joshdover]

I think we should defer on adding any agent version constraints at this time and ensure that the health reporting works, especially to solve the case the was mentioned above where an older agent is enrolled in a policy that already has an incompatible integration.

Input health status reporting is baked into the new v2 control protocol for Elastic Agent, which is being implemented in 8.4: elastic/elastic-agent-client#32

Cloudbeat should be able to leverage this once libbeat is migrated to v2 to be able to report detailed information about requiring the user to upgrade. This information will surfaced in the UI which is being tracked in https://github.com/elastic/security-team/issues/3494.

@michalpristas do you have any more detailed information or timeline about when Cloudbeat would be able to report it's status and how that would work with libbeat?

[eyalkraft]

Thanks for your reply @joshdover!
Seems like this would solve the problem for version 8.4+ of the agent and Kibana. This will be enough for us.

I'll open an issue for cloudbeat to support the health status interface. (elastic/cloudbeat#239)

Just another though:
Thinking about future beats (input types) that will be released after the V2 change. Ideally if an agent will be configured with an (agent policy +) integration that requires an input type that was introduced in a later version, The agent will somehow indicate that as well. This can't be dependent on the beat implementing some interface since in this case the beat won't even be downloaded since it doesn't exist in the artifactory for the (older) agent's version.

I went over the issues you linked and the health status reporting doc but I wasn't sure where to comment that.

[ishleenk17]

Another scenario where having the elastic agent version would be helpful:
elastic/integrations#9221 (comment)

[jsoriano]

I guess that we could at least:

• Add a new agent.version condition.
• Fleet prevents updating policies that have enrolled agents that don't match the condition of the upgraded package. (What would happen with packages that are automatically upgraded?).
• elastic-package allows to test with versions of the agent different to the version of the stack.

[jlind23]

What should happen if users try to enroll an Elastic Agent to a policy where there is an uncompatible integrations? Shall we block this or shall we allow it but one of the input might report as degraded / unhealthy?

[jsoriano]

What should happen if users try to enroll an Elastic Agent to a policy where there is an uncompatible integrations? Shall we block this or shall we allow it but one of the input might report as degraded / unhealthy?

Maybe the default is to block this, and if there is some kind of forced enrollment, it is enrolled but reported as degraded.

[amitkanfer]

What do we do when installing an integration that requires root privileges on an agent that is running as non-root? I believe we let the installation pass and then show the agent as unhealthy. Shouldn't we be consistent?

[jlind23]

@amitkanfer this is exactly what we came up with today during the integration office hours, @jsoriano will update this issue shortly.

[jsoriano]

@amitkanfer @jlind23 I have added an update to the description summarizing the discussions. I think we have three main tasks that may be developed more or less independently. As proposed in the meeting, we can start implementing the warnings in Kibana based on the kibana version constraint. For testing we can leverage the recently added independent agents to run them with versions different to the stack version.

[lucabelluccini]

This item was discussed during the Integrations Dev group as stated by Jaime.

First milestone:

• Kibana Fleet UI should warn if an Elastic Agent is running an Elastic Agent policy which contains one or more integration policies which require a kibana.version higher than the one of the Elastic Agent

On the long term:

• Each integration should define a recommended/minimum Elastic Agent version (elastic_agent.version or equivalent) in the manifest
• Kibana Fleet UI should warn if an Elastic Agent is running an Elastic Agent policy which contains one or more integration policies which require a elastic_agent.version (or equivalent) higher than the one of the Elastic Agent
  • The Elastic Agent diagnostic should contain the recommended version of each integration within the policy, so this can be easily spotted during a review or using an automated process