[ Fleet ] Fleet UI should allow upgrades to agents where the patch version is higher than Kibana

[leandrojmp]

Describe the feature:

Currently if you are on version X.Y.0 and want to upgrade an Elastic Agent to version X.Y.1, you can't do this through Fleet UI because the upgrade option will be grayed out, this means that the Elastic Agent version you can upgrade through Fleet UI is pinned to the patch version of Kibana.

The product compatibility does not mention this, it says for example that Elasticsearch 8.10.X is compatible with Elastic Agent 7.17.X - 8.10.X, so an User would expect to be able to run an Elastic Agent 8.10.3 even if the rest of the stack is on a lower patch version.

Kibana documentation mentions that generally Kibana is compatible with Elasticsearch on different patch versions, but does not mention if Kibana patch can be higher than Elastisearch patch, it also recommends to run the exact same version.

So if an user wants to follow this recommendation and needs to upgrade just a couple of agents to fix some issue, it would need to upgrade the entire stack, which is far from the ideal.

The solution in this case would be to manually upgrade an agent, which is also not ideal when you have many agents.

If the Elastic Agent needs to be so tight coupled with the Kibana version, then this needs to be reflected in the Support Matrix page.

Describe a specific use case for the feature:

Users should be able to upgrade the patch version of an Elastic Agent through the Fleet UI, this may happen when you need to upgrade for just fix a bug.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[rodrigc]

Following the discussion on this on https://discuss.elastic.co/t/elastic-agent-upgrade-option-is-grayed-out-on-fleet-server/344753

[juliaElastic]

I think the reason we don't allow this is laying on the agent side, @cmacknz correct me if I'm wrong. I tried to find the recent gh comments about this, but couldn't.

[lucabelluccini]

Just a comment about what we say in the docs.

For Elastic Agents there's a constraint on Fleet Server (docs)

The version of Elastic Agents running Fleet Server must be greater than or equal to the version of the Elastic Agents that enroll in them.

Here, we do not mention the patch version.

Meaning the Fleet Server version >= Elastic Agents. Usually Fleet Server is aligned with the stack.

On our doc we have also:

For version compatibility: Elasticsearch >= Fleet Server >= Elastic Agent (except for bugfix releases).
Kibana should be on the same minor version as Elasticsearch.

So we always mention minor version but not the patch.

[rodrigc]

@juliaElastic In this post: https://discuss.elastic.co/t/elastic-agent-upgrade-option-is-grayed-out-on-fleet-server/344753/5?u=craig_rodrigues
@leandrojmp mentioned that it is possible to upgrade from the command-line to elastic-agent 8.10.4 against a Kibana 8.10.2 server. But the Fleet UI does not allow this.

So this restriction/bug is in the Fleet UI.

[leandrojmp]

So we always mention minor version but not the patch.

@lucabelluccini The support matrix page also does not mention anything about the patch version, normally the compatibility is mentioned up to the minor version, like 8.10.X

So based on the documentation available, the user will assume that an agent on version 8.10.4 is compatible with the stack running on version 8.10.2, but currently the patch version is pinned on Kibana version.

The use case for having an Agent on a patch version higher than the rest of the stack would be to fix some bugs, like a recent one that broke some networking monitoring on Linux systems and was fixed on 8.10.3.

It is pretty common for on-premises deployments to have maintenance windows to upgrade the stack, and this cannot be done for every patch version, for example 8.10.3 was released on October 10th and 8.10.4 was released on October 17th, just one week late, but upgrading agents is easier than upgrading the entire stack, so it would be nice to be able to upgrade the agents through Fleet UI.

Upgrading the agent using the command line works without any issue, but this is not practical when you have hundreds or thousands of agents.

So based on this I think that the Kibana Fleet UI should allow the upgrade to agents on a higher patch version, if this is not possible then the documentation should reflect it and at least provide some orientation to the user on how to proceed with manual upgrades.

[lucabelluccini]

I'm not against your assessment. I was just collecting/gathering what we report in the documentation.

[cmacknz]

There is a version constraint in Fleet Server that will not accept agents with a higher minor or major version. It is explicitly allowing agents in the same minor release to connect regardless of the patch version. So the intent is to allow what is described here at the Fleet Server level.

https://github.com/elastic/fleet-server/blob/d591423a489d45c383f8c2b32c4211fe9046a3cf/internal/pkg/api/verConst.go#L17-L24

func BuildVersionConstraint(verStr string) (version.Constraints, error) {
	ver, err := version.NewVersion(verStr)
	if err != nil {
		return nil, err
	}
	verStr = maximizePatch(ver)
	return version.NewConstraint(fmt.Sprintf(">= %s, <= %s", MinVersion, verStr))
}

The agent doesn't enforce any version restrictions itself, it simply reports its current version to Fleet Server and acts on the response it receives.

[rodrigc]

What logic does the Kibana UI use for greying out the available upgrade?

I tried looking at the code in:

• useAgentVersion()
• sendGetAgentsAvailableVersions()

but didn't have time to follow it all

[rodrigc]

Maybe this line in Kibana is kicking out availalbe versions where MAJOR and MINOR are the same, but PATCH is different:

kibana/x-pack/plugins/fleet/public/hooks/use_agent_version.ts

Line 36 in 1fb81fb

return semverLt(version, kibanaVersion);

          agentVersionToUse =
            availableVersions.find((version) => {
              return semverLt(version, kibanaVersion);
            }) || availableVersions[0];

[juliaElastic]

I see your point, I think we can then easily change this logic in Fleet UI to allow upgrading agents to the latest patch on the same major.minor as kibana.

[juliaElastic]

@kpollich @jen-huang Could we add this to next sprint? It's a quick fix.

[kpollich]

Added to sprint 20

[joshdover]

While we're fixing this, it would be nice to just go ahead and add support for the upcoming "build releases" to Fleet UI / API at the same time. They will have the version suffix that will match this regex \+build[0-9]+$. For example, 8.11.0+build123456789. Fleet should allow this in the validation and retain the full version number when sending to Fleet Server in the action document, similar to how we handle -SNAPSHOT versions now.

[harshitgupta-qasource]

Hi Team,

We have created 03 testcases under testrail for this feature under Fleet test suite at links:

• Users are able to upgrade elastic agents from 8.x.0 to 8.x.1 in kibana stack 8.x.0.
• Users are able to schedule upgrade elastic agents from 8.x.0 to 8.x.1 in kibana stack 8.x.0
• Users are able to upgrade standalone elastic agents from 8.x.0 to 8.x.1 in kibana stack 8.x.0.

Please let us know if any other scenario needs to be added from our end.

Thanks!

[harshitgupta-qasource]

Hi Team,

We have executed 03 testcases under the Feature test run for the 8.12.1 release at the link:

• Patch Dependent

Status:

• PASS: 03

Build details:
VERSION: 8.12.1
BUILD: 70228
COMMIT: 3457f32

As the testing is completed on this feature, we are marking this as QA:Validated.
Please let us know if anything else is required from our end.

Thanks