Fleet-server stucks in updating state when Kibana upgraded from 7.13 > 7.16

[dikshachauhan-qasource]

Kibana version:* 7.16 bc6 Kibana cloud-staging environment

Browser version: Chrome
Host OS: n/a

Preconditions:

1. 7.16 BC6 Kibana cloud-staging environment should be available.

Build Details:

BUILD 46139
COMMIT a35f543b45a71a3029c7b5e3624081b27532207f
Artifact link: https://staging.elastic.co/7.16.0-b878d709/summary-7.16.0.html

2. Create new policy after each upgrade having few integrations in it.

Steps to reproduce:

• Create Kibana deployment with 7.13 version.
• Upgrade same to 7.14.2 after performing pre-requisite 2.
• Now wait till new fleet-server is available.
• Repeat steps 2 and 3 till kibana upgrade reaches to 7.16.
• Observe at deployment page, no errors are recorded in activity logs.
• However at fleet side, latest fleet server stuck in updating state.

Expected result:

7.16 Fleet server should be available in healthy state.

Screenshot:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[dikshachauhan-qasource]

cc @EricDavisX

[dikshachauhan-qasource]

Hi @EricDavisX

We have validated that even if we upgrade Elastic stack from 7.13 to 7.16 same is reproducible.

Please let us know if more is required from our end.

Thanks
QAS

[kpollich]

This should be fixed by #119925, which should land in 7.16.1. I'm not sure if/when this will land in a 7.16 BC, as I'm not familiar with that process.

[EricDavisX]

Hi - I see the backport PR for 119925 is here #120015 and was aimed at 7.16, so it should be picked up in the 7.16 build today for the next BC (o the current 7.16 snapshot). I can confirm the build commit cited in test is from Nov 24 which is before the relating changes were merged so the above is expected until we get re-test results.

fyi:
edavis-mbp:kibana_elastic edavis$ git show -s a35f543
commit a35f543
Date: Wed Nov 24 09:55:03 2021 +0100

That PR has really great steps for testing, Kyle - thanks for that, btw. :)

Assigned to Diksha to re-test tomorrow or asap.

[dikshachauhan-qasource]

Hi @EricDavisX

We have revalidated this today and found it working fine on 7.16 BC7.

Build details:
BUILD: 46197
Commit: b7b724b

Screenshot:

Hence, closing this out.

Thanks
QAS