[MAC 12]: Agent failed to upgrade from 8.4.2 to 8.5.0 BC1 for MAC 12 agent using agent binary. #1298

amolnater-qasource · 2022-09-26T07:35:22Z

Kibana version: 8.5 BC1 Kibana cloud environment

Host OS: MAC12

Build details:
VERSION: 8.5 BC1
BUILD: 56595
COMMIT: 0d8de4df69f8084a94cdd9638d7de510813cb5ce
Artifact link: https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.4.2-darwin-x86_64.tar.gz

Preconditions:

8.5 BC1 Kibana cloud environment should be available.
Lower version MAC agent v8.4.2 should be installed.

Steps to reproduce:

Navigate to Fleet> Settings tab and add agent binary: https://staging.elastic.co/8.5.0-77585599/downloads/.
Navigate to Fleet>Agents tab.
Trigger agent upgrade for MAC agent.
Observe after 4-5 minutes agent went unhealthy and failed to upgrade to latest version.

Screenshot:

Logs:
elastic-agent-diagnostics-2022-09-26T07-02-28Z-00.zip

Expected Result:
Agent should upgrade from 8.4.2 to 8.5.0 for MAC 12 agent using agent binary.

The text was updated successfully, but these errors were encountered:

amolnater-qasource · 2022-09-26T07:35:33Z

@manishgupta-qasource Please review.

manishgupta-qasource · 2022-09-26T07:46:11Z

Secondary review for this ticket is Done

cmacknz · 2022-09-27T14:48:19Z

I don't actually see anything related to upgrades in the agent logs.

In elastic-agent-20220926-2.ndjson I see failures to download and verify the 8.4.2 Filebeat artifacts followed by an eventual success:

{"log.level":"error","@timestamp":"2022-09-26T07:00:46.309Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-09-26T03:00:46-04:00 - message: Application: filebeat--8.4.2[90e2a6ea-436c-405a-96f4-ea7eb7e6a69a]: State changed to FAILED: 2 errors occurred:\n\t* package '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz' not found: open /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz: no such file or directory\n\t* call to 'https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz' returned unsuccessful status code: 404\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-26T07:00:46.309Z","log.origin":{"file.name":"operation/operation_retryable.go","file.line":85},"message":"operation operation-fetch failed, err: 2 errors occurred:\n\t* package '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz' not found: open /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz: no such file or directory\n\t* call to 'https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz' returned unsuccessful status code: 404\n\n","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-26T07:00:46.309Z","log.origin":{"file.name":"application/managed_mode.go","file.line":274},"message":"could not recover state, error operator: failed to execute step sc-run, error: 2 errors occurred:\n\t* package '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz' not found: open /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz: no such file or directory\n\t* call to 'https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz' returned unsuccessful status code: 404\n\n: 2 errors occurred:\n\t* package '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz' not found: open /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz: no such file or directory\n\t* call to 'https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz' returned unsuccessful status code: 404\n\n, skipping...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:46.312Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":68},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:46.313Z","log.origin":{"file.name":"application/managed_mode.go","file.line":316},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:46.313Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":70},"message":"Metrics endpoint listening on: /Library/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///Library/Elastic/Agent/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:46.321Z","log.origin":{"file.name":"artifact/config.go","file.line":138},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://staging.elastic.co/8.5.0-77585599/downloads/\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:46.322Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is Zgj9TE4s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:46.322Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 3 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:47.425Z","log.origin":{"file.name":"http/downloader.go","file.line":307},"message":"download from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz completed in Less than a second @ +InfYBps","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:47.516Z","log.origin":{"file.name":"http/downloader.go","file.line":307},"message":"download from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:47.516Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":75},"message":"downloaded binary 'filebeat.8.4.2' into '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz' as part of operation 'operation-fetch'","ecs.version":"1.6.0"}

Similarly in elastic-agent-20220926-3.ndjson I see download failures:

{"log.level":"info","@timestamp":"2022-09-26T07:00:48.937Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 3 step(s)","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-26T07:00:49.310Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"error\": \"app filebeat--8.4.2-0a6d0f0c: operation 'operation-verify' failed to verify filebeat.8.4.2: 2 errors occurred:\\n\\t* fetching asc file from '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc': open /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: no such file or directory\\n\\t* fetching asc file from https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: call to 'https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc' returned unsuccessful status code: 404\\n\\n\"","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-26T07:00:49.310Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-09-26T03:00:49-04:00 - message: Application: filebeat--8.4.2[90e2a6ea-436c-405a-96f4-ea7eb7e6a69a]: State changed to FAILED: operation 'operation-verify' failed to verify filebeat.8.4.2: 2 errors occurred:\n\t* fetching asc file from '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc': open /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: call to 'https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc' returned unsuccessful status code: 404\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-26T07:00:49.311Z","log.origin":{"file.name":"operation/operation_retryable.go","file.line":85},"message":"operation operation-verify failed, err: operation 'operation-verify' failed to verify filebeat.8.4.2: 2 errors occurred:\n\t* fetching asc file from '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc': open /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: call to 'https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc' returned unsuccessful status code: 404\n\n","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-26T07:00:49.312Z","log.origin":{"file.name":"application/managed_mode.go","file.line":274},"message":"could not recover state, error operator: failed to execute step sc-run, error: operation 'operation-verify' failed to verify filebeat.8.4.2: 2 errors occurred:\n\t* fetching asc file from '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc': open /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: call to 'https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc' returned unsuccessful status code: 404\n\n: operation 'operation-verify' failed to verify filebeat.8.4.2: 2 errors occurred:\n\t* fetching asc file from '/Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc': open /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc: call to 'https://staging.elastic.co/8.5.0-77585599/downloads/beats/filebeat/filebeat-8.4.2-darwin-x86_64.tar.gz.asc' returned unsuccessful status code: 404\n\n, skipping...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:49.317Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":68},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-26T07:00:49.317Z","log.origin":{"file.name":"application/managed_mode.go","file.line":316},"message":"Agent is starting","ecs.version":"1.6.0"}

In elastic-agent-20220926.ndjson, I just see an unknown service command failing to run:

{"log.level":"error","@timestamp":"2022-09-26T06:55:32.057Z","log.origin":{"file.name":"emitter/controller.go","file.line":123},"message":"Failed to render configuration with latest context from composable controller: operator: failed to execute step sc-run, error: context canceled: context canceled","ecs.version":"1.6.0"}

cmacknz · 2022-09-27T18:55:26Z

@amolnater-qasource Is this reproducible, or did it only happen once?

amolnater-qasource · 2022-09-28T05:40:24Z

Hi @cmacknz
Thank you for looking into this.

We have revalidated this issue on latest 8.5.0 BC1 Kibana cloud environment with 02 different mac agents and found this issue still reproducible:

Agent failed to upgrade from 8.4.2 to 8.5.0 BC1 for MAC 12 agent using agent binary.

Build details:
VERSION: 8.5.0
BUILD: 56595
COMMIT: 0d8de4df69f8084a94cdd9638d7de510813cb5ce

Screenshots:

Logs:
elastic-agent-diagnostics-2022-09-28T05-28-23Z-00.zip

Please let us know if anything else is required from our end.
Thanks

cmacknz · 2022-09-29T19:11:40Z

Pulling into the current sprint and assigning to @michalpristas to investigate.

michalpristas · 2022-09-30T07:21:28Z

the error messages you have found are funny in a sense that they are after upgrade and still pointing to 8.4.2
you can see this from hash used in a path: /Library/Elastic/Agent/data/elastic-agent-d3eb3e/downloads/filebeat-8.4.2-darwin-x86_64.tar.gz where d3eb3e matches 8.4.2

something went wrong and was not reported or symlink change was performed incorrectly. looking at it

michalpristas · 2022-09-30T08:28:56Z

found the root cause , the archive content structure is different and breaking
issue introduced in this PR #714

as PR does not touch ChangeSymlinks I don't expect Upgrade to work in 8.5 as well meaning going from 8.5 up (havent tested)

Fixing this is not that easy. With change in structure we need to

fix changeSymlink to address new structure
provide backward compatible way (detect if we're linking to *.app and if so proceed down to executable
backport change in changeSymlink to 8.4 and update docs so customer is aware that the way to 8.5 goes through this emergency 8.4 release. we have few days to 8.4.3 still

michalpristas · 2022-09-30T08:48:26Z

agreed with @jlind23 that the best way to approach this will be

revert for 8.5
archive structure change in 8.6
upgrade handle change in 8.5 so it is capable of handling *.app archive structure

Revert PR 8.5: #1387
Revert PR main: #1388
Followup issue created: #1386

bradenlpreston · 2022-09-30T13:29:45Z

Can we consider not reverting the changes and making the necessary changes for 8.5? We have strategic customers waiting on this capability today.
@nimarezainia , @mukeshelastic

jlind23 · 2022-09-30T14:04:16Z

@bradenlpreston if we do not revert it, upgrading from any version to 8.5 would be failing due to this change.
Which means that we may have to do another 8.4.X to prepare the logic in order to accept this new feature.

But even if we do an 8.4.X, it means that we will have to recommend our users to do the following:
Any version > 8.4.X > 8.5.X
Because 8.4.X will be the only version with enough logic to allow an upgrade to 8.5.X

james-elastic · 2022-09-30T16:33:10Z

@jlind23 If customers stay on 8.4 then want to jump to 8.6/8.7 they'd still get into a broken state is that correct?

bradenlpreston · 2022-09-30T17:29:23Z

@james-elastic - I was thinking the same. In every scenario it seems there will be a required upgrade path. Which I think would mean a full stack upgrade and agent upgrade (twice.) Is that correct @jlind23

Also cc: @aleksmaus , @ferullo , and @crowens

aleksmaus · 2022-09-30T18:00:37Z

just wanted to add the note about another possible workaround as we discussed:
we could ship 8.5 with the migration/setup script/binary that would be uncompressed into the location that 8.4 symlink would point to,
for example

lrwxr-xr-x   elastic-agent -> data/elastic-agent-2f286f/elastic-agent

where the data/elastic-agent-2f286f/elastic-agent could be the script/binary that fixes up the symlink and restarts the service

all versions upgrade handles after 8.5 will already use the new app bundle path so the symlink will always point to the correct binary inside of the bundle and we can remove this temporary script from the distribution in the next versions

jlind23 · 2022-09-30T18:41:41Z

@bradenlpreston yes that is correct.

jlind23 · 2022-09-30T18:42:40Z

To what @aleksmaus suggested, 8.5 will stay be a mandatory release for all users before upgrading to a later one.

nimarezainia · 2022-09-30T18:48:14Z

@jlind23 What options exist in fixing this in 8.5 (per Michal's recommendations above)? Is it time? I appreciate that it may be risky but seems very reproducible.
We could ask the release to be extended.

aleksmaus · 2022-09-30T22:41:09Z

There is another possible solution to fix it for 8.5, talked to @jlind23 about this on Slack, will work on PR

amolnater-qasource · 2022-10-06T12:21:28Z

Hi Team
We have revalidated this issue by upgrading MAC agents on 8.5.0 BC3 kibana cloud environment and found it fixed now.

Observations:

MAC Agent upgraded successfully from 8.4.3 to 8.5.0 BC3 using agent binary.

Build details:
BUILD: 56932
COMMIT: 1bb0d052c8d6842b88665c8c489f3a2d4cf4b46a

Screenshot:

Hence, marking this issue as QA:Validated.
Thanks

ghost · 2022-11-22T10:58:48Z

`Bug Conversion`

Test-Case not required as this particular checkpoint is already covered in the following testcase:
- https://elastic.testrail.io/index.php?/cases/view/167037&group_by=cases:section_id&group_order=asc&display_deleted_cases=0&group_id=8423

Thanks!

amolnater-qasource added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team impact:high Short-term priority; add to current release, or definitely next. labels Sep 26, 2022

cmacknz added 8.6-candidate v8.5.0 labels Sep 27, 2022

cmacknz assigned michalpristas Sep 29, 2022

This was referenced Sep 30, 2022

Revert: Bundle elastic-agent.app for MacOS… #714 #1387

Merged

Revert: Bundle elastic-agent.app for MacOS… #714 #1388

Closed

aleksmaus mentioned this issue Oct 2, 2022

Fix: Agent failed to upgrade from 8.4.2 to 8.5.0 BC1 for MAC 12 agent using agent binary. #1392

Merged

2 tasks

jlind23 closed this as completed Oct 3, 2022

aleksmaus mentioned this issue Oct 3, 2022

[backport main] Fix: Agent failed to upgrade from 8.4.2 to 8.5.0 BC1 for MAC 12 agent using agent binary. #1401

Merged

2 tasks

amolnater-qasource added the QA:Ready For Testing Code is merged and ready for QA to validate label Oct 4, 2022

juliaElastic mentioned this issue Oct 4, 2022

[Fleet]: No appropriate agent upgrade failed message is available if agents fails to upgrade to the latest version. elastic/kibana#140936

Open

amolnater-qasource added QA:Validated Validated by the QA Team and removed QA:Ready For Testing Code is merged and ready for QA to validate labels Oct 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MAC 12]: Agent failed to upgrade from 8.4.2 to 8.5.0 BC1 for MAC 12 agent using agent binary. #1298

[MAC 12]: Agent failed to upgrade from 8.4.2 to 8.5.0 BC1 for MAC 12 agent using agent binary. #1298

amolnater-qasource commented Sep 26, 2022

amolnater-qasource commented Sep 26, 2022

manishgupta-qasource commented Sep 26, 2022

cmacknz commented Sep 27, 2022

cmacknz commented Sep 27, 2022

amolnater-qasource commented Sep 28, 2022

cmacknz commented Sep 29, 2022

michalpristas commented Sep 30, 2022

michalpristas commented Sep 30, 2022

michalpristas commented Sep 30, 2022 •

edited

Loading

bradenlpreston commented Sep 30, 2022

jlind23 commented Sep 30, 2022

james-elastic commented Sep 30, 2022

bradenlpreston commented Sep 30, 2022

aleksmaus commented Sep 30, 2022 •

edited

Loading

jlind23 commented Sep 30, 2022

jlind23 commented Sep 30, 2022

nimarezainia commented Sep 30, 2022

aleksmaus commented Sep 30, 2022

amolnater-qasource commented Oct 6, 2022

ghost commented Nov 22, 2022

[MAC 12]: Agent failed to upgrade from 8.4.2 to 8.5.0 BC1 for MAC 12 agent using agent binary. #1298

[MAC 12]: Agent failed to upgrade from 8.4.2 to 8.5.0 BC1 for MAC 12 agent using agent binary. #1298

Comments

amolnater-qasource commented Sep 26, 2022

amolnater-qasource commented Sep 26, 2022

manishgupta-qasource commented Sep 26, 2022

cmacknz commented Sep 27, 2022

cmacknz commented Sep 27, 2022

amolnater-qasource commented Sep 28, 2022

cmacknz commented Sep 29, 2022

michalpristas commented Sep 30, 2022

michalpristas commented Sep 30, 2022

michalpristas commented Sep 30, 2022 • edited Loading

bradenlpreston commented Sep 30, 2022

jlind23 commented Sep 30, 2022

james-elastic commented Sep 30, 2022

bradenlpreston commented Sep 30, 2022

aleksmaus commented Sep 30, 2022 • edited Loading

jlind23 commented Sep 30, 2022

jlind23 commented Sep 30, 2022

nimarezainia commented Sep 30, 2022

aleksmaus commented Sep 30, 2022

amolnater-qasource commented Oct 6, 2022

ghost commented Nov 22, 2022

Bug Conversion

michalpristas commented Sep 30, 2022 •

edited

Loading

aleksmaus commented Sep 30, 2022 •

edited

Loading

`Bug Conversion`