[Agent] When the Agent Upgrade feature is used, Agent doen't use the new version until after a reboot

[EricDavisX]

This is with the 7.10 BC2 build that was re-signed by Infra to help us continue testing.

Using a 7.11 snapshot kibana and the *just now re-signed 7.11 snapshot.
Kibana snapshot commit: a6a82aee0dfc9f5a1e2b4173b3154e2236ffb330

It is a Centos 8 system and I installed the .tar.gz Agent package with the 'install' with no errors prior (and no errors show in the log at all, actually).

The only problem I see is that after posting the API call to upgrade the Agent, it comes back on-line citing it is still the 7.10 version. I got a tip from Michal that if I restarted (the host I assumed) it would pick and use the new Agent, which it did.

So, I think it isn't critical blocker urgent but would be good to get in to the 7.x and 8 lines and the 7.10.1 that will come if we can.

7.10 end of log snippet:
{"log.level":"info","@timestamp":"2020-10-16T15:38:49.293-0400","log.origin":{"file.name":"process/app.go","file.line":161},"message":"Signaling application to stop because of shutdown: filebeat--7.10.0--36643631373035623733363936343635","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-10-16T15:38:49.293-0400","log.origin":{"file.name":"application/managed_mode.go","file.line":306},"message":"Agent is stopped","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-10-16T15:38:49.293-0400","log.origin":{"file.name":"cmd/run.go","file.line":143},"message":"Shutting down completed.","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-10-16T15:38:49.294-0400","log.logger":"control","log.origin":{"file.name":"server/listener.go","file.line":52},"message":"Failed to cleanup /run/elastic-agent.sock: remove /run/elastic-agent.sock: no such file or directory","ecs.version":"1.5.0"}

screenshots and logs:

7.10 agent log
7.10-agent.log

7.11 agent log
7.11-agent.log

to test this I used the Kibana API, and this one 7.11 build is the only one that I know of that is signed correctly so that we have any hope of testing this:
/api/fleet/agents/44208410-39c0-484e-825f-f0e2c6daasdf/upgrade

{"source_uri":"https://snapshots.elastic.co/7.11.0-82af3740/downloads/","version":"7.11.0-SNAPSHOT"}

link to specific agent:
https://snapshots.elastic.co/7.11.0-82af3740/downloads/beats/elastic-agent/elastic-agent-7.11.0-SNAPSHOT-linux-x86_64.tar.gz

macOS:
https://snapshots.elastic.co/7.11.0-82af3740/downloads/beats/elastic-agent/elastic-agent-7.11.0-SNAPSHOT-darwin-x86_64.tar.gz

windows x64 - this one may not be great to test with due to other current build errors, but here in the link anyhow:
https://snapshots.elastic.co/7.11.0-82af3740/downloads/beats/elastic-agent/elastic-agent-7.11.0-SNAPSHOT-windows-x86_64.zip

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[EricDavisX]

I've confirmed this is working in a 7.11 build, upgraded through the ui with a 7.10 Agent.