Successful upgrades should cleanup old downloads

[joshdover]

After an upgrade is successful, the downloaded files are not deleted in Elastic/Agent/data/elastic-agent-*/downloads. This can lead to several GB of unneeded data being preserved on disk after several subsequent upgrades.

We should delete the downloads files to save space once an upgrade has successfully completed.

[pierrehilbert]

Won't it be better to clean it in any case?
Correct me if I'm wrong but I understood that if files are there but empty, the upgrade will fail instead of trying to download them again.

[joshdover]

Won't it be better to clean it in any case?
Correct me if I'm wrong but I understood that if files are there but empty, the upgrade will fail instead of trying to download them again.

Agreed, to also solve #693 we should probably do:

• On a new upgrade attempt, delete any files previously downloaded
• After a successful upgrade, delete any unneeded files previously downloaded

[michel-laterman]

So to itemize all scenarios across this and #693:

Assuming we're upgrading from 8.2.0 -> 8.X

• when an upgrade starts:
  • delete all artifacts in downloads/ that are NOT 8.2.0
• If an upgrade fails:
  • delete all 8.X artifacts (should we instead only delete the artifacts if the download or verification step fails?)
• If the upgrade succeeds:
  • delete all contents from 8.2.0's downloads dir?

[pierrehilbert]

@joshdover: what do you think about @michel-laterman proposal?

[WiegerElastic]

To chime in:

Would it be an idea to keep N-1 on the machine? Ideally, we should be able to revert to the previous version. Given the problems we are seeing with failed downloads, it would be worth considering keeping the install files from the latest success full install on the machine so there is a more smooth way of reverting. It could also save some money in the long run, because we don't need to download those files over and over again.

Example:

Agent version 8.3.1. has been installed successfully. That means that everything older than (in this case) 8.3.0 can be removed.

The version is upgraded from 8.3.1. to 8.3.2. That means that everything older than 8.3.1. can be removed.

Please note that at this point in time, there isn't a way for Agent or Fleet to revert back to a previous version using files that are already on the machine. Also, Fleet would need to be able to check the integrity of the install files on the machine before installing them (but that shouldn't be a problem since we are already working with hashes). Then again, it would be worth thinking about this now in case we need it in the future.

If an upgrade fails:

delete all 8.X artifacts (should we instead only delete the artifacts if the download or verification step fails?)

The response capabilities that are coming in 8.4. can also help out here. Right now, customers don't have a way of interacting with the filesystem on which Agent is installed. Internally, we solve this by executing local commands through Endgame or doing a clean-up using Jamf/InTune. Once Agent has this capability, removing failed downloads from an operator perspective will become much easier.

[aarju]

• If an upgrade fails:
  • delete all 8.X artifacts (should we instead only delete the artifacts if the download or verification step fails?)

For failed upgrades I think it should be enough to delete the downloaded files only if the checksum fails. If the checksum succeeds then the failed upgrade is likely caused by some other reason.

[amolnater-qasource]

Hi @joshdover
While testing agent upgrades on 8.4 Snapshot we have observed that:

• Older folder of elastic-agent gets removed on successful agent upgrade.
• No Old version artifacts are available under Elastic/Agent/data/elastic-agent-*/downloads.

Query:
Could you please confirm if we expect newer folder should also remove any latest version files after upgrade under Elastic/Agent/data/elastic-agent-*/downloads?

Screenshot:

Build details:

BUILD: 54585
COMMIT: f9e2ed4d9f38424676a558c34b74f0a031746c9e

Please let us know if we are missing anything here.
Thanks