Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Heartbeat] The /tmp folder doesn’t seem to be getting cleaned up #35646

Closed
paulb-elastic opened this issue Jun 1, 2023 · 5 comments · Fixed by #35843
Closed

[Heartbeat] The /tmp folder doesn’t seem to be getting cleaned up #35646

paulb-elastic opened this issue Jun 1, 2023 · 5 comments · Fixed by #35843
Assignees
Labels
bug Team:obs-ds-hosted-services Label for the Observability Hosted Services team

Comments

@paulb-elastic
Copy link
Contributor

We are observing that the /tmp folder within the running container of an Elastic Agent being used for a Private Location, is continually growing and may eventually cause the host to fill up.

This has been observed with both 8.7.0 and 8.8.0, where /tmpis growing ~40MB per day (on an example Private Location running 5 browser monitors, 3 of which run every 5 minutes, 2 of which run every 3 minutes).

Looking at /tmp on the running container, there are many elastic-synthetics-unzip-nnnn subfolders, for example:

drwxrwx--- 5 elastic-agent elastic-agent  4096 Jun  1 13:22 elastic-synthetics-unzip-2857552961
drwxrwx--- 5 elastic-agent elastic-agent  4096 Jun  1 14:22 elastic-synthetics-unzip-2865044517
drwxrwx--- 5 elastic-agent elastic-agent  4096 Jun  1 12:37 elastic-synthetics-unzip-2881487560
drwxrwx--- 5 elastic-agent elastic-agent  4096 Jun  1 14:27 elastic-synthetics-unzip-2914827838
drwxrwx--- 5 elastic-agent elastic-agent  4096 Jun  1 13:17 elastic-synthetics-unzip-2929753614
drwxrwx--- 5 elastic-agent elastic-agent  4096 Jun  1 14:07 elastic-synthetics-unzip-2955987070
drwxrwx--- 5 elastic-agent elastic-agent  4096 Jun  1 13:47 elastic-synthetics-unzip-3022192703
@paulb-elastic paulb-elastic added bug Team:obs-ds-hosted-services Label for the Observability Hosted Services team labels Jun 1, 2023
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (Team:Uptime)

@emilioalvap emilioalvap self-assigned this Jun 6, 2023
@paulb-elastic
Copy link
Contributor Author

Quick update, Private Location still running and can see /tmp folders aren't getting cleared up (e.g. folders still from 1st June and today, 6th June):

drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  5 06:37 elastic-synthetics-unzip-980335601
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  3 07:27 elastic-synthetics-unzip-980872639
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  6 02:57 elastic-synthetics-unzip-983399808
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  2 01:37 elastic-synthetics-unzip-984332218
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  6 06:12 elastic-synthetics-unzip-986934786
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  6 08:17 elastic-synthetics-unzip-987210611
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  4 21:42 elastic-synthetics-unzip-987343959
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  2 18:02 elastic-synthetics-unzip-989015961
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  3 22:37 elastic-synthetics-unzip-990759646
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  5 04:02 elastic-synthetics-unzip-991498596
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  5 04:27 elastic-synthetics-unzip-992381412
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  4 12:17 elastic-synthetics-unzip-992467994
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  2 07:17 elastic-synthetics-unzip-992949381
drwxrwx--- 5 elastic-agent elastic-agent 4096 Jun  1 17:42 elastic-synthetics-unzip-99495229

Which corresponds with the /tmp folder growing (up to ~188MB just now across ~4300 folders, compared with ~3.1MB on 1st June).

@andrewvc
Copy link
Contributor

andrewvc commented Jun 6, 2023

Hmmmm, I've just confirmed that file based reload does correctly reload the monitor with delete, I'll keep investigating. Perhaps it is only on agent that this fails (that's much harder to debug)

@emilioalvap
Copy link
Collaborator

@andrewvc heartbeat is not cleaning the unzipped folder for project monitors after every run. Considering it fetches the source for every run, it adds up. It doesn't seem to be an agent issue

@emilioalvap emilioalvap assigned andrewvc and unassigned emilioalvap Jun 8, 2023
andrewvc added a commit that referenced this issue Jun 22, 2023
Fixes #35646 by only unpacking project monitors once.

This fixes the ever growing temp folder issue and is more efficient to boot. Previously we would call fetch on a monitor source every time it was run, but only cleanup the fetched resource once, when the monitor was unloaded. We now fetch once and cleanup once.

This project also fixes the very confusing issue of two files browser/project.go and browser/source/project.go, we have renamed browser/project.go to browser/sourcejob.go which makes reasoning about this change simpler.
mergify bot pushed a commit that referenced this issue Jun 22, 2023
Fixes #35646 by only unpacking project monitors once.

This fixes the ever growing temp folder issue and is more efficient to boot. Previously we would call fetch on a monitor source every time it was run, but only cleanup the fetched resource once, when the monitor was unloaded. We now fetch once and cleanup once.

This project also fixes the very confusing issue of two files browser/project.go and browser/source/project.go, we have renamed browser/project.go to browser/sourcejob.go which makes reasoning about this change simpler.

(cherry picked from commit f7111dc)
andrewvc added a commit that referenced this issue Jun 22, 2023
Fixes #35646 by only unpacking project monitors once.

This fixes the ever growing temp folder issue and is more efficient to boot. Previously we would call fetch on a monitor source every time it was run, but only cleanup the fetched resource once, when the monitor was unloaded. We now fetch once and cleanup once.

This project also fixes the very confusing issue of two files browser/project.go and browser/source/project.go, we have renamed browser/project.go to browser/sourcejob.go which makes reasoning about this change simpler.

(cherry picked from commit f7111dc)

Co-authored-by: Andrew Cholakian <[email protected]>
@emilioalvap
Copy link
Collaborator

Moving to Done Done as it's been validate by @paulb-elastic and myself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants