Fix double fetch

[andrewvc]

Fixes #35646 by only unpacking project monitors once.

This fixes the ever growing temp folder issue and is more efficient to boot. Previously we would call fetch on a monitor source every time it was run, but only cleanup the fetched resource once, when the monitor was unloaded. We now fetch once and cleanup once.

This project also fixes the very confusing issue of two files browser/project.go and browser/source/project.go, we have renamed browser/project.go to browser/sourcejob.go which makes reasoning about this change simpler.

Manual testing

Test with a config like this one by placing it in your monitors.d folder in x-pack/heartbeat.

Then, in your heartbeat.yml enable config reloading. Finally, test by running:

mage build && ELASTIC_SYNTHETICS_CAPABLE=true ./heartbeat -e 2>&1 -E cloud.id=$CLOUD_ID -E cloud.auth=$CLOUD_AUTH -E output.elasticsearch.allow_older_versions=true -E logging.level=debug | jq .message | grep "browser project"

You should see logs like the following:

"browser project: unpack source: /var/folders/pl/wjbh5yrd2dz19mlwpckxrvdm0000gn/T/elastic-synthetics-unzip-1594999618"
"browser project: re-use already unpacked source: /var/folders/pl/wjbh5yrd2dz19mlwpckxrvdm0000gn/T/elastic-synthetics-unzip-1594999618"
"browser project: re-use already unpacked source: /var/folders/pl/wjbh5yrd2dz19mlwpckxrvdm0000gn/T/elastic-synthetics-unzip-1594999618"
// Change made to config file here forcing reload
"browser project: close project source: /var/folders/pl/wjbh5yrd2dz19mlwpckxrvdm0000gn/T/elastic-synthetics-unzip-1594999618"
"browser project: unpack source: /var/folders/pl/wjbh5yrd2dz19mlwpckxrvdm0000gn/T/elastic-synthetics-unzip-2539246959"
"browser project: re-use already unpacked source: /var/folders/pl/wjbh5yrd2dz19mlwpckxrvdm0000gn/T/elastic-synthetics-unzip-2539246959"

You should then be able to editor your config in monitors.d, perhaps changing the schedule field, etc, forcing it to reload, and see the temp directories used by heartbeat changing, but that only one at a time exists, with the old ones being cleaned up.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @andrewvc? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-22T00:41:45.955+0000

• Duration: 41 min 2 sec

Test stats 🧪

Test	Results
Failed	0
Passed	285
Skipped	0
Total	285

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[emilioalvap]

@andrewvc, can we defer this assignment until the source is successfully extracted? If it fails for a transient error, as it is it will never retry

[andrewvc]

I've done so, but it's unlikely to be transient as we're just unpacking a file, though I suppose a full disk could be freed (unlikely on a container however).

[emilioalvap]

@andrewvc I hope you don't mind, I pushed some fixes to unit tests since I was testing locally

[emilioalvap]

LGTM