Remote Zip URL support for heartbeat synthetics

[andrewvc]

Is your feature request related to a problem? Please describe.

Today users need to use some custom process of their own design to move files onto the same machine as heartbeat for synthetic suites. This won't work for cloud and is a waste of effort given that most people can / will use git.

Describe the solution you'd like

We should make it possible for heartbeat to download remote zip files (authenticated via basic auth / TLS). The syntax could be as simple as:

synthetic_suites:
- type: git
  url: https://github.com/elastic/synthetics/archive/master.zip
  username: myuser
  password: mypassword
  directory: examples

[vigneshshanmugam]

Thanks Andrew for kick starting this issue. Totally 💯 to the approach and we would need this to improve the DX. Just one idea, may be we can use api tokens instead of the username and password fields, one advantage is that the users can limit the scope of the tokens (read_repo, write_repo, etc.).

[paulb-elastic]

We had a discussion about this earlier, considering that the initial version might be best to get the suite files from an HTTP endpoint (e.g. ZIP download), as opposed to a git clone. This still delivers value in being able to get the suite assets on demand when the test runs, with the following benefits:

• Don’t need to download the git repository
  • There may be security concerns from users (even with the option to limit the folder to download), as the credentials/token could still potentially be used to download the full repository)
• Can use an HTTP HEAD request to identify if the asset has been updated since the last download
• The suite assets (that would be downloaded via HTTP) could be staged as part of the users’ build pipeline

[andrewvc]

@paulb-elastic I've updated the issue description to reflect the move to Zip over HTTP from git.

[paulb-elastic]

For reference, an example configuration in a heartbeat.docker.yml for a remote ZIP location downloading from GitHub, could be:

- type: browser
  name: synth-zip-location
  id: synth-zip-location
  schedule: '@every 120s'
  source:
    zip_url:
        url: "https://github.com/elastic/synthetics/archive/refs/heads/master.zip"
        folder: "examples/todos"
        target_directory: "/tmp/synthetics/suite"
        username: myuser
        password: mypassword

[justinkambic]

Was able to test this feature using the supplied example zip.