Make smoke tests more deterministic

[FredKSchott]

Changes

• Moves the actual download/setup/sync logic of our remote smoke tests to a nightly CI task. Instead of downloading on demand, we now download once nightly.
• Now smoke tests should pass or fail with more reliability, since all test contents will static and checked in across every PR and test run. Essentially, we're back to "local" behavior for test running.
• Uses the same "create PR" flow as our yarn.lock upgrade script.

Testing

• Is test. I will run the nightly job manually after merging to confirm that it worked.

Docs

• N/A

[changeset-bot]

⚠️ No Changeset found

Latest commit: dd355fc

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[natemoo-re]

I don't know if we actually want these files in the repo and showing up in diffs?

[FredKSchott]

If you think of the pros, it means that we can reliably reproduce our tests going all the way back in the git history, in case you need to track down when something broke. If a smoke test passes one day and fails the next, you can guarantee that it's because the code changed and not that there was some network issue, or that two people have two different sets of files downloaded on their machine. From experience, debugging that kind of thing can be a nightmare.

FWIW only the nightly job should ever be touching this directory in specific update commits that go directly onto main, so our own PR diffs should be relatively unaffected.

[natemoo-re]

I know I jokingly say this all the time, but this actually seems like a case for submodules? When people clone this repo, they'll just get an empty file unless they pass --recurse-submodules, which is what we'd do in CI. More about Submodules. Seems kinda like this is what they were designed for.

[FredKSchott]

Submodules would probably fix some bugs with our current implementation, but doesn't change the main pro/cons around determinism. Likewise, false positives/negatives are possible (likely?) in the current system due to network, fs, git, etc. Basically, if I see a smoke test failure (locally or CI) and you can't reproduce on your machine, there's a huge surface area to go debug whats wrong.

[natemoo-re]

I'm still not a big fan of this code being checked into our repo, especially since some contributors have already mentioned that the monorepo is difficult to clone on a slow connection. But I won't block this PR if you think this is the best approach for the time being.

[FredKSchott]

Lets try it out for a few weeks and if we don't like it I'm happy to revert. I agree that a longer initial download time is a pain but I have faith that the confidence boost + being able to turn smoke tests back on will be well worth it