build: use http_archive instead of git_repository

[goaway]

Description: This will hopefully allow for both faster and more reliable builds. http_archive contains built-in retry logic, and pulls down smaller artifacts (no history).
Risk Level: Low
Testing: Local and CI

Signed-off-by: Mike Schore [email protected]

[keith]

FWIW git_repository also doesn't pull down history https://github.com/bazelbuild/bazel/blob/49ea9589f0255b0250cb4438cab88eceb91be62d/tools/build_defs/repo/git_worker.bzl#L52 and there is 1 (mostly theoretical but has happened once before) risk with using http_archive. The problem is that GitHub's tars are generated as needed, so when you lock the sha, if they update their version of tar, the sha could change. This happened at one point when they did an OS update on their machines. But realistically it's probably fine to ignore this and fix it if that happens.

[goaway]

Good points. Even if we are avoiding pulling down history (though my read of that code is that we're maybe still pulling down some unless shallow_since is set really precisely?), the main win (hopefully) here will be the built-in retry logic in http_archive helping us around the connection flakiness we've seen in azure/github actions.

Good callout on the potential sha issue. Something to keep an eye out for.

[keith]

We should be setting shallow_since because otherwise Bazel warns too. But I think this is safe to test!

[artem-zinnatullin]

I'd recommend not switching to http_archive without updating Bazel to 0.29.x, there were important connection/stream retry fixes in 0.29.x:

• Fix retrying of SocketTimeoutExceptions in HttpConnector bazelbuild/bazel#9008
• Fallback to next urls if download fails in HttpDownloader  bazelbuild/bazel#9015 (not going to help since we don't have fallback urls here)

[goaway]

Thanks! We're on 0.29.1 :)

I did consider listing the same URL multiple times, but read somewhere that it would retry (like 8 times?) regardless. (I didn't read the current source to confirm this.)

[artem-zinnatullin]

My bad, didn't notice you've updated Bazel yesterday lol

Listing same URL multiple times might work, but I also haven't checked if it dedups them before attempting the download

[goaway]

Thank you @keith and @artem-zinnatullin!