Improve overnight tests for remote-asset based caching

[Cynical-Optimist]

See original issue on GitLab
In GitLab by [Gitlab user @Qinusty] on Aug 24, 2020, 14:53

Background

Currently overnight-tests run a build of FDSDK against a persistent external cache (https://cache-test.buildstream.build:11002).

This test definition means that each run of tests may differ based on the state of the external cache state. This could lead to nightly tests passing due to a fully cached build state.

Task description

My proposed solution to this is as follows, however input via comments is expected.

• Clean this external server nightly to ensure that WRITE operations work as expected. (Or salt build)
• Perform a second build against the fully cached build to ensure that READ operations are also successful
• Apply a patch to the FDSDK sources and cause an incremental build.

Additionally, it would be ideal for these tests to be executed against both a bst-artifact-server deployment as well as a third party cache stack supporting buildstream i.e. buildbarn with bb-asset-hub.

Acceptance Criteria

• Buildstream CI must test the functionality of Buildstream to successfully cache builds using remote-asset/remote execution storage solutions.
• Buildstream should ensure compatibility with a defined version of a third party cache deployment is maintained via overnight tests. (Follow up of https://gitlab.com/BuildStream/buildstream/-/merge_requests/2037)

---

[Cynical-Optimist]

In GitLab by [Gitlab user @Qinusty] on Aug 24, 2020, 14:53

changed title from Improve overnight tests for remote-asset to Improve overnight tests for remote-asset{+ based caching+}

[Cynical-Optimist]

In GitLab by [Gitlab user @Qinusty] on Aug 24, 2020, 14:53

changed the description

[Cynical-Optimist]

In GitLab by [Gitlab user @jjardon] on Aug 24, 2020, 15:02

[Gitlab user @Qinusty] AFAIK bst-artifact-server is going away do I'm ok to simply replace current job to use an asset based server implementation

[Cynical-Optimist]

In GitLab by [Gitlab user @willsalmon] on Aug 24, 2020, 15:21

For quite a long time we had a issue were if the weak cache key already existed on the server then it would not update it as it used to ( in 1.4) but fail to push anything (master is now fixed and can push a new strict and updates the weak)

I think your proposed 3 pass approach should catch this, and the 3 pass model should not be too long as the second two should mostly just pull and not build.