roachtest: longer term archiving of artifacts

[herkolategan]

Currently, we only store logs and other debugging info (artifacts) from a roachtest for a limited time on TeamCity.
Once a build reaches a certain age (~1 month) the artifacts are lost.

This issue is a proposal to store the data longer, on cloud storage, in a bucket.

The aim is not to store data for all tests, but rather a subset that has the following properties:

• The test failed
• It didn't fail because of an infra flake, timeout, or VM preemption
• Has a small footprint in terms of storage required, for example less than 100MB.
• A daily / weekly limit of storage has not been reached (could cap the total combined amount from all tests)

Storing this data would allow us to build up training data for classifying failures.
In the future we should be able to gather all the issues and resulting discussions and combine it with data from the test.
This might be useful to train an LLM to provide assistance during triages.

Proposed solution:
There should be some mechanism that detects the test adhered to the requirements for longer storage. The data should then be backed up to cloud storage and linked to the issue, maybe with a label on the issue indicating that it has been selected for long term storage.

Additional ideas:
Maybe allow adding a label on an issue that will invoke long term storage on that test if done in the initial storage phase. This would require some external process to detect the label and copy the data from TeamCity to longer storage.

Jira issue: CRDB-44966

[blathers-crl]

cc @cockroachdb/test-eng