Investigate how to upload bench results to AWS

[Akirathan]

Motivation

GH has 90 days retention policy for all the artifacts. So the artifacts (bench-results.xml) that were created during benchmark workflow runs will be deleted after 90 days. This makes it difficult to maintain the bench_downloader script that generates the bench results website.

Let's investigate how difficult it is to upload benchmark results in the workflow run directly to AWS.

Issue created from comment:

Issues a warning if the user wants to download benchmarks that are expired according to the GH policy.

• The default retention policy for all the artifacts is 90 days. Cannot be changed.

If we need to, it is rather straightforward to add a step to the benchmark job that not only uploads the results as a GH artifact, but also as a file to S3 - we already have/had such functionality. On S3 we can store the results indefinitely, if we want to.

Upload to S3 would be very nice, that would solve a lot of problems. Do you know how to do that? Or refer me to the point when we used to do that?

I've been writing this using the Github Actions a long time ago. I think we are still doing this, but I'm less familiar with the new Rust CI structure - I think

enso/build/build/src/aws.rs

Lines 88 to 123 in b8edf6a

	pub async fn update_manifest(repo_context: &impl IsRepo, edition_file: &Path) -> Result {
	let bucket_context = BucketContext {
	client: aws_sdk_s3::Client::new(&aws_config::load_from_env().await),
	bucket: EDITIONS_BUCKET_NAME.to_string(),
	upload_acl: ObjectCannedAcl::PublicRead,
	key_prefix: Some(repo_context.name().to_string()),
	};
	let new_edition_name = Edition(
	edition_file
	.file_stem()
	.context("Edition file path is missing filename stem!")?
	.as_str()
	.to_string(),
	);
	ide_ci::fs::expect_file(edition_file)?;
	let manifest = bucket_context.get_yaml::<Manifest>(MANIFEST_FILENAME).await?;
	debug!("Got manifest index from S3: {:#?}", manifest);
	let (new_manifest, nightlies_to_remove) =
	manifest.with_new_nightly(new_edition_name, NIGHTLY_EDITIONS_LIMIT);
	for nightly_to_remove in nightlies_to_remove {
	debug!("Should remove {}", nightly_to_remove);
	}
	let new_edition_filename =
	edition_file.file_name().context("Edition file path is missing filename!")?;
	bucket_context
	.put(new_edition_filename.as_str(), ByteStream::from_path(&edition_file).await?)
	.await?;
	bucket_context.put_yaml("manifest.yaml", &new_manifest).await?;
	Ok(())
	}

may be a starting point but you may need to ask @mwu-tow for more details.

Originally posted by @radeusgd in #7599 (comment)

[JaroslavTulach]

@PabloBuchu mentioned to me there is going to be a batch execution of Enso scripts including access to some DB in Enso Cloud. I'd be good to eat our own dog food.

[mwu-tow]

@Akirathan I think this was covered by #9075?

[Akirathan]

@Akirathan I think this was covered by #9075?

@mwu-tow Yes, you are right. This issue is no longer relevant.