[Tracking/Request] Access AWS through CNCF credits

[nikimanoledaki]

Unblocks #1

What is needed?

We need to open a Servicedesk ticket to get access to AWS so that we can use the credits offered by AWS for the CNCF: https://github.com/cncf/servicedesk#how-can-i-use-the-computing-infrastructure-provided-by-the-cncf

Why is it needed?

This will enable the Green Reviews WG to create an S3 bucket to store the OpenTofu state for the Equinix Metal infrastructure & k3s-created cluster.

See also:

• Provision cluster and bootstrap flux #6
• Design doc discussion

Who can access this?

AWS access can be limited to the TAG ENV leads for now, which is what we are doing with the Equinix infrastructure as well.

[jeefy]

Ping @vielmetti -- does Equinix have an s3-compatible object storage offering that we can leverage here? It feels kinda weird and inefficient needing Provider A around to store state for Provider B.

[Callisto13]

Equinix does technically have s3 object storage, but it is not part of or tied into Metal (you can't just click around in your customer console and "get" object storage in the same way you can through AWS and the like). The s3 storage from Equinix is part of managed services, meaning it is not built or provided by the company. In this case the provider is Netapp, and the managed service offered through Equinix is basically StorageGrid. It is only available in the Netherlands, although I think another EU metro is going to be added "soon". The only way I can see to get it right now is via a sales call, and I believe Equinix will connect it all up with the customer's other services (like Equinix Fabric etc). I am not sure what the user experience is like, how the docs are, what the cost would be, or whether there is a fixed contract term (I am finding out these things), but my gut would say AWS would be an easier choice.

If you would like to stay within Metal, and don't mind the maintenance overhead, another option would be to deploy an extra Metal server which is optimised for storage (basically choose one with large disks) and run MinIO on it. Or you could run object storage in the kubernetes cluster itself, again with MinIO or similar.

@nikimanoledaki lmk if you need anything else 👍

[rossf7]

Thanks @Callisto13! Since the s3 storage is not part of Metal I also think AWS is going to be the easier choice.

Object storage is a better fit rather than a dedicated server because we just need to store the tofu state but we can't run it inside the cluster due to the usual chicken egg problem fun :)

[nikimanoledaki]

Thank you all for your valuable input! Access to AWS for S3 would be the simplest solution with the least amount of overhead, which would be best in our case given that this is a community-led project so time and effort are limited.

Unfortunately, object storage access currently blocks the cluster provisioning PR and the rest of the WG's pipeline implementation. 🙁

@jeefy @vielmetti, we would appreciate your feedback and help with unblocking this! Thank you 😊

[vielmetti]

There are a couple of tools that provide S3 compatible object storage. One or more of them might be appropriate for your needs. @Callisto13 referenced Minio - but I guess what I'd like to know first is how much storage you need (megabytes, gigabytes, terabytes)?

I am OOO until 12/11 and I know @jeefy is away but we hopefully can make some progress.

[rossf7]

but I guess what I'd like to know first is how much storage you need (megabytes, gigabytes, terabytes)?

@jeefy @vielmetti We only need a small amount of storage. Currently the tofu state file is 10kb in the S3 bucket I tested with. It might grow a bit but not above the megabytes range.

cc @nikimanoledaki

[vielmetti]

@rossf7 Is OpenTofu limited to only using the S3 protocol, or are there other file storage / file retrieval options available? Trying to understand what's possibly variable in the configuration, e.g. can this be done with a web server of some kind instead.

Very obviously "let's spin up a dedicated server to store a 10kb file" is not a reasonable option.

I am looking at https://opentofu.org/docs/language/settings/backends/configuration which appears to be the relevant docs for the relevant configs, and it looks like there's more than one useful option.

[vielmetti]

I reached out to the "OpenTofu Community" Slack to ask for someone there to weigh in on the backend configuration options question.

[rossf7]

Hi @vielmetti, thanks for reaching out to the OpenTofu Slack. I also joined there and good idea to see if there is another backend we can use.

[rossf7]

@vielmetti thanks again for reaching out in the OpenTofu Slack. The feedback was very helpful and we could use a cloud backend instead of S3. https://opentofu.org/docs/cli/cloud/settings

One of the Spacelift cofounders kindly offered we could use their service for free since we are a CNCF project. I'm going to try this out to see how it compares with using S3.

I've added an item to discuss this at the next WG meeting (Wed 13 December @ 17:00 CET).

[rossf7]

Hi @jeefy I investigated using Spacelift as an alternative and we discussed it during the WG meeting yesterday and with @vielmetti. We feel using an S3 bucket from AWS is a better solution.

We could use the Spacelift CLI to manage the state. However under the hood Spacelift uses an AWS S3 bucket and we would be adding a 3rd party service as a dependency for the WG.

With an AWS S3 bucket we can use the OpenTofu CLI directly. This aligns with the goal we have of using CNCF projects when possible for our stack.

cc @nikimanoledaki @leonardpahlke @AntonioDiTuri

[nikimanoledaki]

Thank you @rossf7 & @vielmetti for syncing and looking into this.

@jeefy we would really appreciate your help with unblocking this, please! Thank you.

[nikimanoledaki]

@leonardpahlke, hi! Could you report back with updates about the service desk ticket, please? Thank you for your help with unblocking this :)

[jeefy]

I'm getting out of the way lol. @idvoretskyi Would you mind tackling setting this up? :)

[idvoretskyi]

@jeefy sure!

[idvoretskyi]

@nikimanoledaki can you please assign this to me?

[nikimanoledaki]

Thank you for looking into this @jeefy & @idvoretskyi! 🙌

[nikimanoledaki]

@idvoretskyi which next steps would you recommend? Would it help if we opened a Service Desk issue to track this? :)

[jeefy]

Quick update, redirected this from @idvoretskyi to someone from LF IT to unblock the request, bucket should be created shortly. :) Thanks!

[ynwa99]

Hello @nikimanoledaki -- My name is Shah and I'm with LF IT. I created a bucket named tag-environmental-sustainability within the CNCF AWS account. By default the bucket blocks all public access. How would you like to best access the bucket? Are there any particular users or a group alias you want me to assign IAM permissions to for this bucket?

[leonardpahlke]

Thank you so much @jeefy and @ynwa99 ! Could we name the bucket tag-env-green-reviews-open-tofu to better capture the use case / purpose of the bucket.

[leonardpahlke]

@ynwa99 — regarding access @cncf-tags/tag-env-leads + @cncf-tags/tag-env-wg-green-reviews-leads is the group that could have access. Not sure how you usually maintain this list. For this useccase it may be a bit overhead. And it may be just easier to create a one technical user with just CLI access to this one bucket which we can use in our gitops worflow.

[ynwa99]

For access, when you say "one technical user", is there a specific person you had in mind?

[leonardpahlke]

Not really. We just need one user. This can be sort of a technical user. tag-environmental-sustainability-tech-user or similar & CLI creds.

[leonardpahlke]

I opened a service desk issue to get a 1Password account to store the credentials

[ynwa99]

Hi Leo, thanks for chatting with me on Slack a bit more about this request. I'm happy to say there is now an S3 bucket in the CNCF AWS account available for the TAG env sustainability team to use. The bucket name is tag-env-green-reviews-open-tofu and the username is tag-env-technical-user

How would you like me to share the access key credentials?

[leonardpahlke]

thanks! If thats ok, you can send me the creds via Slack DM. We will get a 1Password account for the TAG but that will likely take a bit. see cncf/tag-env-sustainability#336

[rossf7]

We've switched to use the new S3 bucket for the cluster. Thanks all for the help with this!