feat(fs/azure): add support for Azure Blob Storage (FS Object Backend)

[erka]

This is a good description of the auth for Azure Blob Storage https://github.com/moby/buildkit?tab=readme-ov-file#azure-blob-storage-cache-experimental

fixes #2289

[github-actions]

Uffizzi Ephemeral Environment deployment-43121

☁️ https://app.uffizzi.com/github.com/flipt-io/flipt/pull/2538

📄 View Application Logs etc.

⏰ This Preview will be destroyed in 1 hours at: Mon Dec 18 20:04:37 UTC 2023

What is Uffizzi? Learn more!

[codecov]

Codecov Report

Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (3665590) 71.62% compared to head (03c03b4) 71.74%.

Files	Patch %	Lines
internal/storage/fs/azblob/azblob_fs.go	70.83%	17 Missing and 4 partials ⚠️
internal/storage/fs/azblob/store.go	77.35%	8 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2538      +/-   ##
==========================================
+ Coverage   71.62%   71.74%   +0.12%     
==========================================
  Files          85       87       +2     
  Lines        8066     8197     +131     
==========================================
+ Hits         5777     5881     +104     
- Misses       1940     1961      +21     
- Partials      349      355       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[markphelps]

Will give this a thorough 👁️ in the morning @erka ! Apologies was out of town this weekend/today. Looks great at a glance though! much appreciated!

[GeorgeMac]

One minor tidy up comment, otherwise, this is super awesome. Thanks for putting this together @erka 💪 Solid.

[markphelps]

Minor JSON config changes

Thank you @erka!!

[markphelps]

I was looking at how our AWS S3 bucket config is authenticated and it seems it supports the default ENV vars supported by the AWS S3 go client:

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

Actually, I think it uses the aws s3 client default access credentials chain: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials

I think we should do the same here for azure blob, and not pass access credentials via FLIPT_ env vars, but instead rely on the azure blob default credential chain: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential

[erka]

I was looking at how our AWS S3 bucket config is authenticated and it seems it supports the default ENV vars supported by the AWS S3 go client:

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

Actually, I think it uses the aws s3 client default access credentials chain: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials

I think we should do the same here for azure blob, and not pass access credentials via FLIPT_ env vars, but instead rely on the azure blob default credential chain: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential

Azure Blob Storage is a bit more complex and provides few ways to auth. There is a default which you mentioned above but there are few others. Please read https://pkg.go.dev/[email protected]/blob/azureblob#hdr-URLs about it.

I have no objectives with dropping FLIPT_ env vars and rely on ENV defined on go-cloud.

[markphelps]

I was looking at how our AWS S3 bucket config is authenticated and it seems it supports the default ENV vars supported by the AWS S3 go client:

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

Actually, I think it uses the aws s3 client default access credentials chain: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials
I think we should do the same here for azure blob, and not pass access credentials via FLIPT_ env vars, but instead rely on the azure blob default credential chain: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential

Azure Blob Storage is a bit more complex and provides few ways to auth. There is a default which you mentioned above but there are few others. Please read https://pkg.go.dev/[email protected]/blob/azureblob#hdr-URLs about it.

I have no objectives with dropping FLIPT_ env vars and rely on ENV defined on go-cloud.

👍🏻 so then we would remove the configuration options FLIPT_STORAGE_OBJECT_AZBLOB_ACCOUNT and FLIPT_STORAGE_OBJECT_AZBLOB_SHARED_KEY and instead check for AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY, matching the behavior of go cloud?

[erka]

@markphelps FLIPT_..._AZBLOB_ACCOUNT and FLIPT_..._SHARED_KEY should be removed and replaced with AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY.

I have excluded endpoint from json as it may leak account name if endpoint is explicitly defined.

BTW maybe s3 bucket should be excluded as well as it's globally unique.

[markphelps]

amazing!! thank you @erka . one minor suggestion.

BTW maybe s3 bucket should be excluded as well as it's globally unique.

I agree, if you dont want to do this in this pr we can do it in another

[markphelps]

looks great!! Thank you @erka ! I will update docs and this will go out in next release 🎉