Add support for Azure Blob Storage (FS Object Backend)

[GeorgeMac]

Update(15/12): I just reworked the abstractions in the storage/fs package. So this description has been updated to reflect that.

We should add support for pushing and pulling flag state via Azure Blob Storage (ABS).

https://azure.microsoft.com/en-gb/products/storage/blobs

Anyone interested in adding support for this is 💯 welcome to contribute!
We would be more than happy to help with advice and of course review.

Currently, we support pushing to AWS S3 via the object type storage backend (which is built ontop of the FS backends constructs). We should implement additional support for ABS in the same manner.

Implementation Details

1. New fs.FS implementation over a ABS container
2. New fs.SnapshotStore implementation over the new absfs.FS
3. Add configuration for this new object subtype
4. Add abs.SnapshotStore to newObjectStore
5. Add integration tests

Implement new `fs.FS` implementation

The purpose of this is expose the contents of a particular ABS container via the fs.FS interface.
This is what the snapshotter expects to consumer when it makes an in-memory snapshot of flag state.
It will traverse this fs.FS like a local directory in search of Flipt flag state.

S3 Bucket reference implementation:

flipt/internal/s3fs/s3fs.go

Lines 21 to 34 in 1326c47

	// FS is only for accessing files in a single bucket. The directory
	// entries are cached. It is specifically intended for use by a source
	// that calls fs.WalkDir and does not fully implement all fs operations
	type FS struct {
	logger *zap.Logger
	s3Client S3ClientAPI
	// configuration
	bucket string
	prefix string
	// cached entries
	dirEntry *Dir
	}

Implement new `fs.SnapshotStore` implementation

The fs.SnapshotStore interface is consumed by the fs.Store to access read-only snapshot views of the underlying state. It is this snapshot stores responsbility to:

1. Establish an initial set of state (or fail early).
2. Start any background processes for keeping state up to date (currently all strategies rely on polling).

S3 reference implementation:

flipt/internal/storage/fs/s3/store.go

Lines 20 to 35 in 1d70e15

	// SnapshotStore represents an implementation of storage.SnapshotStore
	// This implementation is backed by an S3 bucket
	type SnapshotStore struct {
	logger *zap.Logger
	s3 *s3.Client
	mu sync.RWMutex
	snap storage.ReadOnlyStore
	endpoint string
	region string
	bucket string
	prefix string
	pollOpts []containers.Option[storagefs.Poller]
	}

Define new configuration

In order to configure a client, source and fs we will need connection details, authentication and other ABS configuration; we will need to make space for it in the configuration of Flipt.
This should be done in a consistent style to that of the S3 implementation.

Note: this will also require updates to our CUE and JSON schema for our configuration.

S3 reference implementation:

flipt/internal/config/storage.go

Lines 136 to 143 in 1326c47

	// S3 contains configuration for referencing a s3 bucket
	type S3 struct {
	Endpoint string `json:"endpoint,omitempty" mapstructure:"endpoint" yaml:"endpoint,omitempty"`
	Bucket string `json:"bucket,omitempty" mapstructure:"bucket" yaml:"bucket,omitempty"`
	Prefix string `json:"prefix,omitempty" mapstructure:"prefix" yaml:"prefix,omitempty"`
	Region string `json:"region,omitempty" mapstructure:"region" yaml:"region,omitempty"`
	PollInterval time.Duration `json:"pollInterval,omitempty" mapstructure:"poll_interval" yaml:"poll_interval,omitempty"`
	}

Wire up new object store type

Add the new SnapshotStore implementation to the newObjectStore function to wire it in.

S3 reference implementation:

flipt/internal/storage/fs/store/store.go

Lines 142 to 160 in 1d70e15

	case config.S3ObjectSubStorageType:
	opts := []containers.Option[s3.SnapshotStore]{
	s3.WithPollOptions(
	storagefs.WithInterval(objectCfg.S3.PollInterval),
	),
	}
	if objectCfg.S3.Endpoint != "" {
	opts = append(opts, s3.WithEndpoint(objectCfg.S3.Endpoint))
	}
	if objectCfg.S3.Region != "" {
	opts = append(opts, s3.WithRegion(objectCfg.S3.Region))
	}
	snapStore, err := s3.NewSnapshotStore(ctx, logger, objectCfg.S3.Bucket, opts...)
	if err != nil {
	return nil, err
	}
	return storagefs.NewStore(snapStore), nil

Add integration tests

We should see if we can setup integration tests in the same way as has been done with Minio and the S3 implementation.

S3 reference implementation:

flipt/build/testing/integration.go

Lines 209 to 239 in 1326c47

	func s3(ctx context.Context, client *dagger.Client, base, flipt *dagger.Container, conf testConfig) func() error {
	minio := client.Container().
	From("quay.io/minio/minio:latest").
	WithExposedPort(9009).
	WithEnvVariable("MINIO_ROOT_USER", "user").
	WithEnvVariable("MINIO_ROOT_PASSWORD", "password").
	WithExec([]string{"server", "/data", "--address", ":9009"})
	_, err := base.
	WithServiceBinding("minio", minio).
	WithEnvVariable("AWS_ACCESS_KEY_ID", "user").
	WithEnvVariable("AWS_SECRET_ACCESS_KEY", "password").
	WithExec([]string{"go", "run", "./build/internal/cmd/minio/...", "-minio-url", "http://minio:9009", "-testdata-dir", testdataDir}).
	Sync(ctx)
	if err != nil {
	return func() error { return err }
	}
	flipt = flipt.
	WithServiceBinding("minio", minio).
	WithEnvVariable("FLIPT_LOG_LEVEL", "DEBUG").
	WithEnvVariable("AWS_ACCESS_KEY_ID", "user").
	WithEnvVariable("AWS_SECRET_ACCESS_KEY", "password").
	WithEnvVariable("FLIPT_STORAGE_TYPE", "object").
	WithEnvVariable("FLIPT_STORAGE_OBJECT_TYPE", "s3").
	WithEnvVariable("FLIPT_STORAGE_OBJECT_S3_ENDPOINT", "http://minio:9009").
	WithEnvVariable("FLIPT_STORAGE_OBJECT_S3_BUCKET", "testdata").
	WithEnvVariable("UNIQUE", uuid.New().String())
	return suite(ctx, "readonly", base, flipt.WithExec(nil), conf)
	}

Useful Links

• Original PR for S3 implementation feat(fs/s3): Initial support for s3 filesystem backend #1900
• Go Azure Blob Storage client: https://github.com/Azure/azure-storage-blob-go

[markphelps]

one alternative is we could use Go Cloud Blob SDK which would allow us to work with all the big 3 blob stores. it wraps their specific impls with a common API.

I had success with this library in a previous role

[GeorgeMac]

one alternative is we could use Go Cloud Blob SDK which would allow us to work with all the big 3 blob stores. it wraps their specific impls with a common API.

I had success with this library in a previous role

de ja vu! Great shout.

[markphelps]

I just really want you to know about Go Cloud Blob SDK

Go Cloud Blob SDK

[GeorgeMac]

This message is sponsored by Go Cloud Blob SDK