Storage Backend Implementation using Stow #189
Conversation
| @@ -50,8 +70,57 @@ type StorageBackend struct { | |||
| S3 *S3 | |||
| } | |||
|
|
|||
| func (s *StorageBackend) GetConfigMap() stow.ConfigMap { | |||
There was a problem hiding this comment.
Why is this struct with the same case statements in every method?
Sounds like this should be an interface with different implementations for each backend.
There was a problem hiding this comment.
Yup, I have created a BackendConfigurer interface which is implemented by all configured backends to return the config to setup stow.
server/jobs/job_store.go
Outdated
| delete(j.jobs, jobID) | ||
| } else if !errors.Is(err, &StorageBackendNotConfigured{}) { |
There was a problem hiding this comment.
You wouldn't need this error check if you were using an interface with different implementations.
There was a problem hiding this comment.
Decided to use a two value return statement in the Write() method. So, we will no longer have to look for custom errors to decide if we want to delete the job from memory.
server/jobs/storage_backend.go
Outdated
| } | ||
|
|
||
| containerFound = true | ||
| _, err = container.Put(key, reader, 100, nil) |
There was a problem hiding this comment.
It's the size of the reader according to the documentation. I've updated the GetReader() method to return the size if the reader which is passed into this method call.
This PR uses stow to create a generic storage backend that can read and write from s3, azure blob storage, local disk storage etc.