Skip to content
This repository has been archived by the owner on Oct 22, 2021. It is now read-only.

feat: use generic fog provider for external blobstore #1531

Merged
merged 2 commits into from
Nov 25, 2020

Conversation

jandubois
Copy link
Member

@jandubois jandubois commented Oct 29, 2020

This PR implements 2 features:

  • Generalize external blobstore support to S3 compatible endpoints, to Google and Azure cloud storage.
  • Configure Eirini to use external blobstore.

The PR removes the old "s3" provider and replaces it with "fog" which provides a superset of capabilities. The JSON schema will make sure that attempts to use "s3" will get an error.

This PR supports the simple configurations from https://docs.cloudfoundry.org/deploying/common/cc-blobstore-config.html (specifying provider, keys, and bucket names), plus some extra settings for non-AWS versions of S3 compatible storage. Things like encryption or service accounts etc. are not implemented, as they all would require mapping for bits-service/eirini. Plain webdav is also not supported, except via the singleton-blobstore.

This PR has been tested with a recent local MinIO server, using both path style addressing (Diego) and vhost addressing (Eirini):

  blobstore:
    provider: fog
    fog:
      connection:
        provider: AWS
        aws_access_key_id: MY-KEY-ID
        aws_secret_access_key: MY-SECRET-KEY
        region: ""
        aws_signature_version: '2'
        endpoint: http://minio.home:9000
        # path_style: true

      app_package_directory_key: cap-staging-blobstore-packages
      buildpack_directory_key: cap-staging-blobstore-buildpacks
      droplet_directory_key: cap-staging-blobstore-droplets
      resource_directory_key: cap-staging-blobstore-resources

Fixes #1540

@jandubois jandubois force-pushed the jandubois/fog-blobstore branch 2 times, most recently from 97f832b to ec9342f Compare November 3, 2020 03:06
@jandubois
Copy link
Member Author

Requires cloudfoundry-incubator/quarks-operator#1220 to be able to store fog_connection details in a secret instead of inlining them in an ops config map.

@jandubois jandubois force-pushed the jandubois/fog-blobstore branch 2 times, most recently from 9166918 to 0ad0df4 Compare November 3, 2020 20:20
@jandubois jandubois added the changelog Issue must be present in the release notes. label Nov 3, 2020
@jandubois jandubois force-pushed the jandubois/fog-blobstore branch from 0ad0df4 to 7ac4029 Compare November 3, 2020 23:43
Example configurations for Azure, AWS, Google Cloud can be found
commented out in chart/values.yaml.
@jandubois jandubois force-pushed the jandubois/fog-blobstore branch from 7ac4029 to 03f703c Compare November 24, 2020 21:15
@jandubois jandubois marked this pull request as ready for review November 24, 2020 21:16
@jandubois jandubois requested a review from mook-as November 24, 2020 21:17
scripts/image_list.rb Show resolved Hide resolved
chart/templates/_capi.tpl Show resolved Hide resolved
chart/templates/_capi.tpl Outdated Show resolved Hide resolved
chart/templates/_capi.tpl Show resolved Hide resolved
chart/templates/bosh_deployment.yaml Show resolved Hide resolved
chart/templates/fog-blobstore.yaml Outdated Show resolved Hide resolved
chart/values.schema.yaml Show resolved Hide resolved
# Additional settings for e.g. MinIO
# ----------------------------------
# aws_signature_version: '2'
# endpoint: S3-ENDPOINT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# endpoint: S3-ENDPOINT
# endpoint: https://custom-s3-endpoint.example.com/

Or whatever the correct syntax is (to show what the correct syntax is).

chart/values.schema.yaml Show resolved Hide resolved
Copy link
Contributor

@mook-as mook-as left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍👍👍

@jandubois jandubois merged commit 522a866 into master Nov 25, 2020
@jandubois jandubois deleted the jandubois/fog-blobstore branch November 25, 2020 01:25
@gaktive gaktive added this to the 2.7.0 milestone Dec 11, 2020
@gaktive gaktive added the Type: Enhancement New feature or request label Dec 11, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
changelog Issue must be present in the release notes. suse-cap Type: Enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

External blobstore with object store implemenatations that are compliant with AWS S3 API
3 participants