S3 backend should not be AWS-specfic

[jpds]

Feature Request:

The S3 storage backend in Vault should not be specifically tied to AWS itself. There are other technologies out there which provide S3 compatible services (Ceph RADOS GW, Minio) which interface with standard commands such as s3cmd.

[jpds]

Related: #2274.

[jpds]

The documentation here is also really confusing: https://www.vaultproject.io/docs/configuration/storage/s3.html

endpoint (string: "") – Specifies an alternative, AWS compatible, S3 endpoint. This can also be provided via the environment variable AWS_S3_ENDPOINT.

region (string "us-east-1") – Specifies the AWS region. This can also be provided via the environment variable AWS_REGION or AWS_DEFAULT_REGION, in that order of preference.

Endpoint suggests that it is possible to choose an alternative, however, I configure Vault with:

 storage:
      s3:
        endpoint: "http://url/"
        access_key: "..."
        secret_key: "..."
        bucket: "vault-test"
        region: ""

And then Vault starts up with:

Error initializing storage of type s3: unable to access bucket 'vault-test' in region us-east-1: RequestError: send request failed
caused by: Get http://vault-test.url/: dial tcp: lookup vault-test.url on dns:53: no such host

[jefferai]

As far as I'm aware it's usable with non-S3 endpoints. That error is indicating that DNS lookup failed for the host you gave.

[jpds]

I'm actually referencing the host by IP, http://10.2.0.1/ however it looks like the client integration sticks the bucket name in front of the IP address http://vault-test.10.2.0.1/. I do not have wildcard DNS for Ceph RADOSGW configured (and this isn't trivial as it's in Kubernetes).

This is how I access the bucket with s3cmd:

export AWS_HOST=10.2.0.1
export AWS_ENDPOINT=10.2.0.1:80
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
s3cmd ls --no-ssl --host=${AWS_HOST} --host-bucket=  s3://vault-test/