Skip to content
This repository has been archived by the owner on Nov 30, 2021. It is now read-only.

Workflow helm install hangs if S3 bucket name has dots (.) in it #700

Closed
greglearns opened this issue Jan 14, 2017 · 3 comments
Closed

Workflow helm install hangs if S3 bucket name has dots (.) in it #700

greglearns opened this issue Jan 14, 2017 · 3 comments

Comments

@greglearns
Copy link
Contributor

TL;DR: Workflow's helm chart should validate that the S3 database_bucket and registry_bucket do not have periods (.) in them.

THIS BREAKS THINGS

Modifying Deis Workflows basic chart values to include these changes break the deployment, and Workflow will hang on the deploy:

global:
  storage: s3

s3:
  # Your buckets.
  registry_bucket: "deis.subdomain.domain.com-registry"
  database_bucket: "deis.subdomain.domain.com-database"
  builder_bucket:  "deis.subdomain.domain.com-builder"

The builder bucket gets created in S3, but not the registry or database buckets.
The database logs include this at the end:

ssl.CertificateError: hostname 'deis.subdomain.domain.com-registry.s3-us-west-2.amazonaws.com' doesn't match either of 's3-us-west-2.amazonaws.com', '*.s3-us-west-2.amazonaws.com', 's3.us-west-2.amazonaws.com', '*.s3.us-west-2.amazonaws.com', 's3.dualstack.us-west-2.amazonaws.com', '*.s3.dualstack.us-west-2.amazonaws.com', '*.s3.amazonaws.com'
2017/01/14 07:20:04 Error creating the registry bucket: exit status 1

THIS FIXES THINGS

Modifying Deis Workflows basic chart values to include these changes works, and Workflow will deploy:

global:
  storage: s3

s3:
  # Your buckets.
  registry_bucket: "deis-subdomain-domain-com-registry"
  database_bucket: "deis-subdomain-domain-com-database"
  builder_bucket:  "deis-subdomain-domain-com-builder"

Versions:

~/c/deis ❯❯❯ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.7", GitCommit:"92b4f971662de9d8770f8dcd2ee01ec226a6f6c0", GitTreeState:"clean", BuildDate:"2016-12-10T04:43:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
~/c/deis ❯❯❯ helm version
Client: &version.Version{SemVer:"v2.1.3", GitCommit:"5cbc48fb305ca4bf68c26eb8d2a7eb363227e973", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.1.3", GitCommit:"5cbc48fb305ca4bf68c26eb8d2a7eb363227e973", GitTreeState:"clean"}
~/c/deis ❯❯❯ deis version
v2.10.0

THOUGHTS

One could argue that this is a user-error (I should have known not to use dots, especially since the examples in the https://github.com/deis/workflow/blob/master/charts/workflow/values.yaml template clearly show hyphens registry_bucket: "your-registry-bucket-name"; however, as a well-intentioned learner of Deis (after taking 3 days of training), I cut and pasted the Kubernetes recommended name for the cluster into the bucket name fields since that seemed to make sense at the time, and I can see how others could make a similar mistake.

p.s.: Loving Deis, Workflow, Helm, and Kubernetes. Thank you for all of your hard (and awesome) work Deis and everyone!

greglearns added a commit to greglearns/workflow that referenced this issue Jan 14, 2017
greglearns added a commit to greglearns/postgres that referenced this issue Jan 14, 2017
RELATED ISSUES

deis/workflow#700
deis/workflow#701 (documentation fix)

Note: this bug also exists for https://github.com/deis/registry/blob/master/rootfs/bin/create-bucket#L27

REPRODUCING THE BUG

Using a period in BUCKET_NAME causes `helm install -f values.yaml deis/workflow --namespace=deis` to fail with this error:
```
ssl.CertificateError: hostname 'deis.subdomain.domain.com-registry.s3-us-west-2.amazonaws.com' doesn't match either of 's3-us-west-2.amazonaws.com', '*.s3-us-west-2.amazonaws.com', 's3.us-west-2.amazonaws.com', '*.s3.us-west-2.amazonaws.com', 's3.dualstack.us-west-2.amazonaws.com', '*.s3.dualstack.us-west-2.amazonaws.com', '*.s3.amazonaws.com'
2017/01/14 07:20:04 Error creating the registry bucket: exit status 1
```

A POSSIBLE FIX

Not sure how to create a test for this (I just started using Deis 4 days ago), but, a hacky test is this:
```
echo "a.bad.thing-mixed-with-a-good-thing" | sed "s/[.]/-/g"
# a-bad-thing-mixed-with-a-good-thing
```

THOUGHTS
@bacongobbler
Copy link
Member

I think the dot-in-bucket-name issue is something S3 doesn't enjoy in certain accounts. Some work, some don't. I agree and think it's fair to say just use other separators. Thanks!

@greglearns
Copy link
Contributor Author

Oh, I should clarify: S3 doesn't complain about the dot in the bucket name... instead, the ssl cert creation has a problem. Full stack trace here from doing helm install -f values.yaml deis/workflow --namespace=deis:

2017/01/14 06:22:18 INFO: using s3 as the backend
Traceback (most recent call last):
  File "/bin/create-bucket", line 28, in <module>
    conn.create_bucket(bucket_name, location=region)
  File "/usr/lib/python3.5/site-packages/boto/s3/connection.py", line 620, in create_bucket
    data=data)
  File "/usr/lib/python3.5/site-packages/boto/s3/connection.py", line 672, in make_request
    retry_handler=retry_handler
  File "/usr/lib/python3.5/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python3.5/site-packages/boto/connection.py", line 943, in _mexe
    request.body, request.headers)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/python3.5/http/client.py", line 1260, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 752, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 988, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 638, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 297, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
ssl.CertificateError: hostname 'deis.subdomain.domain.com-registry.s3-us-west-2.amazonaws.com' doesn't match either of 's3-us-west-2.amazonaws.com', '*.s3-us-west-2.amazonaws.com', 's3.us-west-2.amazonaws.co
m', '*.s3.us-west-2.amazonaws.com', 's3.dualstack.us-west-2.amazonaws.com', '*.s3.dualstack.us-west-2.amazonaws.com', '*.s3.amazonaws.com'
2017/01/14 06:22:18 Error creating the registry bucket: exit status 1

greglearns added a commit to greglearns/postgres that referenced this issue Jan 14, 2017
Using a period in an S3 BUCKET_NAME causes `helm install -f values.yaml deis/workflow --namespace=deis` to fail.

REPRODUCING THE BUG

Using a period in BUCKET_NAME causes `helm install -f values.yaml deis/workflow --namespace=deis` to fail with this error:
```
ssl.CertificateError: hostname 'deis.subdomain.domain.com-registry.s3-us-west-2.amazonaws.com' doesn't match either of 's3-us-west-2.amazonaws.com', '*.s3-us-west-2.amazonaws.com', 's3.us-west-2.amazonaws.com', '*.s3.us-west-2.amazonaws.com', 's3.dualstack.us-west-2.amazonaws.com', '*.s3.dualstack.us-west-2.amazonaws.com', '*.s3.amazonaws.com'
2017/01/14 07:20:04 Error creating the registry bucket: exit status 1
```

A POSSIBLE FIX

Not sure how to create a test for this (I just started using Deis 4 days ago), but, a hacky test is this:
```
echo "a.bad.thing-mixed-with-a-good-thing" | sed "s/[.]/-/g"
```

RELATED ISSUES

deis/workflow#700
deis/workflow#701 (documentation fix)

Note: this bug also exists for https://github.com/deis/registry/blob/master/rootfs/bin/create-bucket#L27
greglearns added a commit to greglearns/workflow that referenced this issue Jan 14, 2017
Warn users not to use periods in S3 bucket names in installing-workflow/configuring-object-storage.md. Related to issue deis#700
greglearns added a commit to greglearns/workflow that referenced this issue Jan 16, 2017
Warn users not to use periods in S3 bucket names in installing-workflow/configuring-object-storage.md. Related to issue deis#700
@bacongobbler
Copy link
Member

closed via #701

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants