Bad error message on alloc exec when streaming through ELB

[cgbaker]

Nomad version

Nomad v0.10.0-rc1 (c49bf41)

Operating system and Environment details

Linux in AWS EC2

Issue

Nomad alloc exec does not work through EC2 Classic ELB. However, the error message returned in this scenario is not human-readable, complicating the ability for operators to diagnose this issue.

Reproduction steps

1. Spin up Nomad cluster in EC2 (for example, using TF config in /terraform)
2. Front the server(s) with a classic ELB, with a listener configured for HTTP/4646 (Layer 7)
3. Note that alloc fs commands work.
4. Note that alloc exec commands do not:

$ nomad alloc exec -job nginx  bash
�0iled to exec into task: Unexpected response code: 400 (�<��
�mHG���������N�x,E��ڋ��XC�2~�
G���-#]���v�O�!��H��"����H�#U�)##6�6�t�<0��z��

Nomad Server log

When encountering the error, the server emits the following error message:

Oct 24 15:43:40 ip-172-31-41-217 nomad[2198]:     2019-10-24T15:43:40.285Z [ERROR] http: request failed: method=GET path=/v1/client/allocation/ea145ac4-56e7-7c45-47a1-2ad3dfb334d2/exec?command=%5B%22bash%22%5D&task=nginx&tty=true error="failed to upgrade connection: websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header" code=500

Summary

The root cause is that Level 7 (HTTP) ELBs in EC2 do not support WebSockets. Readers should note that there are multiple approaches to resolving this: change the listener to Level 4 (TCP); or use the newer EC2 Application Load Balancer (ALB). Regardless, the error message emitted by the CLI should propagate the error from the server, if possible.

[drewbailey]

Adding some more details

example curl to nomad fronted w/ ELB

curl -v "$NOMAD_ADDR/v1/client/allocation/14df86fb-b7b2-b429-6150-68e6fe251865/exec?task=cache&command=%5B%22bin%22%2C%22bash%22%5D"

< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< Date: Thu, 07 Nov 2019 18:53:45 GMT
< Sec-Websocket-Version: 13
< Vary: Accept-Encoding
< Vary: Origin
< X-Content-Type-Options: nosniff
< Content-Length: 149
< Connection: keep-alive
<
Bad Request
* Connection #0 to host nomad-server-lb-1857302615.us-east-1.elb.amazonaws.com left intact
failed to upgrade connection: websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header% 

[drewbailey]

error on master:

→ nomad alloc exec -job example bin/bash                                                                                
failed to exec into task: Unexpected response code: 400 (Bad Request
failed to upgrade connection: websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.