Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Empty service token results in weird behaviour #774

Closed
errordeveloper opened this issue Dec 16, 2015 · 4 comments · Fixed by #1357
Closed

Empty service token results in weird behaviour #774

errordeveloper opened this issue Dec 16, 2015 · 4 comments · Fixed by #1357
Assignees
@tomwilkie
Milestone
0.15.0

Comments

@errordeveloper
Copy link
Contributor 

0 %> ssh_cloud -i ~/.ssh/ec2_terraform.us-east-1.pem [email protected]
Last login: Mon Nov 23 15:04:39 2015 from 37.157.33.76

   __|  __|  __|
   _|  (   \__ \   Amazon ECS-Optimized Amazon Linux AMI 2015.09.b
 ____|\___|____/

For documentation visit, http://aws.amazon.com/documentation/ecs
10 package(s) needed for security, out of 22 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-1-62 ~]$ docker ps
CONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS                        NAMES
66470487961c        amazon/amazon-ecs-agent:latest   "/w/w /agent"          11 minutes ago      Up 11 minutes       127.0.0.1:51678->51678/tcp   ecs-agent           
5fda481ad130        weaveworks/weaveexec:1.3.1       "/home/weave/weavepr   12 minutes ago      Up 12 minutes                                    weaveproxy          
6a9cb504ce22        weaveworks/weave:1.3.1           "/home/weave/weaver    12 minutes ago      Up 12 minutes                                    weave               
8fb4d5ee0ec8        weaveworks/scope:0.10.0          "/home/weave/entrypo   12 minutes ago      Up 12 minutes                                    weavescope          
[ec2-user@ip-172-31-1-62 ~]$ docker logs 8fb4d5ee0ec8
<probe> 2015/12/16 00:27:09 probe starting, version 0.10.0, ID 36180f85ae6fa9ed
<probe> 2015/12/16 00:27:09 publishing to: scope.weave.works:80, 127.0.0.1, 172.31.0.65, 172.31.1.63
<probe> 2015/12/16 00:27:09 docker container: collecting stats for 8fb4d5ee0ec8f71731484170d84b6064e5e98e9857d6a9787749956dbb421705
<probe> 2015/12/16 00:27:09 docker container: collecting stats for d7511a82937ad7dd4e3c048f78203243255bf0c92c79e568357f85f96bb1370d
<probe> 2015/12/16 00:27:09 docker container: collecting stats for 56a663b14d1bc0477b6cb998ce475f0a9049e15cc196da99fc453189c63b4c9e
<probe> 2015/12/16 00:27:09 multi-publisher set: 172.31.0.65:4040 (172.31.0.65:4040): Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:09 Error fetching app details: Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:09 Error doing controls for 172.31.0.65:4040, backing off 1s: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:09 multi-publisher set: 172.31.1.63:4040 (172.31.1.63:4040): Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:09 Error fetching app details: Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:09 multi-publisher set: 127.0.0.1:4040 (127.0.0.1:4040): Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:09 Error doing controls for 52.5.175.31:80, backing off 1s: websocket: bad handshake
<probe> 2015/12/16 00:27:09 Error fetching app details: Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:10 Error doing controls for 172.31.0.65:4040, backing off 2s: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:10 Error doing controls for 52.5.175.31:80, backing off 2s: websocket: bad handshake
<probe> 2015/12/16 00:27:10 docker container: error reading event, did container stop? read unix @->/var/run/docker.sock: use of closed network connection
<probe> 2015/12/16 00:27:10 docker container: stopped collecting stats for d7511a82937ad7dd4e3c048f78203243255bf0c92c79e568357f85f96bb1370d
<probe> 2015/12/16 00:27:10 Error publishing to http://52.5.175.31:80/api/report, backing off 1s: 502 Bad Gateway
<probe> 2015/12/16 00:27:12 Error publishing to http://52.5.175.31:80/api/report, backing off 2s: 502 Bad Gateway
<probe> 2015/12/16 00:27:12 Error doing controls for 172.31.0.65:4040, backing off 4s: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:12 docker container: collecting stats for 6a9cb504ce22113d4ca155fcfa672bb141f5041056b80f124f4fa74983b9ac7b
<probe> 2015/12/16 00:27:12 docker container: collecting stats for 6a9cb504ce22113d4ca155fcfa672bb141f5041056b80f124f4fa74983b9ac7b
<probe> 2015/12/16 00:27:12 Error doing controls for 52.5.175.31:80, backing off 4s: websocket: bad handshake
<probe> 2015/12/16 00:27:13 docker container: error reading event, did container stop? read unix @->/var/run/docker.sock: use of closed network connection
<probe> 2015/12/16 00:27:13 docker container: stopped collecting stats for 56a663b14d1bc0477b6cb998ce475f0a9049e15cc196da99fc453189c63b4c9e
<probe> 2015/12/16 00:27:14 Error publishing to http://52.5.175.31:80/api/report, backing off 4s: 502 Bad Gateway
<probe> 2015/12/16 00:27:14 docker container: collecting stats for cbc76b4bf7d944b227684f37982d2509c9481eeda3945e39d2bae571b1fd1dc6
<probe> 2015/12/16 00:27:15 docker container: collecting stats for 5fda481ad130d64de653ae7841ca0360a10465936076b6847ef5882f30ca9dd4
<probe> 2015/12/16 00:27:15 docker container: collecting stats for 5fda481ad130d64de653ae7841ca0360a10465936076b6847ef5882f30ca9dd4
<probe> 2015/12/16 00:27:16 Error doing controls for 172.31.0.65:4040, backing off 8s: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:16 docker container: error reading event, did container stop? read unix @->/var/run/docker.sock: use of closed network connection
<probe> 2015/12/16 00:27:16 docker container: stopped collecting stats for cbc76b4bf7d944b227684f37982d2509c9481eeda3945e39d2bae571b1fd1dc6
<probe> 2015/12/16 00:27:16 Error doing controls for 52.5.175.31:80, backing off 8s: websocket: bad handshake
<probe> 2015/12/16 00:27:18 docker container: collecting stats for 66470487961c1a1f5f0104140967bdb41bd4a1988e5acbcfec62208b9948ec38
<probe> 2015/12/16 00:27:18 Error publishing to http://52.5.175.31:80/api/report, backing off 8s: 502 Bad Gateway
<probe> 2015/12/16 00:27:19 docker container: collecting stats for 5e05d27dc238488851bf36ddef5085b12f8888808ab7813c40a258f4b16ad840
<probe> 2015/12/16 00:27:19 docker container: stopped collecting stats for 5e05d27dc238488851bf36ddef5085b12f8888808ab7813c40a258f4b16ad840
<probe> 2015/12/16 00:27:19 multi-publisher set: 172.31.0.65:4040 (172.31.0.65:4040): Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:19 Error fetching app details: Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:19 multi-publisher set: 172.31.1.63:4040 (172.31.1.63:4040): Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:19 Error fetching app details: Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:19 multi-publisher set: 127.0.0.1:4040 (127.0.0.1:4040): Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:19 Error fetching app details: Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:20 Error publishing to http://52.5.175.31:80/api/report, backing off 1s: 502 Bad Gateway
<probe> 2015/12/16 00:27:20 docker container: collecting stats for 015de66ced63937c4d1b9ace2ad49d646550c188798218925cf4b5ed270fcfb7
<probe> 2015/12/16 00:27:20 docker container: stopped collecting stats for 015de66ced63937c4d1b9ace2ad49d646550c188798218925cf4b5ed270fcfb7
<probe> 2015/12/16 00:27:24 Error publishing to http://52.5.175.31:80/api/report, backing off 2s: 502 Bad Gateway
<probe> 2015/12/16 00:27:24 Error doing controls for 172.31.0.65:4040, backing off 16s: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:24 Error doing controls for 52.5.175.31:80, backing off 16s: websocket: bad handshake
<probe> 2015/12/16 00:27:27 Error publishing to http://52.5.175.31:80/api/report, backing off 4s: 502 Bad Gateway
<probe> 2015/12/16 00:27:29 multi-publisher set: 172.31.1.63:4040 (172.31.1.63:4040): Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:29 Error fetching app details: Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:29 multi-publisher set: 127.0.0.1:4040 (127.0.0.1:4040): Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:29 Error fetching app details: Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:29 multi-publisher set: 172.31.0.65:4040 (172.31.0.65:4040): Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:29 Error fetching app details: Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:30 Error publishing to http://52.5.175.31:80/api/report, backing off 1s: 502 Bad Gateway
<probe> 2015/12/16 00:27:33 Error publishing to http://52.5.175.31:80/api/report, backing off 2s: 502 Bad Gateway
<probe> 2015/12/16 00:27:36 Error publishing to http://52.5.175.31:80/api/report, backing off 4s: 502 Bad Gateway
<probe> 2015/12/16 00:27:39 multi-publisher set: 127.0.0.1:4040 (127.0.0.1:4040): Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:39 Error fetching app details: Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:39 multi-publisher set: 172.31.0.65:4040 (172.31.0.65:4040): Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:39 Error fetching app details: Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:39 multi-publisher set: 172.31.1.63:4040 (172.31.1.63:4040): Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:39 Error fetching app details: Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:40 Error doing controls for 172.31.0.65:4040, backing off 32s: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:40 Error doing controls for 52.5.175.31:80, backing off 32s: websocket: bad handshake
<probe> 2015/12/16 00:27:42 Error publishing to http://52.5.175.31:80/api/report, backing off 1s: 502 Bad Gateway
<probe> 2015/12/16 00:27:45 Error publishing to http://52.5.175.31:80/api/report, backing off 2s: 502 Bad Gateway
<probe> 2015/12/16 00:27:48 Error publishing to http://52.5.175.31:80/api/report, backing off 4s: 502 Bad Gateway
<probe> 2015/12/16 00:27:49 multi-publisher set: 127.0.0.1:4040 (127.0.0.1:4040): Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:49 Error fetching app details: Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:49 multi-publisher set: 172.31.0.65:4040 (172.31.0.65:4040): Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:27:49 Error fetching app details: Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused

...

<probe> 2015/12/16 00:39:12 Error publishing to http://52.5.175.31:80/api/report, backing off 1s: 502 Bad Gateway
<probe> 2015/12/16 00:39:12 Error doing controls for 172.31.0.65:4040, backing off 1m0s: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:39:12 Error doing controls for 52.5.175.31:80, backing off 1m0s: websocket: bad handshake
<probe> 2015/12/16 00:39:15 Error publishing to http://52.5.175.31:80/api/report, backing off 2s: 502 Bad Gateway
<probe> 2015/12/16 00:39:18 Error publishing to http://52.5.175.31:80/api/report, backing off 4s: 502 Bad Gateway
<probe> 2015/12/16 00:39:19 multi-publisher set: 172.31.1.63:4040 (172.31.1.63:4040): Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:39:19 Error fetching app details: Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:39:19 multi-publisher set: 127.0.0.1:4040 (127.0.0.1:4040): Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:39:19 Error fetching app details: Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> 2015/12/16 00:39:19 multi-publisher set: 172.31.0.65:4040 (172.31.0.65:4040): Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:39:19 Error fetching app details: Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused
<probe> 2015/12/16 00:39:21 Error publishing to http://52.5.175.31:80/api/report, backing off 1s: 502 Bad Gateway
[ec2-user@ip-172-31-1-62 ~]$ ps -ef | grep scope
root      2346     1  0 00:26 ?        00:00:00 /bin/bash /etc/weave/run.sh scope
root      3231  3229  6 00:27 ?        00:00:45 /home/weave/scope-probe -docker=true -token= scope.weave.works:80 127.0.0.1 172.31.0.65 172.31.1.63
ec2-user 13654 13376  0 00:39 pts/0    00:00:00 grep --color=auto scope
[ec2-user@ip-172-31-1-62 ~]$ logout
@errordeveloper
Copy link
Contributor Author

weaveworks/integrations#43

@errordeveloper errordeveloper mentioned this issue Dec 16, 2015
Closed
@errordeveloper
Copy link
Contributor Author

We are currently doing ARG_VALUE=$(echo "$1" | sed 's/\-\-service-token=\(.*\)/\1/') in docker/entrypoint.sh, but this needs to be stronger.

@errordeveloper
Copy link
Contributor Author

Also, why do we accept and empty flag?

@errordeveloper
Copy link
Contributor Author

The bigger problem is that this actually results in us not listening on port 4040.

ec2-user@ip-172-31-1-62 ~]$ sudo netstat -nap | grep scope
tcp        0      0 172.31.1.62:32835           52.3.121.160:80             ESTABLISHED 3231/scope-probe    
tcp        0      0 172.31.1.62:32876           52.3.121.160:80             ESTABLISHED 3231/scope-probe    
tcp        0      0 172.31.1.62:32843           52.3.121.160:80             ESTABLISHED 3231/scope-probe    
tcp        0      0 172.31.1.62:32877           52.3.121.160:80             ESTABLISHED 3231/scope-probe    
tcp        0      0 172.31.1.62:32861           52.3.121.160:80             ESTABLISHED 3231/scope-probe    
tcp        0      0 172.31.1.62:32853           52.3.121.160:80             ESTABLISHED 3231/scope-probe    
tcp        0      0 172.31.1.62:32871           52.3.121.160:80             ESTABLISHED 3231/scope-probe    
unix  3      [ ]         STREAM     CONNECTED     14077  3231/scope-probe    
unix  3      [ ]         STREAM     CONNECTED     12123  3231/scope-probe    
unix  3      [ ]         STREAM     CONNECTED     14071  3231/scope-probe    
unix  3      [ ]         STREAM     CONNECTED     12131  3231/scope-probe    
unix  3      [ ]         STREAM     CONNECTED     15292  3231/scope-probe    
unix  3      [ ]         STREAM     CONNECTED     16246  3231/scope-probe    
unix  3      [ ]         STREAM     CONNECTED     15286  3231/scope-probe    
unix  3      [ ]         STREAM     CONNECTED     12121  3231/scope-probe

Hence:

<probe> 2015/12/16 00:39:19 multi-publisher set: 172.31.1.63:4040 (172.31.1.63:4040): Get http://172.31.1.63:4040/api: dial tcp 172.31.1.63:4040: getsockopt: connection refused
<probe> 2015/12/16 00:39:19 multi-publisher set: 172.31.0.65:4040 (172.31.0.65:4040): Get http://172.31.0.65:4040/api: dial tcp 172.31.0.65:4040: getsockopt: connection refused

@tomwilkie tomwilkie changed the title Empty service toke results in weird behaviour Empty service token results in weird behaviour Dec 16, 2015
errordeveloper added a commit to weaveworks/integrations that referenced this issue Jan 15, 2016
@errordeveloper
Handle weaveworks/scope#774 and #43
b8f5394
@tomwilkie tomwilkie self-assigned this Apr 20, 2016
@tomwilkie tomwilkie added this to the 0.15.0 milestone Apr 20, 2016
@tomwilkie tomwilkie mentioned this issue Apr 20, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomwilkie tomwilkie

Labels
None yet
Projects
None yet
Milestone
0.15.0
Development

Successfully merging a pull request may close this issue.

Move all the flag parsing to main.go; implement scope help weaveworks/scope
2 participants
@errordeveloper @tomwilkie