Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Weave Net connections missing on Scope Service #1162

Closed
2opremio opened this issue Mar 11, 2016 · 3 comments
Closed

Weave Net connections missing on Scope Service #1162

2opremio opened this issue Mar 11, 2016 · 3 comments
Assignees
@paulbellamy
Labels
bug Broken end user or developer functionality; not working as the developers intended it
Milestone
0.14.0

Comments

@2opremio
Copy link
Contributor

If I launch the ECS demo with Scope in Standalone mode and repeatedly connect to one of the httpservers, everything works/looks as expected:

standalone_expected

and the containers show their Weave IP as expected:

standalone_details

Here's the report: report_standalone_expected.json.gz

However, if I launch it in Cloud Service mode, I miss the edges between the httpserver and the data-producers:

screen shot 2016-03-11 at 8 08 16 pm

but, unlike in #643, the containers do show their Weave IPs

screen shot 2016-03-11 at 9 03 58 pm

Report: report_service_unexpected.json.gz

The logs from the weavescope container in one of the hosts don't reveal anything unusal: https://gist.github.com/2opremio/d231744001f69b51472d

Here are the arguments passed to the scope probe:

[ec2-user@ip-172-31-0-8 ~]$ ps aux | grep scope-probe
root      2775  3.5  4.4 483840 44900 ?        Sl   19:04   0:39 scope-probe probe -docker=true -token=omitted scope.weave.works:443

(it only adds -token=omitted scope.weave.works:443 compared to the standalone demo)

@tomwilkie Any ideas? This looks remarkably similar to #643 but:

  • The probes seem to be talking to weave (see Success collecting weave info in the logs), plus I don't see how launching the probes in Service mode would make a difference in this respect.
  • Restarting the probes to ensure they boot after Weave Net is running doesn't help (although it shouldn't matter since Make scope deal with weave restarts (or weave starts after scope) #510 is now closed).

Note: the screenshots were taken during different runs of the demo while investigating, so don't expect IPs and other metrics to be consistent across them.
@2opremio 2opremio added the bug Broken end user or developer functionality; not working as the developers intended it label Mar 11, 2016
@2opremio 2opremio added this to the 0.14.0 milestone Mar 11, 2016
@2opremio
Copy link
Contributor Author

I may be missing something since I don't know the rendering code too well but AFAIU, these two container nodes and endpoint node from the report should result in a rendered edge between them.

Container node for the httpserver (IP 10.40.0.2):

      "3af59b5a36ba8e2a8b612622b943c85709a254e61ae4ff737573158f89d1b42c;<container>": {
        "id": "3af59b5a36ba8e2a8b612622b943c85709a254e61ae4ff737573158f89d1b42c;<container>",
        "topology": "container",
        "counters": {},
        "sets": {
          "docker_container_ips": [
            "10.40.0.2",
            "172.17.0.4"
          ],
          "docker_container_ports": [
            "172.31.0.8:80->80/tcp",
            "fe80::20cb:28ff:fe1d:de6d:80->80/tcp",
            "fe80::38b1:feff:fe0b:8bc0:80->80/tcp",
            "fe80::492:67ff:febb:4d31:80->80/tcp"
          ],
          "docker_container_ips_with_scopes": [
            ";10.40.0.2",
            "ip-172-31-0-8;172.17.0.4"
          ]
        },
        "adjacency": null,
        "edges": {},
        "controls": {
          "timestamp": "2016-03-11T19:19:02.792805538Z",
          "controls": [
            "docker_attach_container",
            "docker_exec_container",
            "docker_pause_container",
            "docker_restart_container",
            "docker_stop_container"
          ]
        },
        "latest": {
          "docker_container_restart_count": {
            "timestamp": "2016-03-11T19:19:02.792802491Z",
            "value": "0"
          },
          "docker_container_uptime": {
            "timestamp": "2016-03-11T19:19:02.792802491Z",
            "value": "12m52s"
          },
          "docker_container_created": {
            "timestamp": "2016-03-11T19:19:02.792769902Z",
            "value": "11 Mar 16 19:06 UTC"
          },
          "docker_container_state": {
            "timestamp": "2016-03-11T19:19:02.792769902Z",
            "value": "running"
          },
          "docker_container_hostname": {
            "timestamp": "2016-03-11T19:19:02.792769902Z",
            "value": "httpserver.weave.local"
          },
          "host_node_id": {
            "timestamp": "2016-03-11T19:19:02.804347528Z",
            "value": "ip-172-31-0-8;<host>"
          },
          "docker_label_com.amazonaws.ecs.task-definition-version": {
            "timestamp": "2016-03-11T19:19:02.792818291Z",
            "value": "517"
          },
          "docker_container_command": {
            "timestamp": "2016-03-11T19:19:02.792769902Z",
            "value": "/w/w bash -c set -x; sleep 7; while true; do DATA_PRODUCER_MESSAGE=`nc ${DATA_PRODUCER_NAME} 4540`; HTML=\"<html> <head> <link rel=\\\"icon\\\" href=\\\"data:;base64,iVBORw0KGgo=\\\"> <title>Weaveworks Amazon ECS Sample App</title> <style>body {margin-top: 40px; background-color: #333;} </style> </head><body> <div style=color:white;text-align:center><h1>Chosen data producer message:</h1> <h2>${DATA_PRODUCER_MESSAGE}</h2> </div>\"; echo -e \"HTTP/1.1 200 OK\\r\\nContent-Type: text/html\\r\\nConnection: close\\r\\nContent-Length: $(echo $HTML | wc -c)\\r\\n\\r\\n$HTML\" | nc -q 0 -l -p 80; done"
          },
          "docker_container_network_mode": {
            "timestamp": "2016-03-11T19:19:02.792802491Z",
            "value": "default"
          },
          "docker_label_com.amazonaws.ecs.container-name": {
            "timestamp": "2016-03-11T19:19:02.792810777Z",
            "value": "httpserver"
          },
          "weave_dns_hostname": {
            "timestamp": "2016-03-11T19:19:02.804890849Z",
            "value": "httpserver.weave.local"
          },
          "docker_label_com.amazonaws.ecs.task-definition-family": {
            "timestamp": "2016-03-11T19:19:02.792815419Z",
            "value": "weave-ecs-demo-task"
          },
          "docker_container_id": {
            "timestamp": "2016-03-11T19:19:02.792769902Z",
            "value": "3af59b5a36ba8e2a8b612622b943c85709a254e61ae4ff737573158f89d1b42c"
          },
          "docker_container_name": {
            "timestamp": "2016-03-11T19:19:02.792769902Z",
            "value": "ecs-weave-ecs-demo-task-517-httpserver-fed8c6faa99fd2934600"
          },
          "control_probe_id": {
            "timestamp": "2016-03-11T19:19:02.792826923Z",
            "value": "34230fa999064d98"
          },
          "weave_mac_address": {
            "timestamp": "2016-03-11T19:19:02.804927563Z",
            "value": "06:9c:05:fe:39:e9"
          },
          "docker_label_com.amazonaws.ecs.task-arn": {
            "timestamp": "2016-03-11T19:19:02.792812981Z",
            "value": "arn:aws:ecs:eu-west-1:376248598259:task/072f55ea-94ad-42cb-9c1a-58b47df77168"
          },
          "docker_image_id": {
            "timestamp": "2016-03-11T19:19:02.792769902Z",
            "value": "a4e0640a29e126734c124298e97dec0e0ff3461487939dcee89a1c55b6e70f02"
          }
        }

Container node for dataproducer (IP 10.36.0.2):

    "1ca4c5605185f34cf7f5a3d27ecc501155632138bce6aa5b69d5d856c1320037;<container>": {
        "id": "1ca4c5605185f34cf7f5a3d27ecc501155632138bce6aa5b69d5d856c1320037;<container>",
        "topology": "container",
        "counters": {},
        "sets": {
          "docker_container_ips": [
            "10.36.0.2",
            "172.17.0.3"
          ],
          "docker_container_ports": null,
          "docker_container_ips_with_scopes": [
            ";10.36.0.2",
            "ip-172-31-0-10;172.17.0.3"
          ]
        },
        "adjacency": null,
        "edges": {},
        "controls": {
          "timestamp": "2016-03-11T19:19:01.566915851Z",
          "controls": [
            "docker_attach_container",
            "docker_exec_container",
            "docker_pause_container",
            "docker_restart_container",
            "docker_stop_container"
          ]
        },
        "latest": {
          "docker_container_restart_count": {
            "timestamp": "2016-03-11T19:19:01.56691334Z",
            "value": "0"
          },
          "weave_mac_address": {
            "timestamp": "2016-03-11T19:19:01.575448178Z",
            "value": "9a:62:dd:99:2e:38"
          },
          "docker_container_uptime": {
            "timestamp": "2016-03-11T19:19:01.56691334Z",
            "value": "12m52s"
          },
          "docker_container_created": {
            "timestamp": "2016-03-11T19:19:01.566899378Z",
            "value": "11 Mar 16 19:06 UTC"
          },
          "docker_container_state": {
            "timestamp": "2016-03-11T19:19:01.566899378Z",
            "value": "running"
          },
          "docker_container_command": {
            "timestamp": "2016-03-11T19:19:01.566899378Z",
            "value": "/w/w sh -c while true; do echo 'Hi, this is the data producer in' `hostname -i | awk '{ print $1 }'` | nc -q 0 -l -p 4540; done"
          },
          "docker_label_com.amazonaws.ecs.task-definition-family": {
            "timestamp": "2016-03-11T19:19:01.566923626Z",
            "value": "weave-ecs-demo-task"
          },
          "docker_image_id": {
            "timestamp": "2016-03-11T19:19:01.566899378Z",
            "value": "a4e0640a29e126734c124298e97dec0e0ff3461487939dcee89a1c55b6e70f02"
          },
          "docker_label_com.amazonaws.ecs.task-arn": {
            "timestamp": "2016-03-11T19:19:01.56692146Z",
            "value": "arn:aws:ecs:eu-west-1:376248598259:task/e7a7f3e3-ab6d-44f2-b39c-27f78376a6da"
          },
          "host_node_id": {
            "timestamp": "2016-03-11T19:19:01.574972002Z",
            "value": "ip-172-31-0-10;<host>"
          },
          "docker_container_network_mode": {
            "timestamp": "2016-03-11T19:19:01.56691334Z",
            "value": "default"
          },
          "docker_container_id": {
            "timestamp": "2016-03-11T19:19:01.566899378Z",
            "value": "1ca4c5605185f34cf7f5a3d27ecc501155632138bce6aa5b69d5d856c1320037"
          },
          "weave_dns_hostname": {
            "timestamp": "2016-03-11T19:19:01.575430126Z",
            "value": "dataproducer.weave.local"
          },
          "control_probe_id": {
            "timestamp": "2016-03-11T19:19:01.566933419Z",
            "value": "19d7128bcf5b25b4"
          },
          "docker_label_com.amazonaws.ecs.container-name": {
            "timestamp": "2016-03-11T19:19:01.566928214Z",
            "value": "dataproducer"
          },
          "docker_container_hostname": {
            "timestamp": "2016-03-11T19:19:01.566899378Z",
            "value": "dataproducer.weave.local"
          },
          "docker_label_com.amazonaws.ecs.task-definition-version": {
            "timestamp": "2016-03-11T19:19:01.566926174Z",
            "value": "517"
          },
          "docker_container_name": {
            "timestamp": "2016-03-11T19:19:01.566899378Z",
            "value": "ecs-weave-ecs-demo-task-517-dataproducer-c0efed88da9a8ff08501"
          }
        },

Endpoint node from 10.40.0.2 to 10.36.0.2:

     ";10.40.0.2;49129": {
        "id": ";10.40.0.2;49129",
        "topology": "endpoint",
        "counters": {},
        "sets": {},
        "adjacency": [
          ";10.36.0.2;4540"
        ],
        "edges": {
          ";10.36.0.2;4540": {}
        },
        "controls": {},
        "latest": {
          "port": {
            "timestamp": "2016-03-11T19:19:01.562989403Z",
            "value": "49129"
          },
          "addr": {
            "timestamp": "2016-03-11T19:19:01.562989403Z",
            "value": "10.40.0.2"
          },
          "conntracked": {
            "timestamp": "2016-03-11T19:19:01.562501115Z",
            "value": "true"
          }
        },
        "parents": {}
      },

@tomwilkie tomwilkie self-assigned this Mar 14, 2016
@tomwilkie tomwilkie removed their assignment Apr 11, 2016
@2opremio 2opremio mentioned this issue Apr 11, 2016
Closed
@paulbellamy paulbellamy self-assigned this Apr 14, 2016
@paulbellamy
Copy link
Contributor

Running a fixprobe with the following json in it "fixes" the graph:

{
    "Host": {
        "nodes": {
            "foobar;<host>": {
                "counters": {},
                "sets": {
                    "local_networks": [
                        "10.32.0.6/12"
                    ]
                }
            }
        }
    },
    "ID": "fooreport"
}

weave on the ecs machines is allocating in 10.32.0.0/12, but are not including that range in their own "local_networks" set. Which, I believe, means it is treating all weave traffic as happening inside the internet node.

@paulbellamy
Copy link
Contributor

The scope app, periodically talks to the weave api to register it's dns address, and call weave expose. calling weave expose causes the weave interface to show which cidr it has, adding that into local_networks, and thereby making the whole thing work.

In service mode there is (by nature) no app running, so the weave expose never happens.

@2opremio 2opremio mentioned this issue Apr 15, 2016
Merged
4 tasks
@paulbellamy paulbellamy mentioned this issue Apr 15, 2016
Merged
@2opremio 2opremio mentioned this issue May 12, 2016
Closed
@2opremio 2opremio mentioned this issue Jun 9, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@paulbellamy paulbellamy

Labels
bug Broken end user or developer functionality; not working as the developers intended it
Projects
None yet
Milestone
0.14.0
Development

No branches or pull requests
3 participants
@paulbellamy @tomwilkie @2opremio