Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker network inspect does not show container in endpoint list if powered on out-of-band #8052

Closed
hickeng opened this issue Jun 14, 2018 · 4 comments
Closed

docker network inspect does not show container in endpoint list if powered on out-of-band #8052

hickeng opened this issue Jun 14, 2018 · 4 comments
Assignees
@yuyangbj
Labels
component/persona/docker component/portlayer/network help wanted impact/test/integration Requires creation of or changes to an integration test kind/defect Behavior that is inconsistent with what's intended

Comments

@hickeng
Copy link
Member

hickeng commented Jun 14, 2018 •
edited
Loading

This issue applies only if the container has been powered on out-of-band. The container is also omitted if it is powered off, but that aligns with regular docker behaviour (although possibly counter-intuitive for a VCH where the connection to the network is independent of an assigned address on it).

We see the same output irrespective of whether the container is just created, powered off, or powered on by out-of-band means.

My primary concern here is whether this occurs during an HA restart.

$ docker inspect bc
<snip>
           "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "",
                    "EndpointID": "1184",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                },
            }

$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "fcb73770a434c90283eacc452f94ce6a9c5b3b66ee3bfb1c4925d877d2b71a25",
        "Created": "2018-06-14T21:22:00.058942976Z",
        "Scope": "",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.16.0.0/16",
                    "Gateway": "172.16.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]
Steps to recreate

$ docker run -dit alpine /bin/ash
62fee89988390b087fe1e8121ab4d452e9d33821ab30430178bfb40c22bc4c6d
$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "de3cf7e053da3842041bd20f6263144239d618f7c98e8347fc00ba318da86676",
        "Created": "2018-06-15T16:53:18.885014671Z",
        "Scope": "",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.16.0.0/16",
                    "Gateway": "172.16.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "62fee89988390b087fe1e8121ab4d452e9d33821ab30430178bfb40c22bc4c6d": {
                "Name": "admiring_carson",
                "EndpointID": "62fee89988390b087fe1e8121ab4d452e9d33821ab30430178bfb40c22bc4c6d",
                "MacAddress": "",
                "IPv4Address": "172.16.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]
$ docker stop -t 1 62
62
$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "de3cf7e053da3842041bd20f6263144239d618f7c98e8347fc00ba318da86676",
        "Created": "2018-06-15T16:55:31.173853457Z",
        "Scope": "",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.16.0.0/16",
                    "Gateway": "172.16.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                        PORTS               NAMES
62fee8998839        alpine              "/bin/ash"          11 minutes ago      Exited (143) 19 seconds ago                       admiring_carson
$ govc vm.power -on admiring_carson*
Powering on VirtualMachine:46... OK
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
62fee8998839        alpine              "/bin/ash"          11 minutes ago      Up 2 seconds                            admiring_carson
$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "de3cf7e053da3842041bd20f6263144239d618f7c98e8347fc00ba318da86676",
        "Created": "2018-06-15T16:56:20.580027666Z",
        "Scope": "",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.16.0.0/16",
                    "Gateway": "172.16.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]
$ docker inspect admiring_carson
[
    {
        "Id": "62fee89988390b087fe1e8121ab4d452e9d33821ab30430178bfb40c22bc4c6d",
        "Created": "2018-06-15T16:52:47.256652718Z",
        "Path": "/bin/ash",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": -1,
            "Error": "",
            "StartedAt": "2018-06-15T16:56:10Z",
            "FinishedAt": ""
        },
        "Image": "3fd9065eaf02feaf94d68376da52541925650b81698c53c6824d92ff63f98353",
        "ResolvConfPath": "",
        "HostnamePath": "",
        "HostsPath": "",
        "LogPath": "",
        "Name": "/admiring_carson",
        "RestartCount": 0,
        "Driver": "vSphere Integrated Containers v1.5.0-dev-99999-06ba697 Backend Engine",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "bridge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "vSphere Integrated Containers v1.5.0-dev-99999-06ba697 Backend Engine",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 0,
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "vSphere Integrated Containers v1.5.0-dev-99999-06ba697 Backend Engine",
            "Data": null
        },
        "Mounts": null,
        "Config": {
            "Hostname": "62fee8998839",
            "Domainname": "",
            "User": "",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm"
            ],
            "Cmd": [
                "/bin/ash"
            ],
            "ArgsEscaped": true,
            "Image": "alpine",
            "Volumes": {},
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "",
                    "EndpointID": "1184",
                    "Gateway": "172.16.0.1/16",
                    "IPAddress": "172.16.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]
$

@hickeng hickeng added kind/defect Behavior that is inconsistent with what's intended component/persona/docker component/portlayer/network help wanted impact/test/integration Requires creation of or changes to an integration test labels Jun 14, 2018
@hickeng hickeng changed the title docker network inspect does not show container list on bridge networks docker network inspect does not show container in endpoint list if powered on out-of-band Jun 14, 2018
@hickeng hickeng mentioned this issue Jun 14, 2018
Closed
@wjun
Copy link
Contributor

wjun commented Jun 25, 2018 •
edited
Loading

The root cause looks during handleEvent https://github.com/vmware/vic/blob/master/lib/portlayer/network/network.go#L120
we need to handle containerPoweredOn event to bindContainer to the network. I also noticed that the container port mapping to host port lost if it is powered on out-of-band.

@hickeng
Copy link
Member Author

hickeng commented Jul 11, 2018

@wjun Correct (and congrats on tracking that down).
The previously assigned IP that is recorded in the cVM config should be used as a hint (#7128).

A fix also needs to ensure that a powerOn event in the docker personality causes a port mapping to be added if not already present.

@wjun
Copy link
Contributor

wjun commented Dec 14, 2018

This is related to #8405 Yang is fixing to process containerPoweredOn event.

@wjun wjun assigned yuyangbj and unassigned wjun Dec 14, 2018
@yuyangbj
Copy link
Contributor

@hickeng @wjun with the fix #8405, this issue does not appear. The steps are:

1> list all containers

yuyangbj@yuyangbj-vic:~/GoProjects/src/github.com/yuyangbj/vic/bin$ docker --tls ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
eb7769610a57 httpd "httpd-foreground" 2 days ago Up 2 days 10.193.51.161:8081->80/tcp portmapping5
c15ae94b9a86 httpd "httpd-foreground" 2 days ago Up 2 days 10.193.51.161:8080->80/tcp portmapping3
315855f7ee28 httpd "httpd-foreground" 2 days ago Up 2 days 50.0.9.2:0->0/, 50.0.9.2:8080->80/tcp portmapping1
b2f8feff9adf httpd "httpd-foreground" 2 days ago Up 2 days 50.0.9.103:0->0/, 50.0.9.103:8080->80/tcp portmapping2

yuyangbj@yuyangbj-vic:~/GoProjects/src/github.com/yuyangbj/vic/bin$ docker --tls network inspect routable
[
{
"Name": "routable",
"Id": "efb5c4bb04fee276b8d4719632042117fa2f39f38f996d6fdbb2ab9c9dfb9a1c",
"Created": "2018-12-17T06:22:36.495267428Z",
"Scope": "",
"Driver": "external",
"EnableIPv6": false,
"IPAM": {
"Driver": "",
"Options": {},
"Config": [
{
"Subnet": "50.0.9.0/24",
"Gateway": "50.0.9.1"
}
]
},
"Internal": false,
"Attachable": false,
"Containers": {
"315855f7ee284d3ffc70403cc9c6de03bbb94a21a08b807ad94b2d0b65397552": {
"Name": "portmapping1",
"EndpointID": "315855f7ee284d3ffc70403cc9c6de03bbb94a21a08b807ad94b2d0b65397552",
"MacAddress": "",
"IPv4Address": "50.0.9.2/24",
"IPv6Address": ""
},
"b2f8feff9adf136ec73671f36d97fa6b801715546ceee473f014b75aa1a52e54": {
"Name": "portmapping2",
"EndpointID": "b2f8feff9adf136ec73671f36d97fa6b801715546ceee473f014b75aa1a52e54",
"MacAddress": "",
"IPv4Address": "50.0.9.103/24",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]

2> Power off the container VM from vSphere client
yuyangbj@yuyangbj-vic:~/GoProjects/src/github.com/yuyangbj/vic/bin$ docker --tls ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
eb7769610a57 httpd "httpd-foreground" 2 days ago Up 2 days 10.193.51.161:8081->80/tcp portmapping5
c15ae94b9a86 httpd "httpd-foreground" 2 days ago Up 2 days 10.193.51.161:8080->80/tcp portmapping3
b2f8feff9adf httpd "httpd-foreground" 2 days ago Up 2 days 50.0.9.103:0->0/*, 50.0.9.103:8080->80/tcp portmapping2

yuyangbj@yuyangbj-vic:~/GoProjects/src/github.com/yuyangbj/vic/bin$ docker --tls network inspect routable
[
{
"Name": "routable",
"Id": "efb5c4bb04fee276b8d4719632042117fa2f39f38f996d6fdbb2ab9c9dfb9a1c",
"Created": "2018-12-17T06:23:35.884309679Z",
"Scope": "",
"Driver": "external",
"EnableIPv6": false,
"IPAM": {
"Driver": "",
"Options": {},
"Config": [
{
"Subnet": "50.0.9.0/24",
"Gateway": "50.0.9.1"
}
]
},
"Internal": false,
"Attachable": false,
"Containers": {
"b2f8feff9adf136ec73671f36d97fa6b801715546ceee473f014b75aa1a52e54": {
"Name": "portmapping2",
"EndpointID": "b2f8feff9adf136ec73671f36d97fa6b801715546ceee473f014b75aa1a52e54",
"MacAddress": "",
"IPv4Address": "50.0.9.103/24",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]

3> Power on the container VM from vSphere client, everything looks good

yuyangbj@yuyangbj-vic:/GoProjects/src/github.com/yuyangbj/vic/bin$ docker --tls ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
80c8f5d5fdea httpd "httpd-foreground" 2 days ago Created portmapping6
eb7769610a57 httpd "httpd-foreground" 2 days ago Up 2 days 10.193.51.161:8081->80/tcp portmapping5
8df096ff571b httpd "httpd-foreground" 2 days ago Created portmapping4
c15ae94b9a86 httpd "httpd-foreground" 2 days ago Up 2 days 10.193.51.161:8080->80/tcp portmapping3
315855f7ee28 httpd "httpd-foreground" 2 days ago Up 10 seconds 50.0.9.2:0->0/, 50.0.9.2:8080->80/tcp portmapping1
b2f8feff9adf httpd "httpd-foreground" 2 days ago Up 2 days 50.0.9.103:0->0/, 50.0.9.103:8080->80/tcp portmapping2
yuyangbj@yuyangbj-vic:/GoProjects/src/github.com/yuyangbj/vic/bin$ docker --tls network inspect routable
[
{
"Name": "routable",
"Id": "efb5c4bb04fee276b8d4719632042117fa2f39f38f996d6fdbb2ab9c9dfb9a1c",
"Created": "2018-12-17T06:24:54.782475216Z",
"Scope": "",
"Driver": "external",
"EnableIPv6": false,
"IPAM": {
"Driver": "",
"Options": {},
"Config": [
{
"Subnet": "50.0.9.0/24",
"Gateway": "50.0.9.1"
}
]
},
"Internal": false,
"Attachable": false,
"Containers": {
"315855f7ee284d3ffc70403cc9c6de03bbb94a21a08b807ad94b2d0b65397552": {
"Name": "portmapping1",
"EndpointID": "315855f7ee284d3ffc70403cc9c6de03bbb94a21a08b807ad94b2d0b65397552",
"MacAddress": "",
"IPv4Address": "50.0.9.2/24",
"IPv6Address": ""
},
"b2f8feff9adf136ec73671f36d97fa6b801715546ceee473f014b75aa1a52e54": {
"Name": "portmapping2",
"EndpointID": "b2f8feff9adf136ec73671f36d97fa6b801715546ceee473f014b75aa1a52e54",
"MacAddress": "",
"IPv4Address": "50.0.9.103/24",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]

This was referenced Jan 3, 2019
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuyangbj yuyangbj

Labels
component/persona/docker component/portlayer/network help wanted impact/test/integration Requires creation of or changes to an integration test kind/defect Behavior that is inconsistent with what's intended
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wjun @hickeng @yuyangbj