NetworkMode overwrites NetworkingConfig and cannot be omitted

[shadjiiski]

I am trying to create a network via docker remote API on VIC and also attach some containers to this network during the network creation. In docker, there are two ways to achieve this: the old one is specifying the name of the network as a value for the NetworkMode property and the new one is providing a NetworkingConfig. These are also available on VIC, but there are a few major differences from traditional docker:

• If NetworkingConfig is provided, NetworkMode cannot be ommited on VIC. This is not the case on Docker
• If NetworkMode is provided along with NetworkingConfig, NetworkMode will overwrite the provided configuration if its value differes from a network name. This means that if I specify NetworkMode: bridge, it doesn't matter what networks I will specify in the NetworkingConfig, they will be ignored and the container will be attached to the bridge network only
• Providing NetworkMode: null is the same as providing NetworkMode: bridge

VIC Version: Jfrog bintray build 5316

Code on Docker

# Create a network
curl --unix-socket /var/run/docker.sock  -H "Content-Type: application/json" -d '{"Name": "alp-network"}' https:/networks/create
# {"Id":"ac7ffb0ce6dd980f283917e0fa5d362c07fb7a85b48cef303274af9f7bd1196e","Warning":""}

# Create a container. Provide only NetworkMode
curl --unix-socket /var/run/docker.sock  -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkMode": "alp-network"}' http:/containers/create?name=test-alp-1
# {"Id":"35eba84af142580fb7ba0b61e8e09e5e8d98f44953ccf6429467509dd6c2c54c","Warnings":null}

# Create a container. Specify only NetworkingConfig
curl --unix-socket /var/run/docker.sock  -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkingConfig": {"EndpointsConfig": {"alp-network": {} } } }' http:/containers/create?name=test-alp-2
# {"Id":"dcd6d8283a3ebc011a1ae82563723143f5b78faafeaf9c7b0702bdcc94231c7a","Warnings":null}

# Create a container. Specify both properties. NetworkMode is the name of the network
curl --unix-socket /var/run/docker.sock  -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkMode": "alp-network", "NetworkingConfig": {"EndpointsConfig": {"alp-network": {} } } }' http:/containers/create?name=test-alp-3
# {"Id":"f3ef93d4229b7514b07dc7da20bab1653dae0b9b24bfb7707ab644d15782562e","Warnings":null}

# Create a container. Specify both properties. NetworMode is bridge
curl --unix-socket /var/run/docker.sock  -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkMode": "bridge", "NetworkingConfig": {"EndpointsConfig": {"alp-network": {} } } }' https:/containers/create?name=alp-4
# {"Id":"1b1a163eb303ba476aecce876c02c8798b73ebdc9f890032c8b3a991ef8ef01f","Warnings":null}

# Start all the containers with successive docker start commands

Code on VIC

# Create a network:
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -d '{"Name": "alp-network"}' https://$IP:2376/networks/create
# {"Id":"7951d04936d187053bafa9bed70a1a2ca235dfcfbfd1a458f2d94156a5f0695c","Warning":""}

# Create a container. Provide only NetworkMode
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkMode": "alp-network"}' https://$IP:2376/containers/create?name=alp-1
# {"Id":"23c23adf83a0230be2219994c26b387b7e110584a229060cbe6a95283611572a","Warnings":null}

# Create a container. Specify only NetworkingConfig
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkingConfig": {"EndpointsConfig": {"alp-network": {} } } }' https://$IP:2376/containers/create?name=alp-2
# Bad request error from portlayer: invalid config

# Create a container. Specify both properties. NetworkMode is the name of the network
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkMode": "alp-network", "NetworkingConfig": {"EndpointsConfig": {"alp-network": {} } } }' https://$IP:2376/containers/create?name=alp-3
# {"Id":"0aa7c2fc2a687d70e2e8e0debd929004c75827fd6a702a28d4d172d68b82b253","Warnings":null}

# Create a container. Specify both properties. NetworMode is bridge
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkMode": "bridge", "NetworkingConfig": {"EndpointsConfig": {"alp-network": {} } } }' https://$IP:2376/containers/create?name=alp-4
# {"Id":"4a4dbc11ab1b9f6ba3bbcae7b7851438b9b6c682a9a52c594920143183305c16","Warnings":null}

# Start all the containers with successive docker start commands

Expected behavior:
4 containers, created, then started, all of them connected to the created alp-network. This is the output of the docker network inspect command on Docker host:

docker network inspect alp-network
[
    {
        "Name": "alp-network",
        "Id": "ac7ffb0ce6dd980f283917e0fa5d362c07fb7a85b48cef303274af9f7bd1196e",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1/16"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "1b1a163eb303ba476aecce876c02c8798b73ebdc9f890032c8b3a991ef8ef01f": {
                "Name": "alp-4",
                "EndpointID": "93cb10f3bd2a825a913a37c0f0f6f7fff86f2371e10c95d6d603bcb1b8780bb8",
                "MacAddress": "02:42:ac:12:00:05",
                "IPv4Address": "172.18.0.5/16",
                "IPv6Address": ""
            },
            "35eba84af142580fb7ba0b61e8e09e5e8d98f44953ccf6429467509dd6c2c54c": {
                "Name": "test-alp-1",
                "EndpointID": "0645749ac7a4ddfc92b437826a500996797a380fbdc573c596a91f6d689736df",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "dcd6d8283a3ebc011a1ae82563723143f5b78faafeaf9c7b0702bdcc94231c7a": {
                "Name": "test-alp-2",
                "EndpointID": "b849ae57568544621382ad701aed3309c54e7f08937a531770058f0b88dea591",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "f3ef93d4229b7514b07dc7da20bab1653dae0b9b24bfb7707ab644d15782562e": {
                "Name": "test-alp-3",
                "EndpointID": "01ecc716d49ecc7a9b3192c871c790f110538f9cd85265f033f34f616d255985",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Actual behavior:
Only three containers are created (alp-2 creation failed), only two of them are connected to the created network, the last one is connected to bridge:

# alp-network:
 docker --tls network inspect alp-network
[
    {
        "Name": "alp-network",
        "Id": "7951d04936d187053bafa9bed70a1a2ca235dfcfbfd1a458f2d94156a5f0695c",
        "Scope": "",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "0aa7c2fc2a687d70e2e8e0debd929004c75827fd6a702a28d4d172d68b82b253": {
                "Name": "alp-3",
                "EndpointID": "0aa7c2fc2a687d70e2e8e0debd929004c75827fd6a702a28d4d172d68b82b253",
                "MacAddress": "",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            },
            "682534adcef9444730f2eb6d24e3ca4c34267f5a612be93db5ae7a4118856e08": {
                "Name": "alp-1",
                "EndpointID": "682534adcef9444730f2eb6d24e3ca4c34267f5a612be93db5ae7a4118856e08",
                "MacAddress": "",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            },
        },
        "Options": {},
        "Labels": {}
    }
]

# bridge network
docker --tls network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "f8bcd3c67f4741106b30462c3b4c026adf984d9f42e55c6c980f68fb85f2b727",
        "Scope": "",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.16.0.0/16",
                    "Gateway": "172.16.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "4a4dbc11ab1b9f6ba3bbcae7b7851438b9b6c682a9a52c594920143183305c16": {
                "Name": "alp-4",
                "EndpointID": "4a4dbc11ab1b9f6ba3bbcae7b7851438b9b6c682a9a52c594920143183305c16",
                "MacAddress": "",
                "IPv4Address": "172.16.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

[mdubya66]

@hmahmood please triage. including it in Hardening sprint until more info is known.

[hmahmood]

Looks like NetworkingConfig always overrides NetworkMode in docker. I believe we just use NetworkMode, which explains the differences.

[mdubya66]

Targeted for hardening sprint by 10/21

[shadjiiski]

@hmahmood , @mdubya66 it also looks like having the NetworkMode set is actually preventing containers from being added to networks with the connect command:

# create a network
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -d '{"Name": "alp-network"}' https://$IP:2376/networks/create

# create a container with NetworkMode set to bridge
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkMode": "bridge" }' https://$IP:2376/containers/create?name=test-alp-1

# start the container
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -X POST https://$IP:2376/containers/test-alp-1/start

# try to connect the container to the network
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -d '{"Container": "test-alp-1"}' https://$IP:2376/networks/alp-network/connect

# inspect networks
docker --tls network inspect alp-network
[
    {
        "Name": "alp-network",
        "Id": "b9d960e7862bbe63e1a7cbdc0cc347758d192f16aae46e61be57886331d7ca61",
        "Scope": "",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.24.0.0/16",
                    "Gateway": "172.24.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]

 docker --tls network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "00bf5a88c2e174545e39f738b6004c478c328338b6d45b0a67cef5d1f0e5552a",
        "Scope": "",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.16.0.0/16",
                    "Gateway": "172.16.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "573b7972c6d7920f6ecad37c9d47e62714bf53928aa1b8769055261f7498ab16": {
                "Name": "test-alp-1",
                "EndpointID": "573b7972c6d7920f6ecad37c9d47e62714bf53928aa1b8769055261f7498ab16",
                "MacAddress": "",
                "IPv4Address": "172.16.0.2/16",
                "IPv6Address": ""
            },
            "84e3f7b54bff6f6abe1c300ebcd35d0187f3314756be0c35c70684f2ef57e11c": {
                "Name": "mysql-mcm882_24166287599",
                "EndpointID": "84e3f7b54bff6f6abe1c300ebcd35d0187f3314756be0c35c70684f2ef57e11c",
                "MacAddress": "",
                "IPv4Address": "172.16.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

The same code on docker host will have the test-alp-1 container added to both alp-network and bridge

[hmahmood]

network connect for running containers is not currently implemented: #745 #743

[shadjiiski]

@hmahmood, thanks for the update. I have confirmed that connect before start works as expected

[chengwang86]

@jzt @shadjiiski @andrewtchin @hmahmood

Due to https://github.com/vmware/vic/blob/master/lib/apiservers/engine/backends/container.go#L1342
if the "HostConfig" is not specified in the form data of the docker rest api call, vic would throw an error msg

time=2016-12-13T14:00:51.474679848Z level=debug msg=[BEGIN] [github.com/vmware/vic/lib/apiservers/engine/backends.validateCreateConfig:1336] Container.validateCreateConfig 
time=2016-12-13T14:00:51.474750805Z level=debug msg=[ END ] [github.com/vmware/vic/lib/apiservers/engine/backends.validateCreateConfig:1336] [67.144µs] Container.validateCreateConfig 
time=2016-12-13T14:00:51.474758553Z level=debug msg=[ END ] [github.com/vmware/vic/lib/apiservers/engine/backends.(*Container).ContainerCreate:223] [226.937µs]  
2016/12/13 14:00:51 http: panic serving 192.168.60.129:59356: runtime error: invalid memory address or nil pointer dereference
goroutine 199 [running]:
net/http.(*conn).serve.func1(0xc420c9af00)
    /usr/local/go/src/net/http/server.go:1491 +0x12a
panic(0x1073680, 0xc42000e060)
    /usr/local/go/src/runtime/panic.go:458 +0x243
github.com/vmware/vic/lib/apiservers/engine/backends.validateCreateConfig(0xc4207b1540, 0x0, 0x0)
    /home/wangcheng/go/src/github.com/vmware/vic/lib/apiservers/engine/backends/container.go:1342 +0xac
github.com/vmware/vic/lib/apiservers/engine/backends.(*Container).ContainerCreate(0xc420468910, 0xc42047390d, 0x5, 0xc420ccb440, 0x0, 0xc42001cec0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/wangcheng/go/src/github.com/vmware/vic/lib/apiservers/engine/backends/container.go:246 +0x1fd
github.com/vmware/vic/vendor/github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersCreate(0xc4203c62d0, 0x7f8b86506938, 0xc42043e030, 0x1d8fb60, 0xc420cab1e0, 0xc420cce870, 0xc420473b90, 0x1d90d60, 0xc42043e030)

so this rest api call

# Create a container. Provide only NetworkMode
curl --insecure --cert cert.pem --key key.pem -H "Content-Type: application/json" -d '{"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkMode": "alp-network"}' https://$IP:2376/containers/create?name=alp-1

does not work on vic (but it still works on regular docker). We need to explicitly specify the "HostConfig" in the form data, e.g.

curl -H "Content-Type: application/json" -d '{"HostConfig":{"CpuShares":0,"Memory":0},"Image": "alpine", "Cmd": ["ping", "127.0.0.1"], "NetworkMode": "alp-network"}' http://192.168.60.130:2375/containers/create?name=alp-1

I think either (1) we should change the code such that an empty HostConfig is created with default values if not specified in the form-data, or (2) we just ask the customer to include HostConfig in their form data for the rest api calls.

[jzt]

@chengwang86 I think it would be simple enough (and appropriate) to update our docker persona to provide a default HostConfig if one is not submitted by the user. Can you open an issue to capture it? Thanks!

[chengwang86]

This issue should be fixed after merging #3700 and fixing #3709.

Then the only difference with regular docker is: our networkConfig would override networkmode, so the "networkmode" under vic becomes "alp-network" in all the four cases, whereas on regular docker the "networkmode" could still be "default" in case 2 and "bridge" in case 4.

[mhagen-vmware]

@chengwang86 Can you please add a test to the PR for this case?