k8s: Unexpected edge to the Internet node

[2opremio]

I expect the nginx container to talk to all the hello nodes but it's only talking to one.

Same repro as #1404 but using service instead of talking directly to pods, i.e.

upstream hello {
    server hello.default.svc.cluster.local;
        keepalive 100;
    }

upstream auth {
    server auth.default.svc.cluster.local;
        keepalive 100;
    }

server {
    listen 443;
    ssl    on;

    ssl_certificate     /etc/tls/cert.pem;
    ssl_certificate_key /etc/tls/key.pem;

    proxy_http_version 1.1;
    proxy_set_header Connection "";

    location / {
            proxy_pass http://hello;
    }

    location /login {
            proxy_pass http://auth;
    }

}

Note how nginx seems to be talking to the internet instead of the other too hello containers

[2opremio]

I am pretty sure this is happening because the nat mapper doesn't handle DNATing

[2opremio]

OK, it seems that keepalive in nginx doesn't work as I expected and the frontend is actually only talking to one of the pods (it seems nginx is simply reusing the same connection, not letting the service load-balance). So, there should only be an edge to one of the containers. However, there shouldn't be an edge to the internet node.

The nginx container and the hello container between which there is an edge are not colocated and netstat in the hello container with an edge shows a connection between the pod ips:

/ # netstat -n                                                                                                                        
Active Internet connections (w/o servers)                                                                                             
Proto Recv-Q Send-Q Local Address           Foreign Address         State                                                             
tcp        0      0 ::ffff:10.36.1.6:80     ::ffff:10.36.2.5:36635  ESTABLISHED                                                       
Active UNIX domain sockets (w/o servers)                                                                                              
Proto RefCnt Flags       Type       State         I-Node Pat

However, netstat is empty in the other two hello containers with missing edges.

And in the nginx container I see a connection to the Virtual service IP:

root@frontend-3616253037-5laba:/# netstat -n | grep ESTA                                                                              
tcp        0      0 10.36.2.5:36635         10.39.249.88:80         ESTABLISHED                                                       
root@frontend-3616253037-5laba:/#                                                                                                     
Connected

which I believe it's causing the outbound connection to the internet node. From the analysis in #1408 (comment) I believe that the natting information is available to conntrack but we are not using it properly for persistent connections.

[2opremio]

Here's a report with the problem: report.json.gz

This patch solves it:

--- report_pprint.json  2016-05-11 11:16:48.000000000 +0100
+++ report_pprint_corrected.json    2016-05-11 11:20:55.000000000 +0100
@@ -9615,17 +9615,15 @@
       ";10.36.2.5;39107": {
         "id": ";10.36.2.5;39107",
         "topology": "endpoint",
         "counters": {},
         "sets": {},
         "adjacency": [
-          ";10.36.0.4;80",
-          ";10.39.249.88;80"
+          ";10.36.0.4;80"
         ],
         "edges": {
-          ";10.39.249.88;80": {},
           ";10.36.0.4;80": {}
         },
         "controls": {},
         "latest": {
           "procspied": {
             "timestamp": "2016-05-11T09:53:42.422846453Z",

So the nat mapper on nginx's end should add the DNATed edge (10.36.2.5;39107 -> ;10.36.0.4;80) and not (10.36.2.5;39107 -> ;10.39.249.88;80)

That should be doable for new connections but I am not so sure we get conntrack flows for preexisting connections (which would also affect SNATing of containers)

I will work on a fix

[2opremio]

I am not so sure we get conntrack flows for preexisting connections (which would also affect SNATing of containers)

It seems we do (we call conntrack -L in the conntrack flow walker) but maybe the persistent connections are not ready when the nat mapper walks over them with those flows.