Wrap IPv6 endpoints in []

[rlguarino]

It seems only thing that is currently stopping IPv6 services from working right now with the NGINX ingress controller (ignoring kube-proxy) is wrapping IPv6 IPs in [] in the upstream.

An example of the error i'm seeing:

I0609 00:53:33.451185       7 queue.go:86] syncing kube-system/default-http-backend
I0609 00:53:35.318753       7 controller.go:1062] getting endpoints for service kube-system/default-http-backend and port 8080
I0609 00:53:35.318797       7 controller.go:1111] endpoints found: [{f00d::a29e:4177:0:e992 8080 0 0}]
I0609 00:53:35.319002       7 controller.go:1062] getting endpoints for service kube-system/default-http-backend and port 8080
I0609 00:53:35.319020       7 controller.go:1111] endpoints found: [{f00d::a29e:4177:0:e992 8080 0 0}]
I0609 00:53:35.319041       7 controller.go:434] obtaining information about stream services of type TCP located in configmap 
I0609 00:53:35.319049       7 controller.go:434] obtaining information about stream services of type UDP located in configmap 
I0609 00:53:35.319147       7 nginx.go:470] adjusting ServerNameHashBucketSize variable to 32
I0609 00:53:35.319295       7 template.go:100] NGINX configuration: {"ProxySetHeaders":{},"MaxOpenFiles":1024,"BacklogSize":65535,"Backends":[{"name":"upstream-default-backend","service":null,"port":0,"secure":false,"secureCert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"f00d::a29e:4177:0:e992","port":"8080","maxFails":0,"failTimeout":0}],"SessionAffinity":{"name":"","CookieSessionAffinity":{"name":"","hash":""}}}],"PassthroughBackends":null,"Servers":[{"hostname":"_","sslPassthrough":false,"sslCertificate":"/ingress-controller/ssl/default-fake-certificate.pem","sslPemChecksum":"0ffa3fcfbb3dde6dfd2b8aa57402dc2a2ebbf763","locations":[{"path":"/","isDefBackend":true,"backend":"upstream-default-backend","service":null,"port":0,"basicDigestAuth":{"type":"","realm":"","file":"","secured":false},"Denied":null,"externalAuth":{"url":"","host":"","signinUrl":"","method":"","sendBody":false,"responseHeaders":null},"rateLimit":{"connections":{"name":"","limit":0,"burst":0,"sharedSize":0},"rps":{"name":"","limit":0,"burst":0,"sharedSize":0}},"redirect":{"target":"","addBaseUrl":false,"sslRedirect":false,"forceSSLRedirect":false,"appRoot":""},"whitelist":{"cidr":null},"proxy":{"bodySize":"1m","conectTimeout":5,"sendTimeout":60,"readTimeout":60,"bufferSize":"4k","cookieDomain":"off","cookiePath":"off"},"certificateAuth":{"AuthSSLCert":{"secret":"","caFilename":"","pemSha":""},"validationDepth":0},"use-port-in-redirects":false,"configuration-snippet":""}]}],"TCPBackends":[],"UDPBackends":[],"HealthzURI":"/healthz","CustomErrors":false,"Cfg":{"app-root":"","custom-http-errors":null,"proxy-body-size":"1m","proxy-connect-timeout":5,"proxy-read-timeout":60,"proxy-send-timeout":60,"proxy-buffer-size":"4k","proxy-cookie-path":"off","proxy-cookie-domain":"off","Resolver":["169.254.255.254"],"skip-access-log-urls":[],"ssl-redirect":true,"force-ssl-redirect":false,"use-port-in-redirects":false,"upstream-max-fails":0,"upstream-fail-timeout":0,"whitelist-source-range":[],"enable-dynamic-tls-records":true,"client-header-buffer-size":"1k","client-body-buffer-size":"8k","enable-underscores-in-headers":false,"ignore-invalid-headers":true,"vts-status-zone-size":"10m","retry-non-idempotent":false,"error-log-level":"notice","http2-max-field-size":"4k","http2-max-header-size":"16k","hsts":true,"hsts-include-subdomains":true,"hsts-max-age":"15724800","keep-alive":75,"keep-alive-requests":100,"large-client-header-buffers":"4 8k","log-format-upstream":"%v - [$the_x_forwarded_for] - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status","log-format-stream":"[$time_local] $protocol $status $bytes_sent $bytes_received $session_time","max-worker-connections":16384,"map-hash-bucket-size":64,"proxy-real-ip-cidr":"0.0.0.0/0","server-name-hash-max-size":1024,"server-name-hash-bucket-size":32,"server-tokens":true,"ssl-ciphers":"ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA","ssl-ecdh-curve":"secp384r1","ssl-protocols":"TLSv1 TLSv1.1 TLSv1.2","ssl-session-cache":true,"ssl-session-cache-size":"10m","ssl-session-tickets":true,"ssl-session-timeout":"10m","ssl-buffer-size":"4k","use-gzip":true,"use-http2":true,"gzip-types":"application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component","worker-processes":"32","load-balance":"least_conn"},"IsIPV6Enabled":true}
W0609 00:53:35.342634       7 queue.go:88] requeuing kube-system/default-http-backend, err 
-------------------------------------------------------------------------------
Error: exit status 1
2017/06/09 00:53:35 [emerg] 48#48: invalid port in upstream "f00d::a29e:4177:0:e992:8080" in /tmp/nginx-cfg970416361:164
nginx: [emerg] invalid port in upstream "f00d::a29e:4177:0:e992:8080" in /tmp/nginx-cfg970416361:164
nginx: configuration file /tmp/nginx-cfg970416361 test failed
-------------------------------------------------------------------------------

I've proved this to myself by changing the line:

index c49ab8d..7a6e130 100644
--- a/home/ross/src/k8s.io/ingress/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl
+++ b/nginx.tmpl
@@ -226,7 +226,7 @@ http {
         {{ $cfg.LoadBalanceAlgorithm }};
         {{ end }}
         {{ end }}
-        {{ range $server := $upstream.Endpoints }}server {{ $server.Address }}:{{ $server.Port }} max_fails={{ $server.MaxFails }} fail_timeout={{ $server.FailTimeout }};
+        {{ range $server := $upstream.Endpoints }}server [{{ $server.Address }}]:{{ $server.Port }} max_fails={{ $server.MaxFails }} fail_timeout={{ $server.FailTimeout }};
         {{ end }}
     }
     {{ end }}

Looking at where the address string is generated is really difficult, but it seems to me that these endpoints are returned to the nginx-ingress-controller from the API as a type string. So unfortunately there looks like there isn't a way to change how they are printed based on their type. Another approach would be to check if the IP has : in it, if it does surround it with [], but I'm not familiar with the code base to tell where that logic should go, if someone want's to point me in the right direction I'm more than happy to create a PR.

[aledbf]

@rlguarino we need to add a helper in the template (go code) to check if the IP address is IPV6
https://github.com/kubernetes/ingress/blob/master/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl#L236

{{ range $server := $upstream.Endpoints }}server {{ $server.Address }}:{{ $server.Port }} max_fails={{ $server.MaxFails }} fail_timeout={{ $server.FailTimeout }};
{{ range $server := $upstream.Endpoints }}server {{ if isIPV6 $server.Address }}[{{ $server.Address }}]{{ else }}{{ $server.Address }}:{{ $server.Port }} max_fails={{ $server.MaxFails }} fail_timeout={{ $server.FailTimeout }};

[rlguarino]

@aledbf I can do that. What about creating a helper called formatIP which takes an IP and changing the line to:

{{ range $server := $upstream.Endpoints }}server {{ $server.Address |  formatIP }}:{{ $server.Port }} max_fails={{ $server.MaxFails }} fail_timeout={{ $server.FailTimeout }};

[rlguarino]

Looking at the code some more there is already a helper function "contains" which could be used like:

{{ range $server := $upstream.Endpoints }}server {{ if contains $server.Address ":" }}[{{ $server.Address }}]{{ else }}{{ $server.Address }}:{{ $server.Port }} max_fails={{ $server.MaxFails }} fail_timeout={{ $server.FailTimeout }};

[aledbf]

@rlguarino using contains seems the easiest way to fix this 👍

[rlguarino]

@aledbf I'm having trouble building the code and I'm not sure if it's me or not. I sent you a message on slack but the gist of it is I'm seing this error:

ross@alnilam:~/src/k8s.io/ingress/controllers/nginx$ go build k8s.io/ingress/controllers/nginx/pkg/template
# k8s.io/ingress/core/pkg/ingress/annotations/authreq
../../core/pkg/ingress/annotations/authreq/main.go:134: ur.Hostname undefined (type *url.URL has no field or method Hostname)

which seems to come from this commit:

ross@alnilam:~/src/k8s.io/ingress/controllers/nginx$ git show d1e77f13
commit d1e77f132ff7a6713fb83992d560633779956714
Author: Manuel de Brito Fontes <[email protected]>
Date:   Wed Apr 26 23:52:03 2017 -0300

    Remove helper required in go < 1.8

diff --git a/core/pkg/ingress/annotations/authreq/main.go b/core/pkg/ingress/annotations/authreq/main.go
index 016cc26..8b15676 100644
--- a/core/pkg/ingress/annotations/authreq/main.go
+++ b/core/pkg/ingress/annotations/authreq/main.go
@@ -131,22 +131,10 @@ func (a authReq) Parse(ing *extensions.Ingress) (interface{}, error) {
 
        return &External{
                URL:             str,
-               Host:            stripPort(ur.Host),
+               Host:            ur.Hostname(),
                SigninURL:       signin,
                Method:          m,
                SendBody:        sb,
                ResponseHeaders: h,
        }, nil
 }
-
-// TODO: Remove after upgrade to Go 1.8
-func stripPort(hostport string) string {
-       colon := strings.IndexByte(hostport, ':')
-       if colon == -1 {
-               return hostport
-       }
-       if i := strings.IndexByte(hostport, ']'); i != -1 {
-               return strings.TrimPrefix(hostport[:i], "[")
-       }
-       return hostport[:colon]
-}

Do I need to update my vendor directory or something?

[aledbf]

../../core/pkg/ingress/annotations/authreq/main.go:134: ur.Hostname undefined (type *url.URL has no field or method Hostname)

You need go 1.8