[nginx] Support SSL for TCP

[od0]

NGINX supports SSL for opened TCP ports – it would be great to be able to use this feature when specifying TCP ports in the --tcp-services-configmap ConfigMap.

Right now the template looks like this:

        listen {{ $tcpServer.Port }};

Perhaps a new optional flag in the ConfigMap could be added and the template modified to something like this:

        listen {{ $tcpServer.Port }}{{ if $tcpServer.SSLEnabled }} ssl{{ end }};

Happy to take a stab at implementing this and open a PR.

Thanks!

[aledbf]

@od0 what's the benefit? Sorry to ask but I never used this option in the stream section.

[od0]

@aledbf I hadn't used it previously but discovered when trying to set up a SSL service on a port other than 443 (Elasticsearch in this case). I was curious if I could get it to work and discovered that NGINX supports it and it's as simple as adding a ssl after the port declaration (as is done automatically for 443 in this ingress).

https://www.nginx.com/resources/admin-guide/nginx-tcp-ssl-termination/

[aledbf]

Closing. Using ssl in nginx requires the ssl certificates.

[naveensrinivasan]

@aledbf We are trying to use nginx ingress for our mqtt service. The present ingress does not support ssl for tcp services. This would be a great feature for services like this.

What are your thoughts? Do you see issues?

[aledbf]

@naveensrinivasan why not use the ssl passthrough option and terminate ssl in the backend?

[naveensrinivasan]

@aledbf doing this dev/local where there aren't certs would be an issue.
IMO nginx ingress is there to do this kind of loading like SSL termination.

[stela]

I have a similar use-case as @od0 and @naveensrinivasan, want to terminate a connection to an XMPP server listening on a TCP port for client connections. I have a certificate from letsencrypt stored in a k8s Secret, which is easy to configure for a bunch of REST endpoints, each application having a k8s Ingress where tls.secretName points to the letsencrypt secret. The non-encrypted XMPP port 5222 is easy to forward, however I'd like to have a TLS-version exposed instead, would be ideal to have the certificate and private key in a single Secret, which cert-manager can renew automatically.

Or is there an easy workaround where you can terminate SSL on the backend using an existing Secret for the certificate? Examples?

Update: In case anybody else needs this, got it working by using an stunnel docker image, https://hub.docker.com/r/dweomer/stunnel/, as a second container in my pod, configured similarly to https://github.com/PalmStoneGames/kube-cert-manager/blob/master/docs/consume-certificates.md to use the key + certificate. You can see which env variables are available to set here: https://github.com/dweomer/dockerfiles-stunnel/blob/master/stunnel.sh. Set STUNNEL_CONNECT=localhost:unencrypted-port-nr. Then use an nginx-ingress ConfigMap to set up TCP ingress, see https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/examples/tcp/tcp-configmap-example.yaml

[jrodonnell]

Wish I found this thread before I opened an identical feature request the other day but commenting here because there's more traction. I realize I am the beggar in this situation but I strongly agree with @od0 and @naveensrinivasan, NGINX is used all over the world today for this exact purpose and I don't see why that functionality should not be exposed when it's deployed as an ingress controller in a Kubernetes cluster. Not every app can terminate ssl nor can every app be a stateless 12-factor RESTful web app and there are apps in the public Helm chart repo today that are forced to rely on things like NodePorts and manual management of external load balancers because there is no kube-native way to manage tcp traffic via Ingress objects or ingress controllers. Elasticsearch, Postgres, RabbitMQ, and Redis are four examples off the top of my head so the need for this functionality is certainly there.

[dannyk81]

@aledbf just wanted to share a similar use case as @stela, we are planning to move our XMPP application over to K8s and we need an SSL/TLS-over-TCP termination on the Ingress Controller.

We looked at having the termination done on the backend (e.g. using an stunnel for example), but that is not a good fit for us for various reasons.

We are now looking into running an haproxy ingress-controller side-by-side just for this application, as it seems to support that option (https://github.com/jcmoraisjr/haproxy-ingress#tcp-services-configmap) however we would prefer to have this in the nginx Ingress Controller and not introduce another piece to the mix.

[aledbf]

@dannyk81 after #2929 we could see exactly what's required to implement this feature. Right now the tcp/udp configuration is stored in a configmap, which only handle strings, where adding additional information is too painful.

[dannyk81]

@aledbf given your previous statement that this enhancement might be considered, could we re-open the issue? to ensure we track this.

Thanks!

[arianitu]

+1 I want to SSL terminate a bunch of services that are not http

[enicdep]

Also from my side the same. I want ingress-nginx able to terminate TLS for tcp (non-http) traffic. Very frustrated about this missing fuctionality

[the-nicolas]

Any update? Also struggling here. Just realized thsatt it is not working when trying to set it up. That's so basic that I even forgot to assume that it is not possible.

[isavcic]

I'm seeing a bunch of people, including myself, wanting the same thing which NGINX supports. Yet, there is this weird pushback from the maintainers of this project. Why is that?

[aledbf]

Yet, there is this weird pushback from the maintainers of this project. Why is that?

There is no such thing. Someone needs to work in this feature. Pull requests are always welcome.

That said, in my previous comment.

Right now the tcp/udp configuration is stored in a configmap, which only handle strings, where adding additional information is too painful.

Is hard to add this or any other additional feature with such restriction. This is the limitation.

Also keep in mind Ingress is only about HTTP/HTTPS and the existence of TCP/UDP in this project is because there is no service type=LoadBalancer option in baremetal. I wish I could remove the TCP/UDP feature in a separate project, but sadly it's too late for that.

[xiangwangcheng]

Looking forward to seeing this feature, too. I think ingress-nginx should support this since Nginx does.

[evgmoskalenko]

What about an update? :-)

[sedflix]

Hey all,

I was able to achieve this via a small(and quite bad) hack.
It involved updating a part of nginx.tmpl. The diif for the same is

--- a/helm-charts/nginx-ingress/nginx/nginx.tmpl
+++ b/helm-charts/nginx-ingress/nginx/nginx.tmpl
@@ -679,15 +679,15 @@ stream {
         }

         {{ range $address := $all.Cfg.BindAddressIpv4 }}
-        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
+        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} proxy_protocol ssl;
         {{ else }}
-        listen                  {{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
+        listen                  {{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} proxy_protocol ssl;
         {{ end }}
         {{ if $IsIPV6Enabled }}
         {{ range $address := $all.Cfg.BindAddressIpv6 }}
-        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
+        listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} proxy_protocol ssl;
         {{ else }}
-        listen                  [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
+        listen                  [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} proxy_protocol ssl;
         {{ end }}
         {{ end }}
         proxy_timeout           {{ $cfg.ProxyStreamTimeout }};
@@ -695,6 +695,21 @@ stream {
         {{ if $tcpServer.Backend.ProxyProtocol.Encode }}
         proxy_protocol          on;
         {{ end }}
+
+        proxy_protocol          on;
+
+        ssl_protocols {{ $cfg.SSLProtocols }};
+        ssl_ciphers '{{ $cfg.SSLCiphers }}';
+        ssl_prefer_server_ciphers on;
+
+        # PEM sha: {{ $cfg.DefaultSSLCertificate.PemSHA }}
+        ssl_certificate     {{ $cfg.DefaultSSLCertificate.PemFileName }};
+        ssl_certificate_key {{ $cfg.DefaultSSLCertificate.PemFileName }};
+
+
+        ssl_session_cache     shared:SSL2:60m;
+        ssl_session_timeout   10h;
+        ssl_handshake_timeout 30s;
     }
     {{ end }}

Reference:

• https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/custom-template/
• https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-tcp/

[rechandler12]

I've got another problem. I want SSL Passthrough to my mqtt broker but an error has occurred:

failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number.

I can connect from inside cluster but I have a problem to do it from the outside. So I assume that this is related with NGINX. My config:

controller:
  config:
    use-proxy-protocol: 'true'

  extraArgs:
    enable-ssl-passthrough: ""

  service:
    annotations:
      service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: 'true'

  metrics:
    enabled: true

    serviceMonitor:
      enabled: true
      additionalLabels:
        prometheus: doks-cluster-monitoring

tcp:
  8883: "mqtt/mosquitto:8883"

SOLUTION
It was caused by PROXY protocol. This config works:

controller:
  config:
    use-proxy-protocol: "true"

  service:
    annotations:
      service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"

  metrics:
    enabled: true

    serviceMonitor:
      enabled: true
      additionalLabels:
        prometheus: doks-cluster-monitoring

tcp:
  8883: "mqtt/mosquitto:8883:PROXY"

[lmserrano]

I've got this to work based on @sedflix comment, but with some changes in the ingress-nginx configuration from the Kubernetes side too, as well as nginx.tmpl changes that better integrate with it with Kubernetes.

To get SSL Termination working for TCP backends, I needed to do the following:

1. Temporarily run an ingress-nginx pod with default settings, and save a copy of its /etc/nginx/template/nginx.tmpl file
   • This is important because I've noticed there are some variations across versions and they can cause issues too. It's important to make sure you make edits over the default for your version
2. Create a new nginx.tmpl file based on the previous, but with the following changes in the # TCP Services section:

   # TCP services
   {{ range $tcpServer := .TCPBackends }}
   server {
   preread_by_lua_block {
   	ngx.var.proxy_upstream_name="tcp-{{ $tcpServer.Backend.Namespace }}-{{ $tcpServer.Backend.Name }}-{{ $tcpServer.Backend.Port }}";
   }
   
   {{ range $address := $all.Cfg.BindAddressIpv4 }}
   listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} ssl;
   {{ else }}
   listen                  {{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} ssl;
   {{ end }}
   {{ if $IsIPV6Enabled }}
   {{ range $address := $all.Cfg.BindAddressIpv6 }}
   listen                  {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }} ssl;
   {{ else }}
   listen                  [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol {{ end }} ssl;
   {{ end }}
   {{ end }}
   proxy_timeout           {{ $cfg.ProxyStreamTimeout }};
   proxy_pass              upstream_balancer;
   
   {{ if $tcpServer.Backend.ProxyProtocol.Encode }}
   proxy_protocol          on;
   {{ end }}
   
   ssl_protocols {{ $cfg.SSLProtocols }};
   ssl_ciphers '{{ $cfg.SSLCiphers }}';
   ssl_prefer_server_ciphers on;
   
   # PEM sha: {{ $cfg.DefaultSSLCertificate.PemSHA }}
   ssl_certificate     {{ $cfg.DefaultSSLCertificate.PemFileName }};
   ssl_certificate_key {{ $cfg.DefaultSSLCertificate.PemFileName }};
   
   
   ssl_session_cache     shared:SSL2:60m;
   ssl_session_timeout   10h;
   ssl_handshake_timeout 30s;
   }
   {{ end }}
   

   • The summary of these changes is, for the "TCP Services" section:
     • Add ssl after each proxy_protocol{{ end }}
     • Add the ssl_* lines after the $tcpServer.Backend.ProxyProtocol.Encode block
3. Make sure you have your certificates data stored in a secret of type kubernetes.io/tls, with data as follows:

   "ca.crt" = "path-to-ca-crt-file"
   "tls.crt" = "path-to-tls-crt"
   "tls.key" = "path-to-tls-key"
   

4. Update your ingress-nginx configuration:
   1. It should use the nginx.tmpl edited as previously described
   2. It should specify a unique ingress class name such as ingressClass: nginx-tls-tcp to avoid conflict with other regular default ingress-nginx installations
   3. It should have the following controller service annotations:

      "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": "*"
      "service.beta.kubernetes.io/aws-load-balancer-backend-protocol": "tcp"
      "service.beta.kubernetes.io/aws-load-balancer-ssl-ports": "https"
      "service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout": "60"
      
      # Also any other relevant annotations such as DNS ones (because unlike the defaults for HTTP or HTTPS, you will not be using an Ingress object for TCP)
      

   4. It should specify the following controller extra arguments:

      "default-ssl-certificate": "your-namespace/your-secret-with-tls-information"
      

   5. It should specify the desired TCP port mappings as follows:

      "5000": your-namespace/your-service:5000:PROXY"
      ...
      

Notes:

• In the event you want to expose some TCP services without SSL or have different default certificates or other mismatching settings in your configuration, you will need to have a deployment of ingress-nginx specific just for that. This is why setting the ingressClass: nginx-tls-tcp to a unique name for the purpose, is important.
• In the event that you are storing your certificates using AWS ACM, instead of specifying a Kubernetes secret with your TLS information and setting it as the default in the extra configuration args, you may be able to alternatively just set another annotation with value: "service.beta.kubernetes.io/aws-load-balancer-ssl-cert": "arn:aws:acm:XX-XXXX-X:XXXXXXXXX:certificate/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX" (although I did not test this in my use case).

Reference:

• https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/custom-template/
• https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-tcp/

[mdiianni]

Please reopen or share the link to the feature request.

[mdiianni]

@lmserrano any chance you could share extra details, I tried exactly the same steps as you posted here yet couldn't manage to make it work. Have you came across a different approach recently? thanks for reading.

[lmserrano]

@mdiianni I've got it to work with the described steps. In fact it was after I managed to find a working solution that I then documented and organized it to make the post so that it could serve as future reference for myself and for everyone else experiencing the same issue. There are some thumbs ups so it leads me to believe it has been working for other people too.

If you have have followed the configuration and it is not working now, it could be possible that the versions you are using could have changed since the time of the post. If you post more details about what you have, you may be able to get more help from the community. Some things worth checking:

1. Check that you are installing ingress-nginx and not another another nginx ingress. Also if you are using Helm charts for the installation, make sure you're using a currently maintained repository, since stable has been decommissioned and will progressively stop to serve charts. Confirm that the new source has the exact configuration and Kubernetes resources that you would expect.
2. Double check that the nginx.tmpl file of a fresh prod is really the starting source for the edits that you will make specific to the SSL TCP support. Each ingress-nginx version will often have slightly different nginx.tmpl files and if you use a base one not belonging to the version you are using, you are likely to encounter issues.
3. Make sure the changes referred to here are still valid for that version. Verify if the ingress controller actually runs successfully and if you can create an ingress working in normal conditions. Check for any hints in log files for clues of any problems related with configuration

If you are using a different cloud provider other than AWS you will also likely have to change some of the controller service annotations. Also, even for AWS, if you use AWS ACM for certificates you will need to configure the respective service annotations.

Hope that helps.

[xtianus79]

5 years later and this is not a standard feature? lol come on guys. http is not the only thing in the world. I think enough people said why they want to use this.

As well, it is highly recommended not to carry ssl traffic through to an MQTT broker. with that said, don't you think this would be a nice feature. Kubernetes is for clusters and besides Redis and Mongo DB this is a pretty important cluster.

[maelp]

Are we expecting this feature to be developped? @aledbf it could be as simple as having a format with a "-ssl" at the end like

{
   "tcp": {
        "8883": "mqtt/mqtt:8883-ssl"
   }
}

to allow to keep using simple strings

[mdiianni]

Alternatively I ended up using a stunnel container sidecar to do the TCP-TLS termination and proxy the connection to the container running the MQTT broker.

[alessandroros]

Alternatively I ended up using a stunnel container sidecar to do the TCP-TLS termination and proxy the connection to the container running the MQTT broker.

Can you describe better this approach? It would be really appriciated.

[mdiianni]

Alternatively I ended up using a stunnel container sidecar to do the TCP-TLS termination and proxy the connection to the container running the MQTT broker.

Can you describe better this approach? It would be really appriciated.

Sure no problem.

Basically what happens is:

• Device Sends Data --> Internet --> Load Balancer listening on port TCP/8443 --> Forwards to Kubernetes Service --> Deployment with Stunnel sidecar listening on port TCP/8443 --> If stunnel cert validation succeeds --> Forwards to Deployment App on port TCP/8080.

How to do it, example:

• Create a configmap with the stunnel configuration (this creates a start.sh script when mounted in the container) and deploy it. Notice the configuration assumes the certificates are also mounted on /conf/ssl.

kind: ConfigMap
apiVersion: v1
metadata:
  name: stunnel-conf
  namespace: default
data:
  start.sh: >
    #!/bin/sh

    apk update

    apk add --no-cache ca-certificates stunnel

    cat << EOF > /etc/stunnel/stunnel.conf

    options = NO_SSLv2

    options = NO_SSLv3

    options = NO_TLSv1

    options = NO_TLSv1.1

    CApath = /conf/ssl/

    pid = /var/run/stunnel.pid

    foreground = yes

    debug = 2


    [service-tls-tcp]

    CAfile = /conf/ssl/ca.crt

    cert   = /conf/ssl/tls.crt

    key    = /conf/ssl/tls.key

    sslVersion = TLSv1.2

    accept = 0.0.0.0:8443

    connect = 127.0.0.1:8080

    TIMEOUTbusy    = 300

    TIMEOUTclose   = 60

    TIMEOUTconnect = 10

    TIMEOUTidle    = 300

    socket = l:SO_LINGER=1:60

    retry = no

    EOF

    echo 'Starting Stunnel TLS termination for TCP Services...'

    stunnel /etc/stunnel/stunnel.conf

• Update your deployment, first ensure the port matches with the config (e.g. TCP/8080), and add the extra container as follows:

    extraContainers:
      - name: stunnel
        image: alpine:3.14.0
        command: ["sh", "/opt/start.sh"]
        volumeMounts:
          - name: stunnel-conf
            mountPath: /opt/
        ports:
          - containerPort: 8443

    extraVolumes:
      - name: stunnel-conf
        configMap:
          name: stunnel-conf

• Apply the changes and test the connection on port TLS+TCP/8443 https://your-service:8443

Sorry I cannot give a full working example right now but hopefully this gives you an idea.