SSL wrong version number transporting non-ssl connection

[bradydean]

Title: SSL wrong version number transporting non-ssl connection

Description:
I have a consul connect mesh with traefik ingress for a tcp (non-ssl) service. Traffic makes it to the envoy instance besides the tcp service but I'm getting an SSL error. I'm assuming this is communication between traefik's sidecar and tcp service's sidecar.

The sidecars are deployed by Nomad: envoyproxy/envoy:v1.23.0

Logs:

[2022-09-17 19:25:24.973][15][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:198] [C1611] new tcp proxy session
[2022-09-17 19:25:24.973][15][debug][filter] [source/common/tcp_proxy/tcp_proxy.cc:370] [C1611] Creating connection to cluster local_app
[2022-09-17 19:25:24.973][15][debug][misc] [source/common/upstream/cluster_manager_impl.cc:1942] Allocating TCP conn pool
[2022-09-17 19:25:24.973][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:290] trying to create new connection
[2022-09-17 19:25:24.973][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0)
[2022-09-17 19:25:24.973][15][debug][connection] [source/common/network/connection_impl.cc:924] [C1612] connecting to 127.0.0.1:29542
[2022-09-17 19:25:24.973][15][debug][connection] [source/common/network/connection_impl.cc:943] [C1612] connection in progress
[2022-09-17 19:25:24.973][15][debug][conn_handler] [source/server/active_tcp_listener.cc:147] [C1611] new connection from 172.26.64.1:53150
[2022-09-17 19:25:24.974][15][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:228] [C1611] remote address:172.26.64.1:53150,TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2022-09-17 19:25:24.974][15][debug][connection] [source/common/network/connection_impl.cc:250] [C1611] closing socket: 0
[2022-09-17 19:25:24.974][15][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:228] [C1611] remote address:172.26.64.1:53150,TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2022-09-17 19:25:24.974][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:654] cancelling pending stream
[2022-09-17 19:25:24.974][15][debug][connection] [source/common/network/connection_impl.cc:139] [C1612] closing data_to_write=0 type=1
[2022-09-17 19:25:24.974][15][debug][connection] [source/common/network/connection_impl.cc:250] [C1612] closing socket: 1
[2022-09-17 19:25:24.974][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:483] [C1612] client disconnected, failure reason: 
[2022-09-17 19:25:24.974][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:453] invoking idle callbacks - is_draining_for_deletion_=false
[2022-09-17 19:25:24.974][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:453] invoking idle callbacks - is_draining_for_deletion_=false
[2022-09-17 19:25:24.974][15][debug][conn_handler] [source/server/active_stream_listener_base.cc:120] [C1611] adding to cleanup list

[bradydean]

I can replicate with this minimal job

job "echo" {

  datacenters = ["dc1"]

  group "echo" {
    network {
      mode = "bridge"
      port "tcp" {}
    }

    service {
      name = "echo"
      port = "tcp"
      tags = [
        "traefik.enable=true",
        "traefik.tcp.routers.echo.rule=HostSNI(`*`)",
        "traefik.tcp.routers.echo.entrypoints=tcp",
      ]
      connect {
        sidecar_service {}
      }
    }
    
    task "echo" {
      driver = "docker"
      env {
        TCP_PORT = "${NOMAD_PORT_tcp}"
      }
      config {
        image = "cjimti/go-echo:latest"
        ports = ["tcp"]
      }
    }
  }
}

minimal traefik job

job "traefik" {
  datacenters = ["dc1"]
  group "traefik" {
    network {
      mode = "bridge"
      port "tcp" {
        static = 5555
      }
    }

    service {
      name = "traefik"
      connect {
        sidecar_service {}
      }
    }

    task "traefik" {
      driver = "docker"

      template {
        destination = "local/traefik.yml"
        data = <<EOF
api:
  dashboard: false
entryPoints:
  tcp:
    address: ":5555"
providers:
  consulCatalog:
    connectAware: true
    connectByDefault: true
    exposedByDefault: false
    endpoint:
      address: 192.168.1.103:8500
EOF
      }

      config {
        image = "traefik:v2.8.5"
        ports = ["tcp"]
        volumes = ["local/traefik.yml:/etc/traefik/traefik.yml"]
      }
    }
  }
}

Connection can't be used

[brady@server ~]$ telnet 192.168.1.103 5555
Trying 192.168.1.103...
Connected to 192.168.1.103.
Escape character is '^]'.
Hello
FConnection closed by foreign host.

EDIT: Also replicated with istio/tcp-echo-server:1.2, client even receives the same non printable + F bytes.

These are the bytes, maybe someone may recognize it.

15 03 01 00 02 02 46

[13567436138]

can you provider your envoy yaml

[bradydean]

Nomad creates this config and executes envoy like this

envoy -c /secrets/envoy_bootstrap.json -l debug --concurrency 1 --disable-hot-restart

In the case of connect-sidecar-echo I found this envoy_bootstrap.json

{
  "admin": {
    "access_log_path": "/dev/null",
    "address": {
      "socket_address": {
        "address": "127.0.0.2",
        "port_value": 19001
      }
    }
  },
  "node": {
    "cluster": "echo",
    "id": "_nomad-task-d828434a-000a-b3af-006e-6390476953ba-group-echo-echo-tcp-sidecar-proxy",
    "metadata": {
      "namespace": "default",
      "partition": "default"
    }
  },
  "layered_runtime": {
    "layers": [
      {
        "name": "base",
        "static_layer": {
          "re2.max_program_size.error_level": 1048576
        }
      }
    ]
  },
  "static_resources": {
    "clusters": [
      {
        "name": "local_agent",
        "ignore_health_on_host_removal": false,
        "connect_timeout": "1s",
        "type": "STATIC",
        "http2_protocol_options": {},
        "loadAssignment": {
          "clusterName": "local_agent",
          "endpoints": [
            {
              "lbEndpoints": [
                {
                  "endpoint": {
                    "address": {
                      "pipe": {
                        "path": "alloc/tmp/consul_grpc.sock"
                      }
                    }
                  }
                }
              ]
            }
          ]
        }
      }
    ]
  },
  "stats_config": {
    "stats_tags": [
      {
        "tag_name": "nomad.alloc_id",
        "fixed_value": "d828434a-000a-b3af-006e-6390476953ba"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.custom_hash"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.service_subset"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.service"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.namespace"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:([^.]+)\\.)?[^.]+\\.internal[^.]*\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.partition"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?([^.]+)\\.internal[^.]*\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.datacenter"
      },
      {
        "regex": "^cluster\\.([^.]+\\.(?:[^.]+\\.)?([^.]+)\\.external\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.peer"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.routing_type"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)",
        "tag_name": "consul.destination.trust_domain"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.target"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)",
        "tag_name": "consul.destination.full_target"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream(?:_peered)?\\.(([^.]+)(?:\\.[^.]+)?(?:\\.[^.]+)?\\.[^.]+\\.)",
        "tag_name": "consul.upstream.service"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?(?:\\.[^.]+)?\\.([^.]+)\\.)",
        "tag_name": "consul.upstream.datacenter"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream_peered\\.([^.]+(?:\\.[^.]+)?\\.([^.]+)\\.)",
        "tag_name": "consul.upstream.peer"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream(?:_peered)?\\.([^.]+(?:\\.([^.]+))?(?:\\.[^.]+)?\\.[^.]+\\.)",
        "tag_name": "consul.upstream.namespace"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?(?:\\.([^.]+))?\\.[^.]+\\.)",
        "tag_name": "consul.upstream.partition"
      },
      {
        "regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.custom_hash"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.service_subset"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.service"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.namespace"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?([^.]+)\\.internal[^.]*\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.datacenter"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.routing_type"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)",
        "tag_name": "consul.trust_domain"
      },
      {
        "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.target"
      },
      {
        "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)",
        "tag_name": "consul.full_target"
      },
      {
        "tag_name": "local_cluster",
        "fixed_value": "echo"
      },
      {
        "tag_name": "consul.source.service",
        "fixed_value": "echo"
      },
      {
        "tag_name": "consul.source.namespace",
        "fixed_value": "default"
      },
      {
        "tag_name": "consul.source.partition",
        "fixed_value": "default"
      },
      {
        "tag_name": "consul.source.datacenter",
        "fixed_value": "dc1"
      }
    ],
    "use_all_default_tags": true
  },
  "dynamic_resources": {
    "lds_config": {
      "ads": {},
      "resource_api_version": "V3"
    },
    "cds_config": {
      "ads": {},
      "resource_api_version": "V3"
    },
    "ads_config": {
      "api_type": "DELTA_GRPC",
      "transport_api_version": "V3",
      "grpc_services": {
        "initial_metadata": [
          {
            "key": "x-consul-token",
            "value": "xxxxxxxxxxxxxxxxxxxxxxx"
          }
        ],
        "envoy_grpc": {
          "cluster_name": "local_agent"
        }
      }
    }
  }
}

[bradydean]

possibly related istio/istio#35870

[bradydean]

Turns out this is a limitation of traefik

traefik/traefik#8601

EDIT: Maybe this isn't the same issue, but I believe the issues lies in traefik now.

[santudiya]

May i know what is the fix for this ? even we are facing intermittent connections issues when we use with other port number but if we use port 8080 no issues its normal only
its in Tanzu environment while communicating to one container to another container.