Consul connect service mesh not working with consul 1.14.0

[suikast42]

Nomad version

v 1.4.2

Consul version

v 1.14.0

Issue

The simple counter dash app form the connect examples does not work anymore if I updgrade consul to the latest verson 1.14.0

After the update to consul 1.14.0 I can see in the log of the connect-proxy-count-dashboard that something goes wrong with the tls connection.

[2022-11-17 21:09:00.873][1][warning][config] [./source/common/config/grpc_stream.h:201] DeltaAggregatedResources gRPC config stream to local_agent closed since 202s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER

The used job file

job "countdash_app_mesh" {
  datacenters = ["nomadder1"]
  group "api" {
    network {
      mode = "bridge"
    }

    service {
      name = "count-api"
      port = "9001"
      connect {
        sidecar_service {}
      }
    }

    task "count-api" {
      driver = "docker"

      config {
        image = "hashicorpnomad/counter-api:v3"
        ports = ["http"]
      }

      resources {
        cpu    = 100
        memory = 128
      }
    }
  }

  group "dashboard" {
    network {
      mode = "bridge"

      port "http" {
        to = 9002
      }
    }

    service {
      name = "count-dashboard"
      port = "9002"
            tags = [
              "traefik.enable=true",
              "traefik.consulcatalog.connect=true",
              "traefik.http.routers.count-dashboard.tls=true",
              "traefik.http.routers.count-dashboard.rule=Host(`count.cloud.private`)"
            ]

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "count-api"
              local_bind_port  = 8080
            }
          }
        }
      }
    }

    task "dashboard" {
      driver = "docker"

      env {
        CONSUL_TLS_SERVER_NAME = "localhost"
        COUNTING_SERVICE_URL   = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
      }

      config {
        image = "hashicorpnomad/counter-dashboard:v3"
      }

      resources {
        cpu    = 100
        memory = 128
      }
    }
  }
}

[jrasell]

Hi @suikast42; you also raised #15266 which described problems with running Nomad Consul Connect integration with Consul 1.14.0, albeit you are seeing different errors. Could you explain a little more about your configuration and how it has changed in order to generate a different error in that other issue? I will close this issue out in order to keep all information in a single place. Thanks.

[suikast42]

Hi @suikast42; you also raised #15266 which described problems with running Nomad Consul Connect integration with Consul 1.14.0, albeit you are seeing different errors. Could you explain a little more about your configuration and how it has changed in order to generate a different error in that other issue? I will close this issue out in order to keep all information in a single place. Thanks.

This is still not working. I tested it with the latest beta relase. See #16186

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.