Problem with Canary deployment

[anthonymq]

Hi,
I'm working on the production deployment of my app and blue green deployment seems to be perfect to expect a 0 downtime.
The thing is that my app is a group of two tasks :

• An Nginx container for my frontend which acts as a reverse proxy for the api
• A Java container for my api

I've added canary_tags to my tasks to add a new traefik route and I can access the two different frontend versions in parallel.
The problem is the connection to the backend. My new frontend make calls to the old backend and vice versa.

How can i isolate the couples frontend/backend ? Or can I specify some canary env vars (to configure the backend url)?

If I only add a task for the app, everything would have been fine.

FYI: Nomad 0.10.5, Consul 1.7.2 and Traefik 2.2

[mocofound]

Hello! Thanks for the question!

How does your front-end currently 'find' the backend? Is it a DNS query to Consul? Are you querying Consul based on tags? You could potentially modify your frontend to query Consul based on 'v1' or 'v2' tags for the backend service.

For example, v1.backend.service.consul
See here for more info: https://www.consul.io/docs/agent/dns.html#standard-lookup

Additionally, if you are using Consul Connect (service mesh), you could do this with Connect's Layer 7 Management capabilities.

Take a look at this Consul Connect demo environment that shows a read-only UI with example Service Routers, Splitters and Resolvers built out in various environments.

[mocofound]

On the Nomad side, you can pass env vars that specify the version of the backend you should target using the env stanza

job "docs" {
  group "example" {
    task "frontend" {
      env {
        backend_to_target = "v2.backend.service.consul"
      }
    }
  }
}

You can get a little bit fancier by populating these dynamically using data retrieved from Consul with dynamic environment variables which utilize Consul Template.

[anthonymq]

Thanks for the quick reply, I'll have a look tomorrow.

[anthonymq]

I had been using a simple solution until now.

task "front" {
      driver = "docker"
      env {
        BACKEND_HOST = "${NOMAD_IP_back_tomcat}"
        BACKEND_PORT = "${NOMAD_PORT_back_tomcat}"
      }

[anthonymq]

The problem with env vars is that i can't change the value only for the canary frontend.

[mocofound]

You may find this NGINX integration example using the Nomad template stanza from the HashiCorp Learn site helpful:

The example job file from the Learn site explicitly writes out the NGINX load balancer config based on IP addresses and Ports for the demo-webapp service retrieved from Consul:

job "nginx" {
  datacenters = ["dc1"]

  group "nginx" {
    count = 1

    task "nginx" {
      driver = "docker"

      config {
        image = "nginx"

        port_map {
          http = 80
        }

        volumes = [
          "local:/etc/nginx/conf.d",
        ]
      }

      template {
        data = <<EOF
upstream backend {
{{ range service "demo-webapp" }}
  server {{ .Address }}:{{ .Port }};
{{ else }}server 127.0.0.1:65535; # force a 502
{{ end }}
}

server {
   listen 80;

   location / {
      proxy_pass http://backend;
   }
}
EOF

        destination   = "local/load-balancer.conf"
        change_mode   = "signal"
        change_signal = "SIGHUP"
      }

      resources {
        network {
          mbits = 10

          port "http" {
            static = 8080
          }
        }
      }

      service {
        name = "nginx"
        port = "http"
      }
    }
  }
}

To expand beyond that learn guide, here is some example syntax you can modify in order to have the template stanza pull IP address and port info for different versions of the backend service, as specified by tags in Consul:

{{range $tag, $service := service "web" | byTag}}
# "{{$tag}}" api providers.
<Proxy balancer://{{$tag}}>
{{range $service}}  BalancerMember http://{{.Address}}:{{.Port}}
{{end}} ProxySet lbmethod=bybusyness
</Proxy>
Redirect permanent /api/{{$tag}} /api/{{$tag}}/
ProxyPass /api/{{$tag}}/ balancer://{{$tag}}/
ProxyPassReverse /api/{{$tag}}/ balancer://{{$tag}}/
{{end}}

It is also possible to specify the tag you want more explicitly, using this syntax from Consul-template:

{{ service "<TAG>.<NAME>@<DATACENTER>~<NEAR>|<FILTER>" }}

Example:

{{ service "v2.backend" }}

For completeness, I also want to mention that you also don't have to write directly to the NGINX config. You may want to instead use the Nomad template stanza to populate environment variables then consume them however you wish (as opposed to writing directly to the NGINX config):

template {
  data = <<EOH
# Lines starting with a # are ignored

# Reads from Consul key/value store
LOG_LEVEL="{{key "service/geo-api/log-verbosity"}}"
EOH

  destination = "secrets/file.env"
  env         = true
}

[anthonymq]

Thanks, I used consuldns tags to get it working !

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.