Skip to content
This repository has been archived by the owner on Dec 4, 2024. It is now read-only.

Home

Jump to bottom
Jason Kölker edited this page Oct 4, 2019 · 28 revisions

Welcome to the marathon-lb wiki!

Examples

Custom HTTP headers in health check

This example adds the Host header to the health check executed by HAProxy:

{
  "id":"app",
  "labels": {
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_BACKEND_HTTP_HEALTHCHECK_OPTIONS": "  option  httpchk GET {healthCheckPath} HTTP/1.1\\r\\nHost:\\ www\n  timeout check {healthCheckTimeoutSeconds}s\n"
  }
}

Setting timeout for long-lived socket connections

If you're trying to run a TCP service which uses long-lived sockets through HAProxy, such as a MySQL instance, you'll need to set longer timeouts for the backend. Try the following:

{
  "id":"app",
  "labels":{
    "HAPROXY_GROUP":"external",
    "HAPROXY_0_BACKEND_HEAD":"backend {backend}\n  balance {balance}\n  mode {mode}\n  timeout tunnel 30m\n"
  }
}

The example above will set the client and server timeout to 30 minutes for the specified backend.

SSL Termination at an Elastic Load Balancer

Sometimes you want to allow an ELB to terminate SSL for you, but you still want marathon-lb to redirect non-HTTPS requests. ELBs use HTTP headers to communicate that the request came in via a secure channel and has been decrypted. Specifically, if the X-Forwarded-Proto header is set to https, then the request was decrypted by the ELB.

Unless you tell HAProxy to look for the X-Forwarded-Proto header, the request will appear as if it's unencrypted and will get redirected using standard the rules.

"labels": {
  "HAPROXY_0_BACKEND_HTTP_OPTIONS": "  acl is_proxy_https hdr(X-Forwarded-Proto) https\n  redirect scheme https unless { ssl_fc } or is_proxy_https\n"
}

This configuration instructs marathon-lb to generate a backend rule that looks for the X-Forwarded-Proto header or a regular TLS connection and redirect if neither are specified.

Enabling TLS v1.0

TLS v1.0 is deprecated, and no longer supported by the default MLB config. If you require TLS v1.0 support, you must supply a custom template for HAPROXY_HEAD. To do this, add a template URI to your MLB app definition like this:

  {
    "id":"/marathon-lb",
    "uris":["https://downloads.mesosphere.com/marathon/marathon-lb/templates-with-tls-10.tgz"]
  }

In order for this to work, you must be using a version of Marathon-LB based on debian:stretch or below. Marathon-LB based on debian:buster does not OS-level support for TLSv1.0 or TLSv1.1.

Specifically, these versions of Marathon-LB do NOT support TLSv1.0/TLSv1.1:

  • 1.8.1
  • 1.11.1

Disable Binding to Service Ports

If you do not want MLB to listen on service ports, you may disable the frontend definitions:

  {
    "labels": {
      "HAPROXY_GROUP": "external",
      "HAPROXY_0_FRONTEND_HEAD": "",
      "HAPROXY_0_FRONTEND_BACKEND_GLUE": ""
    }
  }

Wildcard vhosts resolution

If you want all subdomains for a given domain to resolve to a particular backend (for HTTP and HTTPS), use the following labels. Note that there is a period . required before the {hostname} in the HAPROXY_0_HTTPS_FRONTEND_ACL label. Note: be sure to disable the vhost map option by removing the --haproxy-map argument, if present.

{
  "labels": {
    "HAPROXY_0_BACKEND_WEIGHT": "-1",
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_HTTP_FRONTEND_ACL": "  acl host_{cleanedUpHostname} hdr_end(host) -i {hostname}\n  use_backend {backend} if host_{cleanedUpHostname}\n",
    "HAPROXY_0_HTTPS_FRONTEND_ACL": "  use_backend {backend} if {{ ssl_fc_sni -m end .{hostname} }}\n",
    "HAPROXY_0_VHOST": "example.com"
  }
}

Enabling HAProxy logging

HAProxy uses socket based logging, and it's configured by default to log to /dev/log. To use HAProxy's logging, you must mount /dev/log into the container and enable logging for any backends or frontends you want to log. Afterward, you can examine the logs with journalctl. First, mount the volume into your /marathon-lb app:

{
  "id": "/marathon-lb",
  "container": {
    "type": "DOCKER",
    "volumes": [
      {
        "containerPath": "/dev/log",
        "hostPath": "/dev/log",
        "mode": "RW"
      }
    ],
    "docker": {
      "image": "mesosphere/marathon-lb:latest",
      "network": "HOST",
      "privileged": true,
      "parameters": [],
      "forcePullImage": true
    }
  }
}

Now, we'll set option httplog on one backend to enable logging. In this example, I'm using my personal website:

{
  "id": "/my-crappy-website",
  "cmd": null,
  "cpus": 0.5,
  "mem": 64,
  "disk": 0,
  "instances": 2,
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": "brndnmtthws/my-crappy-website",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 0,
          "servicePort": 10012,
          "protocol": "tcp",
          "labels": {}
        }
      ],
      "privileged": false,
      "parameters": [],
      "forcePullImage": true
    }
  },
  "healthChecks": [
    {
      "path": "/",
      "protocol": "HTTP",
      "portIndex": 0,
      "gracePeriodSeconds": 10,
      "intervalSeconds": 15,
      "timeoutSeconds": 2,
      "maxConsecutiveFailures": 3,
      "ignoreHttp1xx": false
    }
  ],
  "labels": {
    "HAPROXY_0_USE_HSTS": "true",
    "HAPROXY_0_REDIRECT_TO_HTTPS": "true",
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_BACKEND_HTTP_OPTIONS": "  option httplog\n  option forwardfor\n  http-request set-header X-Forwarded-Port %[dst_port]\n  http-request add-header X-Forwarded-Proto https if { ssl_fc }\n",
    "HAPROXY_0_VHOST": "diddyinc.com,www.diddyinc.com"
  },
  "portDefinitions": [
    {
      "port": 10012,
      "protocol": "tcp",
      "labels": {}
    }
  ]
}

Note: Enabling the httplog option will only affect the backend for the service port. To enable it for ports 80 and 443, you must modify the global HAProxy template.

Now, if you SSH into any public slave, you can view the logs using journalctl:

# journalctl -f -l SYSLOG_IDENTIFIER=haproxy

Custom HAProxy error response

You may specify a custom HAProxy error response by overriding the default errorfile directive. For example, you could return a redirect to a different backend in the event that no backends are available. To do this, add a template URI to your MLB app definition like this:

  {
    "id":"/marathon-lb",
    "uris":["https://downloads.mesosphere.com/marathon/marathon-lb/templates-custom-500-response.tar.gz"]
  }

The example above simply returns a custom 503 page, but you could instead return a redirect (by updating the templates/500.http file within the archive), like this:

HTTP/1.1 302 Found
Location: http://my-redirect-handler.computers.com/

Templates

(Pulled from https://docs.d2iq.com/mesosphere/dcos/services/marathon-lb/1.14/mlb-configuration/)

Marathon-LB has a templating feature for specifying custom HAProxy configuration parameters. Templates can be set either globally (for all apps), or on a per-app basis using labels. Let’s demonstrate an example of how to specify our own global template. Here’s the template we’ll use:

Global Template

To specify a global template:

  1. On your local machine, create a file called HAPROXY_HEAD in a directory called templates with the contents below:
global
  log /dev/log local0
  log /dev/log local1 notice
  spread-checks 5
  max-spread-checks 15000
  maxconn 4096
  tune.ssl.default-dh-param 2048
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:!aNULL:!MD5:!DSS
  ssl-default-bind-options no-sslv3 no-tlsv10 no-tls-tickets
  ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:!aNULL:!MD5:!DSS
  ssl-default-server-options no-sslv3 no-tlsv10 no-tls-tickets
  stats socket /var/run/haproxy/socket expose-fd listeners
  server-state-file global
  server-state-base /var/state/haproxy/
  lua-load /marathon-lb/getpids.lua
  lua-load /marathon-lb/getconfig.lua
  lua-load /marathon-lb/getmaps.lua
  lua-load /marathon-lb/signalmlb.lua
defaults
  load-server-state-from-file global
  log               global
  retries                   3
  backlog               10000
  maxconn                3000
  timeout connect          5s
  timeout client          20s
  timeout server          40s
  timeout tunnel        3600s
  timeout http-keep-alive  1s
  timeout http-request    15s
  timeout queue           30s
  timeout tarpit          60s
  option            dontlognull
  option            http-server-close
  option            redispatch
listen stats
  bind 0.0.0.0:9090
  balance
  mode http
  stats enable
  monitor-uri /_haproxy_health_check
  acl getpid path /_haproxy_getpids
  http-request use-service lua.getpids if getpid
  acl getvhostmap path /_haproxy_getvhostmap
  http-request use-service lua.getvhostmap if getvhostmap
  acl getappmap path /_haproxy_getappmap
  http-request use-service lua.getappmap if getappmap
  acl getconfig path /_haproxy_getconfig
  http-request use-service lua.getconfig if getconfig

  acl signalmlbhup path /_mlb_signal/hup
  http-request use-service lua.signalmlbhup if signalmlbhup
  acl signalmlbusr1 path /_mlb_signal/usr1
  http-request use-service lua.signalmlbusr1 if signalmlbusr1

In the code above, the following items have changed from the default: maxconn, timeout client, and timeout server.

  1. Tar or zip the file. Here's an example of how to do this:

    #!/bin/bash

mkdir -p templates
cat >templates/HAPROXY_HEAD <<EOL
global
  daemon
  log /dev/log local0
  log /dev/log local1 notice
  maxconn 4096
  tune.ssl.default-dh-param 2048
defaults
  log               global
  retries           3
  maxconn           3000
  timeout connect   5s
  timeout client    30s
  timeout server    30s
  option            redispatch
listen stats
  bind 0.0.0.0:9090
  balance
  mode http
  stats enable
  monitor-uri /_haproxy_health_check
EOL

tar czf templates.tgz templates/

Take the file you created (templates.tgz if you use the script), and make it available from an HTTP server. If you’d like to use the sample one, use this URI: https://downloads.mesosphere.com/marathon/marathon-lb/templates.tgz

  1. Augment the Marathon-LB config by saving the following JSON in a file called options.json:

    {
  "marathon-lb": {
    "template-url":"https://downloads.mesosphere.com/marathon/marathon-lb/templates.tgz"
  }
}

  2. Launch the new Marathon-LB:

    dcos package install --options=options.json marathon-lb

    Your customized Marathon-LB HAProxy instance will now be running with the new template. [A full list of the templates available can be found here][2].

Clone this wiki locally