[envoy] new integration

envoy/CHANGELOG.md

@@ -0,0 +1,10 @@
+# CHANGELOG - Envoy
+
+1.0.0 / Unreleased
+==================
+
+### Changes
+
+* [FEATURE] add Envoy integration. See #1156
+
+<!--- The following link definition list is generated by PimpMyChangelog --->

envoy/MANIFEST.in

@@ -0,0 +1,6 @@
+include README.md
+include requirements.in
+include requirements.txt
+include requirements-dev.txt
+graft datadog_checks
+graft tests

envoy/README.md

@@ -0,0 +1,141 @@
+# Agent Check: Envoy
+## Overview
+
+This check collects distributed system observability metrics from [Envoy](https://www.envoyproxy.io).
+
+## Setup
+### Installation
+
+The Envoy check is packaged with the Agent, so simply [install the Agent](https://app.datadoghq.com/account/settings#agent) on your server.
+
+If you need the newest version of the Envoy check, install the `dd-check-envoy` package; this package's check overrides the one packaged with the Agent. See the [integrations-core repository README.md for more details](https://docs.datadoghq.com/agent/faq/install-core-extra/).
+
+### Configuration
+
+Create a file `envoy.yaml` in the Datadog Agent's `conf.d` directory. See the [sample envoy.yaml](https://github.com/DataDog/integrations-core/blob/master/envoy/conf.yaml.example) for all available configuration options:
+
+There are 2 ways to setup the `/stats` endpoint:
+
+#### Unsecured stats endpoint
+
+Be sure the Datadog Agent can access Envoy's [admin endpoint](https://www.envoyproxy.io/docs/envoy/latest/operations/admin). Here's an example Envoy admin configuration: 
+
+```yaml
+admin:
+  access_log_path: "/dev/null"
+  address:
+    socket_address:
+      address: 0.0.0.0
+      port_value: 8001
+```
+
+#### Secured stats endpoint
+
+Create a listener/vhost that routes to the admin endpoint (Envoy connecting to itself), but only has a route for `/stats`; all other routes get a static/error response. Additionally, this allows nice integration with L3 filters for auth, for example.
+
+Here's an example config (from [this gist](https://gist.github.com/ofek/6051508cd0dfa98fc6c13153b647c6f8)):
+
+```json
+{
+  "listeners": [
+    {
+      "address": "tcp://0.0.0.0:80",
+      "filters": [
+        {
+          "type": "read",
+          "name": "http_connection_manager",
+          "config": {
+            "codec_type": "auto",
+            "stat_prefix": "ingress_http",
+            "route_config": {
+              "virtual_hosts": [
+                {
+                  "name": "backend",
+                  "domains": ["*"],
+                  "routes": [
+                    {
+                      "timeout_ms": 0,
+                      "prefix": "/stats",
+                      "cluster": "service_stats"
+                    }
+                  ]
+                }
+              ]
+            },
+            "filters": [
+              {
+                "type": "decoder",
+                "name": "router",
+                "config": {}
+              }
+            ]
+          }
+        }
+      ]
+    }
+  ],
+  "admin": {
+    "access_log_path": "/dev/null",
+    "address": "tcp://0.0.0.0:8001"
+  },
+  "cluster_manager": {
+    "clusters": [
+      {
+        "name": "service_stats",
+        "connect_timeout_ms": 250,
+        "type": "logical_dns",
+        "lb_type": "round_robin",
+        "hosts": [
+          {
+            "url": "tcp://127.0.0.1:8001"
+          }
+        ]
+      }
+    ]
+  }
+}
+```
+
+### Validation
+
+[Run the Agent's `status` subcommand](https://docs.datadoghq.com/agent/faq/agent-commands/#agent-status-and-information) and look for `envoy` under the Checks section:
+
+```
+  Checks
+  ======
+    [...]
+
+    envoy
+    -----
+      - instance #0 [OK]
+      - Collected 244 metrics, 0 events & 1 service check
+
+    [...]
+```
+
+## Compatibility
+
+The Envoy check is compatible with all platforms.
+
+## Data Collected
+### Metrics
+
+See [metadata.csv](https://github.com/DataDog/integrations-core/blob/master/envoy/metadata.csv) for a list of metrics provided by this check.
+See [metrics.py](https://github.com/DataDog/integrations-core/blob/master/envoy/datadog_checks/envoy/metrics.py) for a list of tags sent by each metric.
+
+### Events
+
+The Envoy check does not include any events at this time.
+
+### Service Checks
+
+`envoy.can_connect`:
+
+Returns CRITICAL if the Agent cannot connect to Envoy to collect metrics, otherwise OK.
+
+## Troubleshooting
+
+Need help? Contact [Datadog Support](http://docs.datadoghq.com/help/).
+
+## Further Reading
+Learn more about infrastructure monitoring and all our integrations on [our blog](https://www.datadoghq.com/blog/)

envoy/conf.yaml.example

@@ -0,0 +1,38 @@
+# This file is overwritten upon Agent upgrade.
+# To make modifications to the check configuration, please copy this file
+# to `envoy.yaml` and make your changes on that file.
+
+init_config:
+
+instances:
+  # For every instance, you need a `stats_url` and can optionally
+  # supply a list of tags. The admin endpoint must be accessible.
+  # https://www.envoyproxy.io/docs/envoy/latest/operations/admin
+
+  - stats_url: http://localhost:80/stats
+
+  #   tags:
+  #     - instance:foo
+
+  #   <<<Note>>> The Envoy admin endpoint does not support auth until:
+  #   https://github.com/envoyproxy/envoy/issues/2763
+  #   For an alternative, see:
+  #   https://gist.github.com/ofek/6051508cd0dfa98fc6c13153b647c6f8
+  #
+  #   If the stats page is behind basic auth:
+  #   username: USERNAME
+  #   password: PASSWORD
+
+  #   The (optional) verify_ssl parameter will instruct the check to validate SSL
+  #   certificates when connecting to Envoy. Defaulting to true, set to false if
+  #   you want to disable SSL certificate validation.
+  #
+  #   verify_ssl: true
+
+  #   The (optional) skip_proxy parameter will bypass any proxy
+  #   settings enabled and attempt to reach Envoy directly.
+  #
+  #   skip_proxy: false
+
+  #   If you need to specify a custom timeout in seconds (default is 20):
+  #   timeout: 20

envoy/datadog_checks/__init__.py

@@ -0,0 +1,4 @@
+# (C) Datadog, Inc. 2018
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)

envoy/datadog_checks/envoy/__about__.py

@@ -0,0 +1,5 @@
+# (C) Datadog, Inc. 2018
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+
+__version__ = '1.0.0'

envoy/datadog_checks/envoy/__init__.py

@@ -0,0 +1,10 @@
+# (C) Datadog, Inc. 2018
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+from .__about__ import __version__
+from .envoy import Envoy
+
+__all__ = [
+    '__version__',
+    'Envoy'
+]

envoy/datadog_checks/envoy/envoy.py

@@ -0,0 +1,80 @@
+# (C) Datadog, Inc. 2018
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+import requests
+
+from datadog_checks.checks import AgentCheck
+
+from .errors import UnknownMetric
+from .parser import parse_metric
+
+
+class Envoy(AgentCheck):
+    SERVICE_CHECK_NAME = 'envoy.can_connect'
+
+    def __init__(self, name, init_config, agentConfig, instances=None):
+        super(Envoy, self).__init__(name, init_config, agentConfig, instances)
+        self.unknown_metrics = set()
+
+    def check(self, instance):
+        custom_tags = instance.get('tags', [])
+
+        try:
+            stats_url = instance['stats_url']
+        except KeyError:
+            msg = 'Envoy configuration setting `stats_url` is required'
+            self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, message=msg, tags=custom_tags)
+            self.log.error(msg)
+            return
+
+        username = instance.get('username', None)
+        password = instance.get('password', None)
+        auth = (username, password) if username and password else None
+        verify_ssl = instance.get('verify_ssl', True)
+        proxies = self.get_instance_proxy(instance, stats_url)
+        timeout = int(instance.get('timeout', 20))
+
+        try:
+            request = requests.get(
+                stats_url, auth=auth, verify=verify_ssl, proxies=proxies, timeout=timeout
+            )
+        except requests.exceptions.Timeout:
+            msg = 'Envoy endpoint `{}` timed out after {} seconds'.format(stats_url, timeout)
+            self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, message=msg, tags=custom_tags)
+            self.log.exception(msg)
+            return
+        except (requests.exceptions.RequestException, requests.exceptions.ConnectionError):
+            msg = 'Error accessing Envoy endpoint `{}`'.format(stats_url)
+            self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, message=msg, tags=custom_tags)
+            self.log.exception(msg)
+            return
+
+        if request.status_code != 200:
+            msg = 'Envoy endpoint `{}` responded with HTTP status code {}'.format(stats_url, request.status_code)
+            self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.CRITICAL, message=msg, tags=custom_tags)
+            self.log.warning(msg)
+            return
+
+        # Avoid repeated global lookups.
+        get_method = getattr
+
+        for line in request.content.decode().splitlines():
+            try:
+                envoy_metric, value = line.split(': ')
+            except ValueError:
+                continue
+
+            value = int(value)
+
+            try:
+                metric, tags, method = parse_metric(envoy_metric)
+            except UnknownMetric:
+                if envoy_metric not in self.unknown_metrics:
+                    self.log.debug('Unknown metric `{}`'.format(envoy_metric))
+                    self.unknown_metrics.add(envoy_metric)
+                continue
+
+            tags.extend(custom_tags)
+            get_method(self, method)(metric, value, tags=tags)
+
+        self.service_check(self.SERVICE_CHECK_NAME, AgentCheck.OK, tags=custom_tags)

envoy/datadog_checks/envoy/errors.py

@@ -0,0 +1,7 @@
+# (C) Datadog, Inc. 2018
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+
+
+class UnknownMetric(Exception):
+    pass