Add MapR integration (#4380)

* Initial POC * Implementation + doc * Apply suggestions from code reviews Co-Authored-By: ruthnaebeck <[email protected]> * Doc and test updates * Add MapR to azure pipelines * Update mapr/README.md Co-Authored-By: Pierre Guceski <[email protected]> * Update mapr/README.md Co-Authored-By: Pierre Guceski <[email protected]> * Update mapr/README.md Co-Authored-By: Alexandre Yang <[email protected]> * Update mapr/README.md Co-Authored-By: Pierre Guceski <[email protected]> * Update mapr/README.md Co-Authored-By: Alexandre Yang <[email protected]> * Update mapr/assets/service_checks.json Co-Authored-By: Alexandre Yang <[email protected]> * Update mapr/datadog_checks/mapr/common.py Co-Authored-By: Alexandre Yang <[email protected]> * Update mapr/datadog_checks/mapr/mapr.py Co-Authored-By: Alexandre Yang <[email protected]> * Address reviews * Update mapr/datadog_checks/mapr/mapr.py Co-Authored-By: cswatt <[email protected]> * Update mapr/README.md Co-Authored-By: cswatt <[email protected]> * Update mapr/datadog_checks/mapr/mapr.py Co-Authored-By: cswatt <[email protected]> * Fix style * mapr.drill.queries_completed as monotonic count
DataDog · Oct 11, 2019 · 605f9fc · 605f9fc
1 parent 8b05e80
commit 605f9fc
Show file tree

Hide file tree

Showing 27 changed files with 2,131 additions and 0 deletions.
diff --git a/.azure-pipelines/all.yml b/.azure-pipelines/all.yml
@@ -198,6 +198,9 @@ jobs:
     - checkName: linux_proc_extras
       displayName: Linux proc extras
       os: linux
+    - checkName: mapr
+      displayName: MapR
+      os: linux
     - checkName: mapreduce
       displayName: MapReduce
       os: linux

diff --git a/mapr/CHANGELOG.md b/mapr/CHANGELOG.md
@@ -0,0 +1 @@
+# CHANGELOG - mapr
diff --git a/mapr/MANIFEST.in b/mapr/MANIFEST.in
@@ -0,0 +1,10 @@
+graft datadog_checks
+graft tests
+
+include MANIFEST.in
+include README.md
+include requirements.in
+include requirements-dev.txt
+include manifest.json
+
+global-exclude *.py[cod] __pycache__
diff --git a/mapr/README.md b/mapr/README.md
@@ -0,0 +1,98 @@
+# Agent Check: MapR
+
+## Overview
+
+This check monitors [MapR][1] 6.1+ through the Datadog Agent.
+
+## Setup
+
+Follow the instructions below to install and configure this check for an Agent running on a host.
+
+### Installation
+
+The MapR check is included in the [Datadog Agent][2] package but requires additional setup operations.
+
+1. Add `/opt/mapr/lib/` to your `ld.so.conf` file (usually in `/etc`). This will help the *mapr-streams-library* used by the agent find the required shared libraries.
+2. Create a password for the `dd-agent` user, then add this user to every node of the cluster with the same `UID`/`GID` so it is recognized by MapR. See [Managing users and groups][10] for additional details.
+3. Install the Agent on every host you want to monitor.
+4. Generate a [long-lived ticket][8] for the `dd-agent` user.
+5. Make sure the ticket is readable by the `dd-agent` user.
+6. Configure the integration (see below)
+
+
+### Configuration
+#### Metric collection
+
+1. Edit the `mapr.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to collect your MapR performance data. See the [sample mapr.d/conf.yaml][3] for all available configuration options.
+2. Set the `ticket_location` parameter to a path to the long-lived ticket you have created.
+3. [Restart the Agent][4].
+
+#### Log collection
+
+MapR uses fluentD for logs. Use the [fluent datadog plugin][11] to collect MapR logs.
+The following command downloads and installs the plugin into the right directory.
+
+`curl https://raw.githubusercontent.com/DataDog/fluent-plugin-datadog/master/lib/fluent/plugin/out_datadog.rb -o /opt/mapr/fluentd/fluentd-<VERSION>/lib/fluentd-<VERSION>-linux-x86_64/lib/app/lib/fluent/plugin/out_datadog.rb`
+
+Then update the `/opt/mapr/fluentd/fluentd-<VERSION>/etc/fluentd/fluentd.conf` with the following section.
+
+```
+<match *>
+  @type copy
+  <store> # This section is here by default and sends the logs to ElasticCache for Kibana.
+    @include /opt/mapr/fluentd/fluentd-<VERSION>/etc/fluentd/es_config.conf
+    include_tag_key true
+    tag_key service_name
+  </store>
+  <store> # This new section also forwards the logs to Datadog
+    @type datadog
+    @id dd_agent
+    include_tag_key true
+    dd_source mapr
+    dd_tags "flo:test"
+    service <YOUR_SERVICE_NAME>
+    api_key <YOUR_API_KEY>
+  </store>
+```
+
+Refer to [fluent_datadog_plugin][11] documentation for more details about the options you can use.
+
+
+### Validation
+
+[Run the Agent's status subcommand][5] and look for `mapr` under the Checks section.
+
+## Data Collected
+
+### Metrics
+
+See [metadata.csv][13] for a list of default metrics provided by this integration.
+
+### Service Checks
+
+- `mapr.can_connect`:
+Returns `CRITICAL` if the Agent fails to connect and subscribe to the stream topic, `OK` otherwise.
+
+### Events
+
+The MapR check does not include any events.
+
+## Troubleshooting
+
+Need help? Contact [Datadog support][6].
+
+[1]: https://mapr.com
+[2]: https://app.datadoghq.com/account/settings#agent
+[3]: https://github.com/DataDog/integrations-core/blob/master/mapr/datadog_checks/mapr/data/conf.yaml.example
+[4]: https://docs.datadoghq.com/agent/guide/agent-commands/?tab=agentv6#start-stop-and-restart-the-agent
+[5]: https://docs.datadoghq.com/agent/guide/agent-commands/?tab=agentv6#agent-status-and-information
+[6]: https://docs.datadoghq.com/help
+[7]: https://mapr.com/docs/61/MapR_Streams/MapRStreamsPythonExample.html
+[8]: https://mapr.com/docs/61/SecurityGuide/GeneratingServiceTicket.html
+[9]: https://mapr.com/docs/60/MapR_Streams/MapRStreamCAPISetup.html
+[10]: https://mapr.com/docs/61/AdministratorGuide/c-managing-users-and-groups.html
+[11]: https://www.rubydoc.info/gems/fluent-plugin-datadog
+[12]: https://mapr.com/docs/61/AdvancedInstallation/SettingUptheClient-install-mapr-client.html
+[13]: https://github.com/DataDog/integrations-core/blob/master/mapr/metadata.csv
+[14]: http://upstart.ubuntu.com/cookbook/#environment-variables
+[15]: https://www.freedesktop.org/software/systemd/man/systemd.service.html#Command%20lines
diff --git a/mapr/assets/logos/avatars-bot.png b/mapr/assets/logos/avatars-bot.png
diff --git a/mapr/assets/logos/saas_logos-bot.png b/mapr/assets/logos/saas_logos-bot.png
diff --git a/mapr/assets/logos/saas_logos-small.png b/mapr/assets/logos/saas_logos-small.png
diff --git a/mapr/assets/service_checks.json b/mapr/assets/service_checks.json
@@ -0,0 +1,11 @@
+[
+    {
+        "agent_version": "6.15.0",
+        "integration":"mapr",
+        "check": "mapr.can_connect",
+        "statuses": ["ok", "critical"],
+        "groups": ["topic"],
+        "name": "Can connect and subscribe to mapr topic",
+        "description": "Returns `CRITICAL` if the Agent fails to connect and subscribe to the stream topic, `OK` otherwise."
+    }
+]
diff --git a/mapr/datadog_checks/__init__.py b/mapr/datadog_checks/__init__.py
@@ -0,0 +1,4 @@
+# (C) Datadog, Inc. 2019
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)
diff --git a/mapr/datadog_checks/mapr/__about__.py b/mapr/datadog_checks/mapr/__about__.py
@@ -0,0 +1,4 @@
+# (C) Datadog, Inc. 2019
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+__version__ = '0.0.1'
diff --git a/mapr/datadog_checks/mapr/__init__.py b/mapr/datadog_checks/mapr/__init__.py
@@ -0,0 +1,7 @@
+# (C) Datadog, Inc. 2019
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+from .__about__ import __version__
+from .mapr import MaprCheck
+
+__all__ = ['__version__', 'MaprCheck']
diff --git a/mapr/datadog_checks/mapr/common.py b/mapr/datadog_checks/mapr/common.py
@@ -0,0 +1,173 @@
+# (C) Datadog, Inc. 2019
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+
+HISTOGRAM_METRICS = {'mapr.db.table.latency'}
+
+COUNT_METRICS = {
+    'mapr.cache.lookups_data',
+    'mapr.cache.lookups_dir',
+    'mapr.cache.lookups_inode',
+    'mapr.cache.lookups_largefile',
+    'mapr.cache.lookups_meta',
+    'mapr.cache.lookups_smallfile',
+    'mapr.cache.lookups_table',
+    'mapr.cache.misses_data',
+    'mapr.cache.misses_dir',
+    'mapr.cache.misses_inode',
+    'mapr.cache.misses_largefile',
+    'mapr.cache.misses_meta',
+    'mapr.cache.misses_smallfile',
+    'mapr.cache.misses_table',
+    'mapr.cldb.rpc_received',
+    'mapr.cldb.rpcs_failed',
+    'mapr.db.append_bytes',
+    'mapr.db.append_rpcrows',
+    'mapr.db.append_rpcs',
+    'mapr.db.cdc.sent_bytes',
+    'mapr.db.checkandput_bytes',
+    'mapr.db.checkandput_rpcrows',
+    'mapr.db.checkandput_rpcs',
+    'mapr.db.flushes',
+    'mapr.db.forceflushes',
+    'mapr.db.fullcompacts',
+    'mapr.db.get_bytes',
+    'mapr.db.get_readrows',
+    'mapr.db.get_resprows',
+    'mapr.db.get_rpcs',
+    'mapr.db.increment_bytes',
+    'mapr.db.increment_rpcrows',
+    'mapr.db.increment_rpcs',
+    'mapr.db.minicompacts',
+    'mapr.db.put_bytes',
+    'mapr.db.put_readrows',
+    'mapr.db.put_rpcrows',
+    'mapr.db.put_rpcs',
+    'mapr.db.repl.sent_bytes',
+    'mapr.db.scan_bytes',
+    'mapr.db.scan_readrows',
+    'mapr.db.scan_resprows',
+    'mapr.db.scan_rpcs',
+    'mapr.db.table.read_bytes',
+    'mapr.db.table.read_rows',
+    'mapr.db.table.resp_rows',
+    'mapr.db.table.rpcs',
+    'mapr.db.table.value_cache_hits',
+    'mapr.db.table.value_cache_lookups',
+    'mapr.db.table.write_bytes',
+    'mapr.db.table.write_rows',
+    'mapr.db.ttlcompacts',
+    'mapr.db.updateandget_bytes',
+    'mapr.db.updateandget_rpcrows',
+    'mapr.db.updateandget_rpcs',
+    'mapr.db.valuecache_hits',
+    'mapr.db.valuecache_lookups',
+    'mapr.fs.bulk_writes',
+    'mapr.fs.bulk_writesbytes',
+    'mapr.fs.kvstore_delete',
+    'mapr.fs.kvstore_insert',
+    'mapr.fs.kvstore_lookup',
+    'mapr.fs.kvstore_scan',
+    'mapr.fs.local_readbytes',
+    'mapr.fs.local_reads',
+    'mapr.fs.local_writebytes',
+    'mapr.fs.local_writes',
+    'mapr.fs.read_bytes',
+    'mapr.fs.read_cachehits',
+    'mapr.fs.read_cachemisses',
+    'mapr.fs.reads',
+    'mapr.fs.statstype_create',
+    'mapr.fs.statstype_lookup',
+    'mapr.fs.statstype_read',
+    'mapr.fs.statstype_write',
+    'mapr.fs.write_bytes',
+    'mapr.fs.writes',
+    'mapr.io.write_bytes',
+    'mapr.io.writes',
+    'mapr.rpc.bytes_recd',
+    'mapr.rpc.bytes_sent',
+    'mapr.rpc.calls_recd',
+    'mapr.streams.listen_bytes',
+    'mapr.streams.listen_msgs',
+    'mapr.streams.listen_rpcs',
+    'mapr.streams.produce_bytes',
+    'mapr.streams.produce_msgs',
+    'mapr.streams.produce_rpcs',
+    'mapr.volmetrics.read_ops',
+    'mapr.volmetrics.write_ops',
+}
+
+MONOTONIC_COUNTER_METRICS = {
+    'mapr.cldb.containers_created',
+    'mapr.drill.queries_completed',
+    'mapr.process.context_switch_involuntary',
+    'mapr.process.context_switch_voluntary',
+    'mapr.process.cpu_time.syst',
+    'mapr.process.cpu_time.user',
+    'mapr.process.disk_octets.read',
+    'mapr.process.disk_octets.write',
+    'mapr.process.disk_ops.read',
+    'mapr.process.disk_ops.write',
+    'mapr.process.page_faults.majflt',
+    'mapr.process.page_faults.minflt',
+}
+
+GAUGE_METRICS = {
+    'mapr.alarms.alarm_raised',
+    'mapr.cldb.cluster_cpu_total',
+    'mapr.cldb.cluster_cpubusy_percent',
+    'mapr.cldb.cluster_disk_capacity',
+    'mapr.cldb.cluster_diskspace_used',
+    'mapr.cldb.cluster_memory_capacity',
+    'mapr.cldb.cluster_memory_used',
+    'mapr.cldb.containers',
+    'mapr.cldb.containers_unusable',
+    'mapr.cldb.disk_space_available',
+    'mapr.cldb.nodes_in_cluster',
+    'mapr.cldb.nodes_offline',
+    'mapr.cldb.storage_pools_cluster',
+    'mapr.cldb.storage_pools_offline',
+    'mapr.cldb.volumes',
+    'mapr.db.cdc.pending_bytes',
+    'mapr.db.get_currpcs',
+    'mapr.db.index.pending_bytes',
+    'mapr.db.put_currpcs',
+    'mapr.db.repl.pending_bytes',
+    'mapr.db.scan_currpcs',
+    'mapr.db.table.latency',
+    'mapr.db.valuecache_usedSize',
+    'mapr.drill.allocator_root_peak',
+    'mapr.drill.allocator_root_used',
+    'mapr.drill.blocked_count',
+    'mapr.drill.count',
+    'mapr.drill.fd_usage',
+    'mapr.drill.fragments_running',
+    'mapr.drill.heap_used',
+    'mapr.drill.non_heap_used',
+    'mapr.drill.queries_running',
+    'mapr.drill.runnable_count',
+    'mapr.drill.waiting_count',
+    'mapr.io.read_bytes',
+    'mapr.io.reads',
+    'mapr.process.cpu_percent',
+    'mapr.process.data',
+    'mapr.process.mem_percent',
+    'mapr.process.rss',
+    'mapr.process.vm',
+    'mapr.status.ok',
+    'mapr.streams.listen_currpcs',
+    'mapr.topology.disks_total_capacity',
+    'mapr.topology.disks_used_capacity',
+    'mapr.topology.utilization',
+    'mapr.volmetrics.read_latency',
+    'mapr.volmetrics.read_throughput',
+    'mapr.volmetrics.write_latency',
+    'mapr.volmetrics.write_throughput',
+    'mapr.volume.logical_used',
+    'mapr.volume.quota',
+    'mapr.volume.snapshot_used',
+    'mapr.volume.total_used',
+    'mapr.volume.used',
+}
+
+ALLOWED_METRICS = GAUGE_METRICS | COUNT_METRICS | MONOTONIC_COUNTER_METRICS
diff --git a/mapr/datadog_checks/mapr/data/conf.yaml.example b/mapr/datadog_checks/mapr/data/conf.yaml.example
@@ -0,0 +1,45 @@
+init_config:
+
+instances:
+  -
+    ## @param ticket_location - string - optional
+    ## The path to the MapR user ticket, if included it overrides the MAPR_TICKETFILE_LOCATION environment variable.
+    ## Either the environment variable or this config option needs to be set if security is enabled on the cluster.
+    #
+    # ticket_location: <TICKETFILE_LOCATION>
+
+    ## @param hostname - string - optional - default: `socket.getfqdn()`
+    ## The MapR host to monitor. This is used to find the correct topic to read metrics from.
+    ## https://mapr.com/docs/61/AdministratorGuide/spyglass-on-streams.html
+    #
+    # hostname: <FQDN>
+
+    ## @param stream_path - string - optional - default: /var/mapr/mapr.monitoring/metricstreams
+    ## The MapR topic path.
+    #
+    # stream_path: /var/mapr/mapr.monitoring/metricstreams
+
+    ## @param metric_whitelist - list - optional
+    ## List of regexes for metrics to collect. Note that you can only collect metrics starting with "mapr.",
+    ## see https://github.com/DataDog/integrations-core/blob/master/mapr/datadog_checks/mapr/common.py
+    ## for the list of metrics you can collect.
+    ## Collect all mapr related metrics by default
+    #
+    # metric_whitelist:
+    #   - '.*'
+
+    ## @param streams_count - integer - optional - default: 1
+    ## The mapr setting for the number of monitoring streams.
+    ## Changing the number of MaprR monitoring streams is a deprecated process, only change this if you
+    ## use more than one.
+    #
+    # streams_count: 1
+
+    ## @param tags  - list of key:value elements - optional
+    ## A list of tags to attach to every metric, event, and service check emitted by this integration.
+    ##
+    ## Learn more about tagging: https://docs.datadoghq.com/tagging/
+    #
+    # tags:
+    #   - <KEY_1>:<VALUE_1>
+    #   - <KEY_2>:<VALUE_2>