From 2f58eb9f6bcf6f369cd912e2d4e75e9416006585 Mon Sep 17 00:00:00 2001 From: Mario Castro Date: Fri, 14 Feb 2020 13:34:52 +0100 Subject: [PATCH] Create-metricset docs slight update (#8375) --- docs/devguide/create-metricset.asciidoc | 10 +++---- docs/devguide/metricset-details.asciidoc | 36 +++++++++--------------- 2 files changed, 19 insertions(+), 27 deletions(-) diff --git a/docs/devguide/create-metricset.asciidoc b/docs/devguide/create-metricset.asciidoc index 36c6d109a83..3331d69d121 100644 --- a/docs/devguide/create-metricset.asciidoc +++ b/docs/devguide/create-metricset.asciidoc @@ -10,14 +10,14 @@ implementation of existing metricsets for inspiration. To create a new metricset: -. Run the following command inside your beat directory: +. Run the following command inside the metricbeat beat directory: + [source,bash] ---- make create-metricset ---- + -You'll be prompted to enter a module and metricset name. Only use characters `[a-z]` +You need Python to run this command, then, you'll be prompted to enter a module and metricset name. Remember that a module represents the service you want to retrieve metrics from (like Redis) and a metricset is a specific set of grouped metrics (like `info` on Redis). Only use characters `[a-z]` and, if required, underscores (`_`). No other characters are allowed. + When you run `make create-metricset`, it creates all the basic files for your metricset, along with the required module @@ -37,12 +37,12 @@ make ---- + The first command, `make collect`, updates all generated files with the most recent files, data, and meta information from the metricset. The second command, -`make`, compiles your source code and provides you with a binary called `{beat}` in the beat folder. You can run the +`make`, compiles your source code and provides you with a binary called metricbeat in the same folder. You can run the binary in debug mode with the following command: + [source,bash] ---- -./{beat} -e -d "*" +./metricbeat -e -d "*" ---- After running the make commands, you'll find the metricset, along with its generated files, under `module/{module}/{metricset}`. This directory @@ -92,7 +92,7 @@ func init() { [float] ===== Definition -The MetricSet type defines all fields of the metricset. As a minimum it must inherit the `mb.BaseMetricSet` fields, +The MetricSet type defines all fields of the metricset. As a minimum it must be composed of the `mb.BaseMetricSet` fields, but can be extended with additional entries. These variables can be used to persist data or configuration between multiple fetch calls. diff --git a/docs/devguide/metricset-details.asciidoc b/docs/devguide/metricset-details.asciidoc index 32e92f05b3e..7e21f314f17 100644 --- a/docs/devguide/metricset-details.asciidoc +++ b/docs/devguide/metricset-details.asciidoc @@ -87,27 +87,21 @@ functionality of the metricset and `Fetch` method from the data mapping. [float] ==== fields.yml -The `fields.yml` file is used for different purposes: - -* Creates the Elasticsearch template -* Creates the Kibana index pattern configuration -* Creates the Exported Fields documentation for the metricset - -To make sure the Elasticsearch template is correct, it's important to keep this file up-to-date -with all the changes. There is a `fields.yml` file under `module/{module}/_meta/fields.yml` that contains -the general top level structure for all metricsets. Normally you only need to modify the description in this file. - -Here an example for the `fields.yml` file from the MySQL module. +You can find up to 3 different types of files named `fields.yml` in the beats repository for each metricbeat module: +* `metricbeat/fields.yml`: Contains the definitions to create the Elasticsearch template, the Kibana index pattern configuration and the exported fields documentation for metricsets. To make sure the Elasticsearch template is correct, it's important to keep this file up-to-date with all the changes. Generally, you shouldn't touch this file manually because it's generated by some commands in the build environment. +* `metricbeat/module/{module}/_meta/fields.yml`: Contains the general top level structure for all metricsets in a module. +Normally you only need to modify the description in this file. Here is an example for the `fields.yml` file from the MySQL module. ++ [source,yaml] ---- include::../../metricbeat/module/mysql/_meta/fields.yml[] ---- - -There is another `fields.yml` file under `module/{module}/{metricset}/_meta/fields.yml` that contains all fields retrieved -by the metricset. As field types, each field must have a core data type ++ +* `metricbeat/module/{module}/{metricset}/_meta/fields.yml`: Contains all fields definitions retrieved by the metricset. +As field types, each field must have a core data type https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-types.html#_core_datatypes[supported by elasticsearch]. Here's a very basic example that shows one group from the MySQL `status` metricset: - ++ [source,yaml] ---- - name: status @@ -117,8 +111,7 @@ https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-types.htm fields: - name: aborted type: group - description: > - Aborted status fields. + description: Aborted status fields. fields: - name: clients type: integer @@ -130,8 +123,7 @@ https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-types.htm description: > The number of failed attempts to connect to the MySQL server. ---- - -As you can see, if there are nested fields, you must use the type `group`. ++ // TODO: Add link to general fields.yml developer guide @@ -148,14 +140,14 @@ We recommend that you use all three when you create a metricset. Unit tests are written in Go and have no dependencies. Integration tests are also written in Go but require the service from which the module collects metrics to also be running. System tests for Metricbeat also require the service to be running in most cases and are -written in Python based on our small Python test framework. +written in Python {python_major_version} based on our small Python test framework. We use `virtualenv` to deal with Python dependencies. You can simply run the command `make python-env` and then `. build/python-env/bin/activate` . You should use a combination of the three test types to test your metricsets because each method has advantages and disadvantages. To get started with your own tests, it's best to look at the existing tests. You'll find the unit and integration tests -in the `_test.go` files under existing modules and metricsets. +in the `_test.go` files under existing modules and metricsets. Integration tests usually take the form of `TestFetch` and `TestData`. The system tests are under `tests/systems`. @@ -240,7 +232,7 @@ func GetEnvPort() string { <2> ---- <1> Add any additional config options your metricset needs here. -<2> The endpoint used by the metricset needs to be configurable for manual and automated testing. +<2> The endpoint used by the metricset needs to be configurable for manual and automated testing. Environment variables should be defined in the module under `_meta/env` and included in the `docker-compose.yml` file. The `TestFetch` integration test will return a single event from your metricset, which you can use to test the validity of the data.