Report data structure documentation #2025

alepuccetti · 2016-11-22T16:15:15Z

This PR adds documentation for the primary data structures used to create a report. It will be of great help to developers that want to build their plugins.

I did this work on the website. Do you want me to add the same information on https://github.com/weaveworks/scope/blob/master/examples/plugins/README.md?

Relevant to #1963

/cc @2opremio @abuehrle

site/plugins.md

+### Time Window
+A report may have an attribute called `Window`.
+This is the time window, expressed as duration, within which the data contained in the report are considered valid.
+The default window is 15 seconds


2opremio · 2016-12-12T17:59:59Z

@ekimekim Considering you would have appreciated this sort of documentation recently, would you mind reviewing it?

ekimekim

A lot of open questions. A few formatting comments.

site/plugins.md

+}
+```
+
+Nodes are stored in dictionary. The ID of nodes representing hosts or containers have the format `ID;<type>`, with type equal `host` or `container` accordingly.


site/plugins.md

+
+- `latest` - the latest values to display.
+- `latestControls` - the latest available controls.
+- `metrics` - the collection of metrics to display in the UI.


site/plugins.md

+
+- `timestamp` is the timestamp of when the control was exposed.
+- `value` is an object containing the control value. At the moment, only the state is available.
+ - `Dead` is a boolean to know the state (active, dead) of a control. It is useful to show controls only when they are in a usable state.


site/plugins.md

+ - `Dead` is a boolean to know the state (active, dead) of a control. It is useful to show controls only when they are in a usable state.
+
+### Metadata Templates
+Metadata Templates describe of a particular metadata.


site/plugins.md

+- `from` indicates where to look for the metadata. The possible values are:
+ - `latest`
+ - `sets`
+ - `counters`


site/plugins.md

+
+- `metric-template-id` and `id` identify a particular metric template.
+- `label` contains the label that will be used by Scope UI.
+- `format` describes who the metrics is formatted.


site/plugins.md

+- `label` contains the label that will be used by Scope UI.
+- `format` describes who the metrics is formatted.
+ - `percent` the metric value is a percentage.
+ - `filesize` the metric value is a file size (e.g. memory usage).


site/plugins.md

+}
+```
+
+### Metrics


site/plugins.md

+
+- `metadata-template-identifier` and `id` identify a particular metadata template.
+- `label` contains the label that will be used by Scope UI.
+- `priority` is a floating point value used to decide the display ordering (lower values are displayed before higher ones).


site/plugins.md

+      "another-metadata-template-id": {...}
+}
+```
+


2opremio · 2017-01-12T13:51:26Z

@alepuccetti Ping

alepuccetti · 2017-01-12T14:05:45Z

@2opremio I will get back to this soon. Lately, I was busy debugging and testing the ebpf program distro agnostic.

alepuccetti · 2017-01-17T12:30:06Z

I added a new commit for an easier review, I will squash it before merge.

2opremio · 2017-03-11T09:47:25Z

@alepuccetti ping

alepuccetti · 2017-03-11T14:21:48Z

I am on vacation, I will be back the 15th.

On Sat, Mar 11, 2017 at 4:47 PM Alfonso Acosta ***@***.***> wrote: @alepuccetti <https://github.com/alepuccetti> ping — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2025 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGRc72Lr0kehwIUV5wSk3VY6UvqOdyxUks5rkm2ugaJpZM4K5oHq> .

-- Alessandro Puccetti

site/plugins.md

+To pair metadata with its template, it is necessary to use the `metadata-template-id` as a key to identify that particular piece of data. Example:
+
+```json
+"metadata-templates": {


site/plugins.md

+A report can contain many types of information.
+If you go back to the [Reporter Interface](#reporter-interface) section, you see the top-level `Plugins` attribute.
+Along with that, a report may contain multiple topologies.
+An example of a report containing few topologies is the following:


site/plugins.md

+
+### Topologies
+Topologies can be of various types.
+Each topology consists of a list of nodes and other information about the topology.


site/plugins.md

+- `table_templates` - contains the templates used to render tables into the Scope UI.
+- `metric_templates` - contains the templates used to render metrics into the Scope UI.
+
+**Note**: These attribute are not required. But a topology with no `nodes` does not have any information to render, `metadata_templates`, as well as `table_templates`, are needed to know how to render the information carried by `nodes` in the Scope UI.


site/plugins.md

+
+### Nodes
+A Node contains information about a specific element of a topology.
+For example the Host topology will contain nodes describing all the hosts in it.


site/plugins.md

+### Nodes
+A Node contains information about a specific element of a topology.
+For example the Host topology will contain nodes describing all the hosts in it.
+The same is for containers and Container topology, pods and Pod topology and so on.


site/plugins.md

+}
+```
+
+- `timestamp` is the timestamp of when the control was exposed.


site/plugins.md

+- `metadata-template-identifier` and `id` identify a particular metadata template.
+- `label` contains the label that will be used by Scope UI.
+- `dataType` specifies the type of data, this will determine how the value is displayed. Possible values for this attribute are: number, ip, and datetime.
+- `priority` is a floating point value used to decide the display ordering (lower values are displayed before higher ones). If it is omitted the UI display the value as last.


site/plugins.md

+- `label` contains the label that will be used by Scope UI.
+- `prefix` is used to identify which metadata templates belong to the table.
+
+If you want to display data in a table, you have to define a table template and prepend the table prefix to all the metadata templates that identifies the data you want to put in such table.


site/plugins.md

+The following is an example of a report with a metric and a its metric template:
+
+```json
+"metric_templates": {


site/plugins.md

+This is the time window, expressed as duration, within which the data contained in the report are considered valid.
+The default window is 15 seconds.
+You may change the window value using the option `-app.window <SECONDS>` when launching scope.
+However, using values less than 15 seconds increases the change of information not be correctly displayed.


alban · 2017-04-04T13:59:43Z

I rebased the branch, squashed the commits & added a couple of new fixes from the review above.

site/plugins.md

+This is the time window, expressed as duration, within which the data contained in the report are considered valid.
+The default window is 15 seconds.
+You may change the window value using the option `-app.window <SECONDS>` when launching scope.
+However, using values less than 15 seconds increases the chance of information not being correctly displayed.


site/plugins.md

-**See Also**
+### <a id="report-data-structures"></a>Report Data structures
+A report can contain many types of information.
+If you go back to the [Reporter Interface](#reporter-interface) section, you see the top-level `Plugins` attribute.


site/plugins.md

+### Topologies
+Topologies can be of various types.
+Each topology consists of some general information and a list of nodes.
+The available topologies are:


site/plugins.md

+Each topology consists of some general information and a list of nodes.
+The available topologies are:
+
+- `Endpoint` nodes are (address, port) tuples on each host.


site/plugins.md

+### Controls
+Controls describe interfaces that expose actions that the user can perform on different objects (e.g. host, container, etc.).
+Controls are an element of nodes. In this way, each control in a node is attached to it and performs an action on the object described by the node itself. Below is an example of how controls are represented in the JSON report.
+In the report the attribute `latest_controls` contains all the controls exposed by scope and/or plugins, but only those alive will be listed in the attribute `controls`.


site/plugins.md

+### Metadata
+All metadata entries are placed within nodes in the section named `latest`.
+This section contains the latest values to display and consists of `timestamp` and `value`.
+Both should be written as strings in the json (with the double quotes).


site/plugins.md

+- `dataType` specifies the type of data, this will determine how the value is displayed. Possible values for this attribute are: "number", "ip", "datetime" and "" for strings.
+- `priority` is a floating point value used to decide the display ordering (lower values are displayed before higher ones). If omitted, the UI will display it last.
+- `from` indicates where to look for the metadata. The possible values are:
+ - `latest`


site/plugins.md

+- `metric-template-id` and `id` identify a particular metric template.
+- `label` contains the label that will be used by Scope UI.
+- `format` describes how the metrics is formatted.
+ - `percent` the metric value is a percentage.


alban · 2017-04-13T09:24:08Z

@ekimekim: please take a second look

ekimekim

A lot better but still missing some important information.

site/plugins.md

+- `Pods` nodes represent all Kubernetes pods running on hosts running probes.
+- `Service` nodes represent all Kubernetes services running on hosts running probes.
+- `Deployment` nodes represent all Kubernetes deployments running on hosts running probes.
+- `ReplicaSet` nodes represent all Kubernetes ReplicaSets running on hosts running probes.


site/plugins.md

+```
+
+Nodes are stored in a dictionary.
+The ID of nodes representing hosts or containers has the format `ID;<type>`, with type equal to the literal string `host` or `container` respectively.


site/plugins.md

+}
+```
+
+- `metadata-template-identifier` and `id` identify a particular metadata template.


site/plugins.md

+- `from` indicates where to look for the metadata. The possible values are:
+  - `latest`
+  - `sets`
+  - `counters`


schu · 2017-04-18T11:32:57Z

@ekimekim please take a look again.

ekimekim

One last thing worth mentioning I just thought of. But everything currently here looks good. I'm happy to merge as is.

site/plugins.md

@@ -467,6 +467,8 @@ In particular, a node may contain:
 - `latest` - an id-value map containing the latest values. Each id has only one value.
 - `latestControls` - the latest available controls.
 - `metrics` - the collection of metrics to display in the UI. Each metric has multiple timestamped values.
+- `sets` - a string->set-of-strings map, for example a list of local networks.
+- `counters` - a string->int map.


abuehrle

small tweaks to the text

site/plugins.md

@@ -16,7 +16,8 @@ The following topics are discussed:
     * [Naming Nodes](#naming-nodes)
 * [A Guide to Developing Plugins](#plugins-developing-guide)
  * [Setting up the Structure](#structure)
-  * [Defining the Reporter Interface](#reporter-interface)
+  * [Defining the Reporter Interface](#defining-reporter-interface)
+  * [Report Data Structures](#report-data-structures)

 Any kind of metrics can be generated and inserted into Scope using custom plugins. Metrics generated through your plugin are displayed in the user interface alongside the standard set of metrics that are found in Weave Scope.



site/plugins.md

+Topologies can be of various types.
+Each topology consists of some general information and a list of nodes.
+These are the available topologies:
+


site/plugins.md

+All metadata entries are placed within nodes in the section named `latest`.
+This section contains the latest values to display and consists of `timestamp` and `value`.
+Both should be written as json strings (with double quotes).
+Scope uses `metadata_templates` to know how to display such data.


site/plugins.md

+
+### Metadata Templates
+Metadata Templates describe a particular metadata item.
+This description is used to extract metadata from a node and display it on Scope UI.



site/plugins.md

+- `id` is a string identifying the particular metadata template (here `traffic-control-pktloss`) and is also used as a key to the template value.
+- `label` contains the label that will be used by Scope UI.
+- `dataType` specifies the type of data, this will determine how the value is displayed. Possible values for this attribute are: "number", "ip", "datetime" and "" for strings.
+- `priority` is a floating point value used to decide the display ordering (lower values are displayed before higher ones). If omitted, the UI will display it last.


site/plugins.md

+
+### Table Templates
+Table Templates describe a table and how to identify which metadata templates belong to the table.
+


site/plugins.md

+- `prefix` is used to identify which metadata templates belong to the table.
+
+If you want to display data in a table, you have to define a table template and prepend the table prefix to all the metadata templates that identify the data you want to put in such table.
+


site/plugins.md

+Scope uses the `metric_templates` to know how to display such data. To pair a metric with its template, it is necessary to use the `metric-template-id` as key to identify that particular metric.
+Metrics are suitable for information such as CPU and memory usage, HTTP requests rate, I/O operations, etc.
+The following is an example of a report with a metric preceded by its metric template:
+


site/plugins.md

+- `metric-template-id` and `id` identify a particular metric template.
+- `label` contains the label that will be used by Scope UI.
+- `format` describes how the metrics is formatted.
+  - `percent` the metric value is a percentage.


site/plugins.md

+The default window is 15 seconds.
+You may change the window value using the option `-app.window <SECONDS>` when launching scope.
+However, using values smaller than 15 seconds increases the chance of information not being correctly displayed.
+


This patch adds the report's data structures outline and specifications for the topologies, the nodes, the metadata templates, the table templates, the metric templates, the controls, the metadata, the metrics and the time window.

From Anita Buehrle's review

alban · 2017-04-26T12:37:24Z

Thanks for the review, I rebased & pushed a new commit with changes from @abuehrle's review.

alban · 2017-04-26T15:02:52Z

lint complains in CircleCI after the rebase, even after a CircleCI build without cache:

cd $SRCDIR; make RM= lint
docker run  -ti \
		-v /home/ubuntu/src/github.com/weaveworks/scope:/go/src/github.com/weaveworks/scope \
		-v /home/ubuntu/src/github.com/weaveworks/scope/.pkg:/go/pkg \
		--net=host \
		-e GOARCH -e GOOS -e CIRCLECI -e CIRCLE_BUILD_NUM -e CIRCLE_NODE_TOTAL \
		-e CIRCLE_NODE_INDEX -e COVERDIR -e SLOW -e TESTDIRS \
		weaveworks/scope-backend-build SCOPE_VERSION=1bc0bca2 GO_BUILD_INSTALL_DEPS=-i CODECGEN_UID=0 lint
make: Entering directory '/go/src/github.com/weaveworks/scope'

./tools/lint -ignorespelling "agre " -ignorespelling "AGRE " .

./extras/dialer/dialer:24:27: reached } without matching [ with ]

./extras/dialer/dialer: run shfmt -i 4 -w ./extras/dialer/dialer

./extras/dialer/time-scope-probe-cgroup:31:18: arithmetic expressions must consist of names and numbers

...

I don't understand why: this PR only changes the documentation, not the code.

ekimekim · 2017-04-26T16:57:03Z

At a guess: this may be another case of a bug I feel I've seen before (maybe only locally? i don't recall) where "no code files changed" somehow becomes "check all files". I'll check it out. In the meantime, feel free to ignore the failing lint check and we can merge once @abuehrle is happy.

abuehrle · 2017-04-26T17:16:44Z

LGTM

alban · 2017-04-27T14:42:58Z

For the record, CircleCI was happy with this PR once it got merged into master: https://circleci.com/gh/weaveworks/scope/7076

alepuccetti force-pushed the alessandro/data-structures branch 4 times, most recently from 9b85635 to 278a356 Compare November 30, 2016 08:46

2opremio reviewed Dec 5, 2016

View reviewed changes

site/plugins.md Outdated

### Time Window

A report may have an attribute called `Window`.

This is the time window, expressed as duration, within which the data contained in the report are considered valid.

The default window is 15 seconds

This comment was marked as abuse.

Sign in to view

ekimekim reviewed Dec 13, 2016

View reviewed changes

alepuccetti force-pushed the alessandro/data-structures branch from 278a356 to 0776d9d Compare January 17, 2017 11:10

alepuccetti force-pushed the alessandro/data-structures branch 4 times, most recently from 2918e8b to f6e4b16 Compare January 17, 2017 15:04

alepuccetti force-pushed the alessandro/data-structures branch 3 times, most recently from c6f0244 to 720365e Compare February 20, 2017 13:58

alban reviewed Apr 4, 2017

View reviewed changes

site/plugins.md Outdated

To pair metadata with its template, it is necessary to use the `metadata-template-id` as a key to identify that particular piece of data. Example:

```json

"metadata-templates": {

This comment was marked as abuse.

Sign in to view

iaguis reviewed Apr 4, 2017

View reviewed changes

alban force-pushed the alessandro/data-structures branch from 720365e to 68d9dcb Compare April 4, 2017 13:58

iaguis reviewed Apr 4, 2017

View reviewed changes

iaguis approved these changes Apr 4, 2017

View reviewed changes

ekimekim suggested changes Apr 13, 2017

View reviewed changes

ekimekim approved these changes Apr 25, 2017

View reviewed changes

abuehrle suggested changes Apr 25, 2017

View reviewed changes

alban mentioned this pull request Apr 26, 2017

plugin documentation: behaviour when merging the same node from reports #2474

Open

Alessandro Puccetti and others added 11 commits April 26, 2017 14:35

site/plugins: add report data structures section

8472324

This patch adds the report's data structures outline and specifications for the topologies, the nodes, the metadata templates, the table templates, the metric templates, the controls, the metadata, the metrics and the time window.

site/plugins: data structures: fixes

d2fc48f

site/plugins.md: fixes after Iago's review

176ffcf

site/plugins.md: clarify the difference between latest and metrics

bcae01c

site/plugins.md: english wording

6966a8c

site/plugins.md: more fixes thanks to Iago

73a2eb0

Mention new ECSTask and ECSService

525beba

site/plugins.md: fix identifier typo

3fb13e8

site/plugins.md: clarify that node ids are different

d9b6e81

site/plugins.md: document sets and counters

beefec1

site/plugins.md: English wording

1bc0bca

From Anita Buehrle's review

alban force-pushed the alessandro/data-structures branch from 4a4a248 to 1bc0bca Compare April 26, 2017 12:36

site/plugins.md: Metadata Templates: change English phrasing

a3d262f

ekimekim merged commit e127392 into weaveworks:master Apr 26, 2017


		### Table Templates
		Table Templates describe a table and how to identify which metadata templates belong to the table.

		- `prefix` is used to identify which metadata templates belong to the table.

		If you want to display data in a table, you have to define a table template and prepend the table prefix to all the metadata templates that identify the data you want to put in such table.

Report data structure documentation #2025

Report data structure documentation #2025

Conversation

alepuccetti commented Nov 22, 2016

This comment was marked as abuse.

2opremio commented Dec 12, 2016

ekimekim left a comment

Choose a reason for hiding this comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

2opremio commented Jan 12, 2017

alepuccetti commented Jan 12, 2017

alepuccetti commented Jan 17, 2017

2opremio commented Mar 11, 2017

alepuccetti commented Mar 11, 2017 via email

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

alban commented Apr 4, 2017

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

alban commented Apr 13, 2017

ekimekim left a comment

Choose a reason for hiding this comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

schu commented Apr 18, 2017

ekimekim left a comment

Choose a reason for hiding this comment

This comment was marked as abuse.

This comment was marked as abuse.

abuehrle left a comment

Choose a reason for hiding this comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.