Add mysql docs #313

mtojek · 2020-03-31T11:14:56Z

Changes:

write basic MySQL README file (output: https://github.com/mtojek/package-registry/blob/add-mysql-docs/dev/packages/beats/mysql-0.0.1/docs/README.md)
use nginx README file as template
implement template function to generated exported fields
apply exported fields to MySQL, nginx READMEs
fix typo: resouce
replace default empty README file with single word "TODO" (smaller empty READMEs)

Resolves: #278

ruflin · 2020-03-31T11:37:23Z

dev/packages/beats/nginx-0.0.1/docs/README.md

+
+| Field | Description | Type |
+|---|---|---|
+| nginx.access.agent | Alias for field "user_agent.original" | alias |


What about all the ECS fields that are used as part of the access dataset? They should also be documented here.

Issue created for this.

ruflin

One thing missing here is the updating the fields.yml files. The problem becomes obvious with the nginx access logs. Historically there was just one massive fields.yml for Filebeat where all the fields were shipped together and we had a tree of definitions: Global, ECS, Module, Fileset. Now that we have a template per Dataset, all the fields used must be defined on the Dataset level. This has the advantage that the template becomes much more compact but brings the challenge, there is no easy way to tell which fields from Global, ECS etc. are used in the dataset. We can't just take all of ECS as this is too many fields and not all are used.

Having it documented separately also solves an other issue: It allows us to document how an ECS fields is exactly used in a dataset. What does source.ip exactly mean in the context of nginx.access logs. So far, we only had a generic doc but now we can fill in the details.

ruflin · 2020-03-31T11:40:29Z

dev/packages/beats/mysql-0.0.1/dataset/status/manifest.yml

@@ -8,7 +8,13 @@ streams:
  - default:
    - root:secret@tcp(127.0.0.1:3306)/
    name: hosts
+  - default: secret


I think we are missing here all the additional fields which are need to make the UI nicer. See https://github.com/elastic/package-registry/blob/master/dev/packages/example/nginx-1.2.0/dataset/access/manifest.yml#L17 for an example.

So this is mostly manual work here. Once I fix it, it would be harder to automatically import beats. Should all options have these properties?

I think most fields will need it, at least part of the options. And yes, this will probably rely on manual work as it is not something we have today.

ruflin · 2020-03-31T11:40:45Z

dev/packages/beats/mysql-0.0.1/dataset/galera_status/manifest.yml

@@ -4,5 +4,17 @@ release: beta
 type: metrics
 streams:
 - input: mysql/metrics
+  vars:


Same like below for the UI part

Responded above

ruflin · 2020-03-31T11:42:42Z

dev/import-beats-resources/nginx/docs/README.md

+}
+```
+
+{{fields "stubstatus"}}


Good to see such a feature. I wonder if this should become part of the overall build step instead of the import part. I'm good having it here at the moment.

I would keep it here at the moment, later on we can think about refactoring this part.

ruflin · 2020-03-31T11:43:11Z

dev/import-beats-resources/nginx/docs/README.md

+An example event for nginx looks as following:
+
+```$json
+{


Where did you take this even from? Existing from Beat? This will probably look slightly different when shipped through the agent.

Yes, I will refresh the samples once I run the agent.

ruflin · 2020-03-31T11:48:04Z

dev/packages/beats/nginx-0.0.1/docs/README.md

+| Field | Description | Type |
+|---|---|---|
+| nginx.access.agent | Alias for field "user_agent.original" | alias |
+| nginx.access.body_sent.bytes | Alias for field "http.response.body.bytes" | alias |


These alias fields are not part of the even anymore. These were only used for migration from 6.x to 7.x and are not needed anymore.

Based on the offline discussion - I will skip all the alias fields with the migration: true property.

ruflin

Approving the PR as I think this is already a big step forward and we can further iterate on it.

ruflin · 2020-03-31T12:16:23Z

@mtojek An other thing that just came to my mind is that as part of the migration we should also start thinking about optimising the dashboards. For example the query for event.dataset should become a query for stream.dataset which is a constant keyword and much more efficient. The gent should always add these fields (@michalpristas will know more if this already happens).

I don't think we need to do all these optimisations directly during the migration but could be done later on. But we should start a checklist with these already now so we don't forget.

mtojek · 2020-03-31T12:23:25Z

@mtojek An other thing that just came to my mind is that as part of the migration we should also start thinking about optimising the dashboards. For example the query for event.dataset should become a query for stream.dataset which is a constant keyword and much more efficient. The gent should always add these fields (@michalpristas will know more if this already happens).

Ok, but this is a new requirement that I wasn't aware before. Probably will have to adjust the dashboard generating process. I will create an issue for this.

EDIT:
Issue: #315

mtojek · 2020-03-31T12:33:25Z

One thing missing here is the updating the fields.yml files. The problem becomes obvious with the nginx access logs. Historically there was just one massive fields.yml for Filebeat where all the fields were shipped together and we had a tree of definitions: Global, ECS, Module, Fileset. Now that we have a template per Dataset, all the fields used must be defined on the Dataset level. This has the advantage that the template becomes much more compact but brings the challenge, there is no easy way to tell which fields from Global, ECS etc. are used in the dataset. We can't just take all of ECS as this is too many fields and not all are used.

Having it documented separately also solves an other issue: It allows us to document how an ECS fields is exactly used in a dataset. What does source.ip exactly mean in the context of nginx.access logs. So far, we only had a generic doc but now we can fill in the details.

#316

Marcin Tojek added 3 commits March 30, 2020 20:35

Render README without fields

7dec8ba

Add MySQL docs

c2d0bee

Fix: mage check

17b29de

mtojek requested a review from ruflin March 31, 2020 11:14

mtojek self-assigned this Mar 31, 2020

ruflin reviewed Mar 31, 2020

View reviewed changes

ruflin approved these changes Mar 31, 2020

View reviewed changes

Remove all aliases if migration=true

f374e96

This was referenced Mar 31, 2020

Use stream.dataset instead of event.dataset #315

Closed

MySQL integration: add ECS fields to docs #316

Closed

This was referenced Mar 31, 2020

MySQL integration: refresh event sample in docs #317

Closed

MySQL integration: additional var properties for UI #318

Closed

mtojek merged commit 1d31dff into elastic:master Mar 31, 2020

mtojek mentioned this pull request May 13, 2020

Discuss: place to store README.md templates elastic/integrations#20

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add mysql docs #313

Add mysql docs #313

mtojek commented Mar 31, 2020 •

edited

Loading

ruflin Mar 31, 2020

mtojek Mar 31, 2020

ruflin left a comment

ruflin Mar 31, 2020

mtojek Mar 31, 2020

mtojek Mar 31, 2020

ruflin Mar 31, 2020

ruflin Mar 31, 2020

mtojek Mar 31, 2020

ruflin Mar 31, 2020

mtojek Mar 31, 2020

ruflin Mar 31, 2020

mtojek Mar 31, 2020

mtojek Mar 31, 2020

ruflin Mar 31, 2020

mtojek Mar 31, 2020

mtojek Mar 31, 2020

ruflin left a comment

ruflin commented Mar 31, 2020

mtojek commented Mar 31, 2020 •

edited

Loading

mtojek commented Mar 31, 2020

Add mysql docs #313

Add mysql docs #313

Conversation

mtojek commented Mar 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

ruflin commented Mar 31, 2020

mtojek commented Mar 31, 2020 • edited Loading

mtojek commented Mar 31, 2020

mtojek commented Mar 31, 2020 •

edited

Loading

mtojek commented Mar 31, 2020 •

edited

Loading