Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Usage improvements #884

Merged
merged 6 commits into from
Jul 13, 2020
Merged

Usage improvements #884

merged 6 commits into from
Jul 13, 2020

Conversation

ebeahan
Copy link
Member

@ebeahan ebeahan commented Jul 8, 2020

Adds USAGE.md covering the getting started experience and usage of the ECS tooling. The current generator.py arguments are documented along with example usage.

The aim is to simplify the experience for users generating their own artifacts (based off the published or custom schema definitions).

USAGE.md is linked from README.md

Caveats

Addresses #746

@ebeahan ebeahan self-assigned this Jul 8, 2020
@ebeahan ebeahan changed the title Usage doc improvements Usage improvements Jul 8, 2020
@ebeahan
Copy link
Member Author

ebeahan commented Jul 9, 2020

@jonathan-buttner @marshallmain Would you take a pass and let me know if you have any feedback on the usage docs for some of the generator features you've added? (#737 #856 #748 #851)

Copy link
Contributor

@jonathan-buttner jonathan-buttner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great readme! Thanks for creating it!

USAGE.md Outdated Show resolved Hide resolved
USAGE.md Outdated Show resolved Hide resolved
@ebeahan ebeahan merged commit b9f806d into elastic:master Jul 13, 2020
@ebeahan ebeahan deleted the usage-doc-improvements branch July 13, 2020 21:12
Copy link
Contributor

@webmat webmat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fantastic, thanks for turning my infamous comment on #746 into proper documentation 😃

I'm providing a few thoughts here for a follow-up PR. That seemed the simplest way to provide this feedback 🙂

Comment on lines +54 to +60
Prior to installing dependencies or running the tools, it's recommended to check out the `git` branch for the ECS version being targeted.

**Example**: For ECS `1.5.0`:

```
$ git checkout v1.5.0
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we have --ref we don't need this step anymore :-)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great point, we should definitely promote --ref instead.

I ran up against an issue (as I was going to update the docs 😄 ) trying to run the generator with --ref (#892), so I may hold off removing this until we squash that bug.

* The script will need to be executed from the top-level of the ECS repo
* The `version` displayed when running `generator.py` is based on the current value of the [version](version) file in the top-level of the repo

The following options add functionality beyond the defaults.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add here that the ECS team maintains the official ECS artifacts by running the script in this way.

Then we can segue from that to explain "and now here's how you add your own fields, or remove the fields you don't need"

@@ -0,0 +1,274 @@
# ECS Tooling Usage

In addition to the published schema and generated artifacts, the ECS repo also contains tools to generate artifacts based on the current published and custom schemas.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should expand more here.

Q. Why do people care?
A.

  • ECS has too many fields
  • ECS doesn't have their custom fields.

=> Users can use ECS tools to fix both of these things in their project. They can maintain only their customizations and use these tools to produce relevant artifacts for their various data sources.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to this, we could specifically call out the two artifacts that I expect users will find most useful. The Elasticsearch template is expected to be most useful for users in general. But Beats contributors may instead be interested in the Beats-formatted YAML field definition files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are great additions. Maybe another nice, future addition is tutorial-like walkthroughs showing end-to-end how to define your own schema customizations, generate artifacts, and then application in Beats or ES.

$ python scripts/generator.py --out ../myproject/ecs/out/
```

Inside the directory passed in as the target dir to the `--out` flag, two directories, `generated` and `docs`, will be created. `docs` will contain three asciidoc files based on the contents of the provided schema. `generated` will contain the various artifacts laid out as in the published repo (`beats`, `csv`, `ecs`, `elasticsearch`).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As soon as either --include or --subset are used, the asciidoc is not generated, though.

$ python scripts/generator.py --ref v1.5.0
```
> Note: `--ref` does have a dependency on `git` being installed and all expected commits/tags fetched from the ECS upstream repo. This will unlikely be an issue unless you downloaded the ECS as a zip archive from GitHub vs. cloning it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to mention 👍


#### Intermediate-Only

The `--intermediate-only` argument is used for debugging purposes. It only generates the "intermediate files" of ECS without generating the rest of the artifacts.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could mention them explicitly here, ecs_flat.yml and ecs_nested.yml

@ebeahan
Copy link
Member Author

ebeahan commented Jul 20, 2020

I'm providing a few thoughts here for a follow-up PR. That seemed the simplest way to provide this feedback

Thanks @webmat for the feedback! I hope this helps clear up some of the common confusion with using the tooling.

I'll work on including your feedback/suggestions in a follow-up PR 😄 . I need to update the documented file format for --subset from #873 as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants