Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor fields.yml generation #7671

Closed
andrewkroh opened this issue Jul 21, 2018 · 1 comment
Closed

Refactor fields.yml generation #7671

andrewkroh opened this issue Jul 21, 2018 · 1 comment
Assignees
@kvch
Labels
:Infra libbeat

Comments

@andrewkroh
Copy link
Member

andrewkroh commented Jul 21, 2018 •
edited
Loading

This is a request to have tooling around fields.yml modified to make it easier to use.

  • Do not depend on libbeat/fields.yml. Instead, always directly use libbeat/_meta/fields.common.yml and libbeat/processors/**/fields.yml when building a Beat's fields.yml. This greatly simplifies the build logic and reduces the possibilities of a race condition during concurrent builds. It also stops the build from generated new files inside of the vendor directory for community beats.
  • Do not write intermediary files to disk such as _meta/fields.generated.yml. Only produce fields.yml.

Relates to #7670.
@ruflin
Copy link
Contributor

ruflin commented Jul 23, 2018

What I have in mind long term is that all of these end up in fields.go files. The final fields.yml will be built by exporting it from the binary if still needed. Metricbeat now already goes into this direction but more work is needed.

@jalvz jalvz mentioned this issue Jul 24, 2018
Closed
@kvch kvch self-assigned this Jul 24, 2018
@kvch kvch mentioned this issue Jul 24, 2018
Merged
kvch added a commit to kvch/beats that referenced this issue Jul 26, 2018
@kvch
simplify global fields.yml generator script
db65390 
* `libbeat/fields.yml` is not a dependency anymore
* intermediary files are not written

Closes elastic#7671
ruflin pushed a commit that referenced this issue Jul 26, 2018
@kvch @ruflin
Simplify fields.yml generator (#7713)
5797005 
This PR simplifies the process of generating a new `fields.yml` file for a Beat.
* `libbeat/fields.yml` is not a dependency anymore
* only `fields.yml` is written to disk

Changes in the asset files are due to different number of newlines in generated `fields.yml` files.

### Interface changes of generator tools
Both `dev-tools/cmd/asset/asset.go` and `libbeat/scripts/cmd/global_fields/main.go` is changed. The first one has two new flags: `-in`, and `-out`. By default it reads from stdin and writes to stdout. Using the flags it's possible to read and write from other files. The second one has one new flag `-out` which specifies the output file. If it's not set, the output is written to stdout.

This makes it possible to generate asset files for Metricbeat without writing `libbeat/fields.yml` to disk. It will be useful in the future when more Beats have modular assets.

Closes #7671
@kvch kvch mentioned this issue Jul 27, 2018
Merged
kvch added a commit to kvch/beats that referenced this issue Jul 27, 2018
@kvch
Simplify fields.yml generator (elastic#7713)
a8146d5 
This PR simplifies the process of generating a new `fields.yml` file for a Beat.
* `libbeat/fields.yml` is not a dependency anymore
* only `fields.yml` is written to disk

Changes in the asset files are due to different number of newlines in generated `fields.yml` files.

### Interface changes of generator tools
Both `dev-tools/cmd/asset/asset.go` and `libbeat/scripts/cmd/global_fields/main.go` is changed. The first one has two new flags: `-in`, and `-out`. By default it reads from stdin and writes to stdout. Using the flags it's possible to read and write from other files. The second one has one new flag `-out` which specifies the output file. If it's not set, the output is written to stdout.

This makes it possible to generate asset files for Metricbeat without writing `libbeat/fields.yml` to disk. It will be useful in the future when more Beats have modular assets.

Closes elastic#7671
(cherry picked from commit 5797005)
@kvch kvch mentioned this issue Jul 27, 2018
Merged
ruflin pushed a commit that referenced this issue Jul 30, 2018
@kvch @ruflin
Simplify fields.yml generator (#7713) (#7778)
b514530 
This PR simplifies the process of generating a new `fields.yml` file for a Beat.
* `libbeat/fields.yml` is not a dependency anymore
* only `fields.yml` is written to disk

Changes in the asset files are due to different number of newlines in generated `fields.yml` files.

### Interface changes of generator tools
Both `dev-tools/cmd/asset/asset.go` and `libbeat/scripts/cmd/global_fields/main.go` is changed. The first one has two new flags: `-in`, and `-out`. By default it reads from stdin and writes to stdout. Using the flags it's possible to read and write from other files. The second one has one new flag `-out` which specifies the output file. If it's not set, the output is written to stdout.

This makes it possible to generate asset files for Metricbeat without writing `libbeat/fields.yml` to disk. It will be useful in the future when more Beats have modular assets.

Closes #7671
(cherry picked from commit 5797005)
kvch added a commit to kvch/beats that referenced this issue Sep 25, 2018
@kvch
Simplify fields.yml generator (elastic#7713)
5c25fbb 
This PR simplifies the process of generating a new `fields.yml` file for a Beat.
* `libbeat/fields.yml` is not a dependency anymore
* only `fields.yml` is written to disk

Changes in the asset files are due to different number of newlines in generated `fields.yml` files.

### Interface changes of generator tools
Both `dev-tools/cmd/asset/asset.go` and `libbeat/scripts/cmd/global_fields/main.go` is changed. The first one has two new flags: `-in`, and `-out`. By default it reads from stdin and writes to stdout. Using the flags it's possible to read and write from other files. The second one has one new flag `-out` which specifies the output file. If it's not set, the output is written to stdout.

This makes it possible to generate asset files for Metricbeat without writing `libbeat/fields.yml` to disk. It will be useful in the future when more Beats have modular assets.

Closes elastic#7671
(cherry picked from commit 5797005)
kvch added a commit that referenced this issue Sep 25, 2018
@kvch
Cherry-pick #7713 to 6.4: Simplify fields.yml generator (#7777)
357d046 
* Simplify fields.yml generator (#7713)

This PR simplifies the process of generating a new `fields.yml` file for a Beat.
* `libbeat/fields.yml` is not a dependency anymore
* only `fields.yml` is written to disk

Changes in the asset files are due to different number of newlines in generated `fields.yml` files.

### Interface changes of generator tools
Both `dev-tools/cmd/asset/asset.go` and `libbeat/scripts/cmd/global_fields/main.go` is changed. The first one has two new flags: `-in`, and `-out`. By default it reads from stdin and writes to stdout. Using the flags it's possible to read and write from other files. The second one has one new flag `-out` which specifies the output file. If it's not set, the output is written to stdout.

This makes it possible to generate asset files for Metricbeat without writing `libbeat/fields.yml` to disk. It will be useful in the future when more Beats have modular assets.

Closes #7671
(cherry picked from commit 5797005)

* update fields of Packetbeat
leweafan pushed a commit to leweafan/beats that referenced this issue Apr 28, 2023
@kvch
Cherry-pick elastic#7713 to 6.4: Simplify fields.yml generator (elast…
defcc22 
…ic#7777)

* Simplify fields.yml generator (elastic#7713)

This PR simplifies the process of generating a new `fields.yml` file for a Beat.
* `libbeat/fields.yml` is not a dependency anymore
* only `fields.yml` is written to disk

Changes in the asset files are due to different number of newlines in generated `fields.yml` files.

### Interface changes of generator tools
Both `dev-tools/cmd/asset/asset.go` and `libbeat/scripts/cmd/global_fields/main.go` is changed. The first one has two new flags: `-in`, and `-out`. By default it reads from stdin and writes to stdout. Using the flags it's possible to read and write from other files. The second one has one new flag `-out` which specifies the output file. If it's not set, the output is written to stdout.

This makes it possible to generate asset files for Metricbeat without writing `libbeat/fields.yml` to disk. It will be useful in the future when more Beats have modular assets.

Closes elastic#7671
(cherry picked from commit ec2e022)

* update fields of Packetbeat
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kvch kvch

Labels
:Infra libbeat
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ruflin @kvch @andrewkroh