Upgrade to ECS 1.7.0 #22571

andrewstucki · 2020-11-12T14:47:36Z

What does this PR do?

This adds the upcoming ECS 1.7.0 yml file and updates all references to ecs.version -- also, it adds ecs.version data to the beats that were lacking it.

It's currently in draft because the final ECS 1.7.0 release isn't cut yet. But, once it is, if anything has changed, it should just be a matter of re-importing the fields.ecs.yml file and re-running make update.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

andrewstucki · 2020-11-12T14:49:54Z

filebeat/docs/fields.asciidoc

@@ -44305,7 +44305,7 @@ Examples include Beats. Agents may also run on observers. ECS agent.* fields sha
 Extended build information for the agent.
 This field is intended to contain any build information that a data source may provide, no specific formatting is required.

-type: keyword
+type: wildcard


So, I'm not entirely sure how we want to document these -- do we need to change the doc generator to specify that sometimes this is a wildcard (if you're Elastic licensed), sometimes it's a keyword (if you're OSS)? Or should we add some sort of top-level documentation that says "all wildcards are keywords in OSS"?

@andrewkroh @webmat @ebeahan any thoughts?

Yeah we have an issue in ECS for 1.8 to document the "fallback" idea for OSS implementations, and what it means at a high level. We can probably cover the Beats and Logstash implementations there as well.

Perhaps Beats docs can link to that?

We can make sure this page is ready in advance of the Stack 7.11 release.

Just to answer your question a little more directly

do we need to change the doc generator...

I don't think we need to. The "OSS" fallback is not an official variant of ECS, it's a best effort workaround for purely OSS implementations. People could even use that to adopt ECS on 6.x if they have complex setups with many versions. But we're not making tons of efforts to fully support this. For example basic types that don't really have a fallback will simply not be supported.

This may deserve a mention somewhere in the Beats documentation, but I'm not even sure if there's a section about OSS Beats.

One thing's for sure, with tens of fields moving to wildcard, I don't think it's worth adding a blurb about this all over the place. The OSS Beats usage is a small percentage, most people use the default/basic Beats, so I think it would be too much noise about something that doesn't affect most users.

👍 -- also, should we mention that we're using "experimental" features of ECS somewhere in our docs?

Not sure. It's likely something that could flip flop from one release to another. For instance, "multi-user" stage 3 RFC is now merged, and will be part of mainline ECS in 1.8. Wildcard is approved, and it's a matter of time until it's merged as well.

So next release this notice may go away. And the following release, if we have another "stage 2" change we're pushing to adopt aggressively, the notice may need to come back... So I think it could add more confusion than anything.

It would be a fair warning to have though, so I wouldn't oppose if you want to add this.

I do think this deserves a mention in the documentation so that users are aware that OSS beats will automatically be using keyword rather than wildcard. And that the same happens if you're using older, pre-wildcard ES versions.

@dedemorton Do you have a recommendation on where to include such a notice?

This is a tough call. It sounds like saying something in the field description (where it would be the most noticeable) is not a good idea because it would be repetitive (plus our docgen process limits us).

We can put a statement at the top-level page in the field reference for each Beat, but IMO it's likely to be overlooked by users.

When are users most likely to care about the type difference? Will they be able to continue using the pre-built dashboards? Are there any upgrade implications if they decide to move to basic later on? Will there be a log message to let users know that keyword is being used instead of wildcard (and the implications/limitations)?

@dedemorton wildcard is a drop-in replacement in terms of functionality. Everything will continue to work functionally the same (including dashboards). Only the performance characteristics are changing. So everything keeps working, some things faster, some things slower (check out the table in the blog post for a breakdown).

A user upgrading Basic Beats from 7.10 to 7.11 or from Beats OSS to Beats Basic 7.11+ will go through the same non-event. The wildcard and keyword types are compatible and mixed indices being queried together will work flawlessly.

Users will only be able to fully take advantage of the performance improvements enabled by wildcard once the older data has aged out, though.

elasticmachine · 2020-11-12T14:55:21Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #22571 updated
Start Time: 2020-12-04T18:39:53.581+0000
Duration: 60 min 40 sec

Test stats 🧪

Test	Results
Failed	0
Passed	17336
Skipped	1373
Total	18709

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	17336
Skipped	1373
Total	18709

libbeat/_meta/fields.ecs.yml

elasticmachine · 2020-11-12T17:56:28Z

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	16618
Skipped	1349
Total	17967

webmat · 2020-11-17T15:35:23Z

@andrewstucki ECS 1.7 is out. You can go grab the official v1.7.0 artifacts to update the PR :-)

elasticmachine · 2020-11-30T16:11:20Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

webmat · 2020-11-30T16:20:16Z

libbeat/_meta/fields.ecs.yml

@@ -1,5 +1,5 @@
 # WARNING! Do not edit this file directly, it was generated by the ECS project,
-# based on ECS version 1.7.0.
+# based on ECS version 1.7.0+exp.


Pristine file 👍 😉

filebeat/module/apache/access/config/access.yml

webmat · 2020-12-02T19:25:08Z

@andrewstucki We found a problem in the Beats YML file today, relating to the APM and ECS logger "tracing" fields. In Beats they're added under a "tracing" section, which makes them incorrectly appear as "tracing.trace.id", "tracing.transaction.id" and so on, instead of "trace.id" and "transaction.id".

We have a PR to address this issue elastic/ecs#1164 already. I think it's worth waiting for it to be merged and pull this bug fix in, before merging this PR.

webmat · 2020-12-02T21:26:10Z

@andrewstucki Alright, the fix has been merged in ECS branch "1.7". You can go grab it.

Note that the tracing fields are now at the same level in the YAML document as field groups usually are. The field groups generally don't have a dot in their name, only fields defined within them do... So I'm not 100% sure if this will work as is. Please let us know if this causes an issue, we can adjust it further. Sorry I only noticed/ thought about that after it was backported.

webmat

Latest import and trace.id correction LGTM 👍

* Upgrade to ECS 1.7.0 * Update changelog * remove user.effective fields from custom fields and update functionbeat * Run make update (not mage update) on heartbeat * Add in dns.address and re-update all beats * fix x-pack metricbeat host processor * Update to released ECS 1.7 * update to fixed ECS * Fix bad filebeat ECS name * metricset -> fileset * Fix up expected load balancer log * Fix up alb expected log (cherry picked from commit a551b25)

* Upgrade to ECS 1.7.0 (#22571) * Upgrade to ECS 1.7.0 * Update changelog * remove user.effective fields from custom fields and update functionbeat * Run make update (not mage update) on heartbeat * Add in dns.address and re-update all beats * fix x-pack metricbeat host processor * Update to released ECS 1.7 * update to fixed ECS * Fix bad filebeat ECS name * metricset -> fileset * Fix up expected load balancer log * Fix up alb expected log (cherry picked from commit a551b25) * remove unnecessary epr file

Upgrade to ECS 1.7.0

7ca132b

andrewstucki added enhancement ecs Team:Security-External Integrations labels Nov 12, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 12, 2020

andrewstucki commented Nov 12, 2020

View reviewed changes

andrewstucki requested a review from a team November 12, 2020 14:50

Update changelog

4522b26

andrewstucki mentioned this pull request Nov 12, 2020

[ECS] Upgrade modules to 1.7 #21674

Closed

webmat reviewed Nov 12, 2020

View reviewed changes

libbeat/_meta/fields.ecs.yml Outdated Show resolved Hide resolved

Andrew Stucki added 3 commits November 12, 2020 10:30

remove user.effective fields from custom fields and update functionbeat

196c88c

Run make update (not mage update) on heartbeat

9ceae4f

Add in dns.address and re-update all beats

f295d3c

fix x-pack metricbeat host processor

1edc350

simitt mentioned this pull request Nov 19, 2020

Upgrade to ECS 1.7 - support wildcards elastic/apm-server#4448

Closed

Andrew Stucki added 2 commits November 30, 2020 10:59

Merge branch 'master' of github.com:elastic/beats into ecs-1.7

6115b00

Update to released ECS 1.7

faed7ac

andrewstucki marked this pull request as ready for review November 30, 2020 16:11

andrewstucki requested a review from a team as a code owner November 30, 2020 16:11

webmat reviewed Nov 30, 2020

View reviewed changes

webmat mentioned this pull request Dec 2, 2020

Incorrect output of the "tracing" fields in the Beats yml file elastic/ecs#1163

Closed

leehinman reviewed Dec 2, 2020

View reviewed changes

filebeat/module/apache/access/config/access.yml Show resolved Hide resolved

legrego mentioned this pull request Dec 2, 2020

[Filebeat] Add module for Kibana audit logs #22696

Merged

6 tasks

leehinman approved these changes Dec 2, 2020

View reviewed changes

Andrew Stucki added 5 commits December 4, 2020 09:59

update to fixed ECS

7237c0e

Fix bad filebeat ECS name

45254b5

metricset -> fileset

e50ff4f

Fix up expected load balancer log

1d27651

Merge branch 'master' of github.com:elastic/beats into ecs-1.7

c884b5f

andrewstucki force-pushed the ecs-1.7 branch from 73a9c7a to 1d27651 Compare December 4, 2020 17:14

Fix up alb expected log

28108c7

webmat approved these changes Dec 4, 2020

View reviewed changes

andrewstucki merged commit a551b25 into elastic:master Dec 4, 2020

andrewstucki deleted the ecs-1.7 branch December 4, 2020 20:24

andrewstucki mentioned this pull request Dec 4, 2020

Cherry-pick #22571 to 7.x: Upgrade to ECS 1.7.0 #22938

Merged

5 tasks

andrewstucki added the v7.11.0 label Dec 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to ECS 1.7.0 #22571

Upgrade to ECS 1.7.0 #22571

andrewstucki commented Nov 12, 2020 •

edited

Loading

andrewstucki Nov 12, 2020

webmat Nov 12, 2020

webmat Nov 12, 2020 •

edited

Loading

andrewstucki Nov 12, 2020

webmat Nov 12, 2020

andrewkroh Nov 16, 2020 •

edited

Loading

dedemorton Nov 17, 2020

webmat Nov 17, 2020

elasticmachine commented Nov 12, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

elasticmachine commented Nov 12, 2020 •

edited by jenkins-beats-ci bot

Loading

Test stats 🧪

webmat commented Nov 17, 2020

elasticmachine commented Nov 30, 2020

webmat Nov 30, 2020

andrewstucki Nov 30, 2020

webmat commented Dec 2, 2020 •

edited

Loading

webmat commented Dec 2, 2020

webmat left a comment

Upgrade to ECS 1.7.0 #22571

Upgrade to ECS 1.7.0 #22571

Conversation

andrewstucki commented Nov 12, 2020 • edited Loading

What does this PR do?

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat Nov 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh Nov 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Nov 12, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Nov 12, 2020 • edited by jenkins-beats-ci bot Loading

💚 Flaky test report

Test stats 🧪

webmat commented Nov 17, 2020

elasticmachine commented Nov 30, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat commented Dec 2, 2020 • edited Loading

webmat commented Dec 2, 2020

webmat left a comment

Choose a reason for hiding this comment

andrewstucki commented Nov 12, 2020 •

edited

Loading

webmat Nov 12, 2020 •

edited

Loading

andrewkroh Nov 16, 2020 •

edited

Loading

elasticmachine commented Nov 12, 2020 •

edited by jenkins-beats-ci bot

Loading

elasticmachine commented Nov 12, 2020 •

edited by jenkins-beats-ci bot

Loading

webmat commented Dec 2, 2020 •

edited

Loading