Phase 2 in conversion to asciidoc #334

karenzone · 2019-02-20T17:37:45Z

More work toward the move to asciidoc:

Added new files and content.
Played with the content of Base Fields to demonstrate a three-column format.
Watch for possible terminology change from "core and extended" to "core and custom" for fields.

Want to see this rendered? https://ecs-docs.firebaseapp.com/ecs2/

webmat

Love it, thank you!

I left a few notes on things I'd like us to tweak.

This also spurred a bit of discussion. Let's wait for Mike's input on a few of those :-)

webmat · 2019-02-20T18:04:49Z

docs/index.asciidoc

-feedback on the general structure, missing fields, or existing fields is appreciated.
-For contributions please read the  https://github.com/elastic/ecs/blob/master/CONTRIBUTING.md[Contribution Guidelines].
+feedback on the general structure, missing fields, or existing fields is
+appreciated. For contributions please read the


I think for 1.0, we should change the message here.

Here's what we're now shooting for:

We are following Semantic Versioning, meaning:

Backwards compatibility during the 1.x lifetime

Possibility of breaking changes only at major versions

Related: We will align ECS major version with Elastic Stack major versions

At least that's how I think we should approach it. This stability will be a big decision factor for people deciding whether or not to adopt ECS.

WDYT @MikePaquette?

cc @ruflin

Also while not technically a beta anymore, perhaps we should still guard and say "this is the first official version, so consider accordingly".

@webmat I like the message you propose. I would make sure to position these as goals, as there may be unforeseen situations that cause us to change these plans.

I would skip the "first official version" note - I the 1.0.0 version already makes that clear.

webmat · 2019-02-20T18:06:30Z

docs/index.asciidoc

+* *Extended fields.* Any fields that are not a core field. 
+Extended fields may apply to more narrow use cases, or may be more open
+to interpretation depending on the use case. Extended fields are more likely to
+change over time.


Love the very direct and straightforward mention of what core and extended mean, right in the ECS index page ❤️

@MarkSettleES let me know that "core" and "custom" have been proposed instead of "core" and "extended".
WDYT @webmat? @MikePaquette? @MarkSettleES? Others?

Core and extended are absolutely both the main levels we use to define ECS fields.

However Mark makes a good point. I have seen some of Mike's content list the 3 together, and I think that's a great idea. I think it could reinforce the idea that ECS is a standard that still embraces the custom that people need around it.

I agree we should add it as a third bullet point in there.

@karenzone @MarkSettleES to close the loop here, "Custom" can be included as a third level, but is not an alternative to "extended".

So I agree with @webmat let's go with three levels:

Core

Extended

Custom

docs/guidelines.asciidoc

webmat · 2019-02-20T18:14:41Z

docs/glossary.asciidoc

+Describes an implementation with all existing fields mapped to ECS fields.
+
+ECS compliant:: Describes an implementation with all existing fields mapped to
+ECS fields, and with as many ECS core fields populated as is practical.


I think we wanted to remove these two distinctions, right @MikePaquette?

These concepts haven't been fully fleshed out, I would remove for now, and only introduce when we're ready to have these kinds of discussions.

@webmat @karenzone
I agree to remove these definitions for now, but I want to make sure we include guidance like: "In order to increase the level of interoperability of your data with future analysis content, your implementation should attempt to populate as many of the Core fields as practical, even if these fields or values are metadata that are not present in your original event, or are already populated in another ECS field"

I think this guidance is fundamental to the success of a common schema, and one I've not been very successful at articulating.

Simple example: when converting a syslog firewall log event to ECS, there maybe no field in the event with the value "syslog" , but a good ECS implementation will still populate agent.type: syslog so that future ECS-aware analysis content that filters events by agent.type will work properly.

I've taken an additional note to reinforce the docs this way, in the docs plan

docs/fields-gen.asciidoc

docs/conventions.asciidoc

docs/convert.asciidoc

webmat · 2019-02-20T18:26:36Z

docs/convert.asciidoc

+implementations to ECS to help you correlate data across all of your products
+and solutions.
+
+Before you start a conversion, be sure that you understand the basic <link here>.


Let's keep a todo to finish this sentence. Or can we for now just say "the basics outlined below:"?

webmat · 2019-02-20T18:29:20Z

docs/convert.asciidoc

+
+An implementation would be considered *compliant* with ECS only if it also
+populates as many of the ECS core fields as is practical. Duplication, adding
+metadata, or deriving values via manipulation and math are acceptable.


In line with my other comment on compatible/compliant, we should consider removing this section.

webmat · 2019-02-20T18:36:06Z

docs/faq.asciidoc

+* Dot notation: `user.firstname: Nicolas`, `user.lastname: Ruflin`
+* Underline notation: `user_firstname: Nicolas`, `user_lastname: Ruflin`
+
+For ECS we decided to use the dot notation. Here's some background on this decision.


Let's reinforce the meaning of dots here at the introduction of the concept. It's well laid out below for those who will pore over each section. Would this make sense?

Suggested change

For ECS we decided to use the dot notation. Here's some background on this decision.

For ECS we decided to use the dot notation, representing nested objects. Here's some background on this decision.

Trying the new suggest feature for changing v7.1 to v7.0. Co-Authored-By: karenzone <[email protected]>

webmat

Thanks for fixing all of these links. This documentation build script is great! It's great to finally be taken through how to leverage it like this :-)

Just noticed one typo in one of the IDs. Once fixed, we can merge.

I can take care of adjusting, if we decide to add "Other", once Mike weighs in.

webmat · 2019-02-21T16:06:06Z

docs/faq.asciidoc


 [float]
+[[nptation-diff]]


I'd go for "notation-diff" here instead ;-)

Fix typo in anchor

Trying to work Mike's points in, as mentioned [here](elastic#334 (comment)).

* Phase 2 in conversion to asciidoc * Update docs/convert.asciidoc Trying the new suggest feature for changing v7.1 to v7.0. Co-Authored-By: karenzone <[email protected]> * Start incorporating review comments * Update faq.asciidoc Fix typo in anchor

Backport of PR #334 to 1.0 branch. Original message: More work toward the move to asciidoc: * Added new files and content. * Played with the content of Base Fields to demonstrate a three-column format. * Watch for possible terminology change from "core and extended" to "core and custom" for fields.

Phase 2 in conversion to asciidoc

d1b4361

webmat reviewed Feb 20, 2019

View reviewed changes

webmat and others added 3 commits February 20, 2019 17:56

Update docs/convert.asciidoc

867c73e

Trying the new suggest feature for changing v7.1 to v7.0. Co-Authored-By: karenzone <[email protected]>

Start incorporating review comments

d2f2c70

Resolve merge conflicts

bd72410

webmat approved these changes Feb 21, 2019

View reviewed changes

Update faq.asciidoc

f58cf92

Fix typo in anchor

karenzone merged commit 166358d into elastic:master Feb 21, 2019

karenzone deleted the round2 branch February 21, 2019 16:40

karenzone mentioned this pull request Feb 22, 2019

Explain the concepts of core and extended #293

Closed

webmat pushed a commit to webmat/ecs that referenced this pull request Feb 26, 2019

Add a 'how to populate ECS fields' section to readme...

60bcbfe

Trying to work Mike's points in, as mentioned [here](elastic#334 (comment)).

This was referenced Feb 26, 2019

Add a 'how to populate ECS fields' section to readme... #345

Merged

Generator for asciidoc representation of all ECS fields #347

Merged

webmat pushed a commit to webmat/ecs that referenced this pull request Mar 4, 2019

Add a 'how to populate ECS fields' section to readme...

79023c9

Trying to work Mike's points in, as mentioned [here](elastic#334 (comment)).

webmat added needs_backport 1.0.0-ga labels Mar 5, 2019

webmat mentioned this pull request Mar 5, 2019

Backport #334 to 1.0: Phase 2 in conversion to asciidoc (#334) #371

Merged

webmat removed the needs_backport label Mar 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Phase 2 in conversion to asciidoc #334

Phase 2 in conversion to asciidoc #334

karenzone commented Feb 20, 2019 •

edited

Loading

webmat left a comment

webmat Feb 20, 2019

webmat Feb 21, 2019

MikePaquette Feb 22, 2019

webmat Feb 20, 2019

karenzone Feb 20, 2019

webmat Feb 21, 2019

MikePaquette Feb 22, 2019

webmat Feb 20, 2019

MikePaquette Feb 22, 2019

webmat Feb 22, 2019

webmat Feb 20, 2019

webmat Feb 20, 2019

webmat Feb 20, 2019

webmat left a comment

webmat Feb 21, 2019

	For ECS we decided to use the dot notation. Here's some background on this decision.
	For ECS we decided to use the dot notation, representing nested objects. Here's some background on this decision.


		[float]
		[[nptation-diff]]

Phase 2 in conversion to asciidoc #334

Phase 2 in conversion to asciidoc #334

Conversation

karenzone commented Feb 20, 2019 • edited Loading

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karenzone commented Feb 20, 2019 •

edited

Loading