Finalize user_agent migration to ECS #10441

webmat · 2019-01-30T20:19:51Z

This PR is built on top of #10472 and should be merged after it.

In 7.0, the output of ingest node "user_agent" processor will match ECS by default (elastic/elasticsearch#37984, elastic/elasticsearch#38094). This means we can remove a lot of temporary field renames that were previously needed, to make its output follow ECS.

This PR will fail CI until the ES Docker image used in CI includes the changes from both issues linked above. Both of them are merged to master already, so we're just waiting for the next successful artifact build for ES.

Filebeat datasets affected:

apache.access
iis.access
nginx.access
suricata.eve (x-pack)
traefik.access
zeek.http (x-pack)

This PR also fixes a few mistakes that were made in ecs-migration.yml.

Caveat

This PR introduces multiple changes / removals of the @timestamp field in the test files because of an unrelated change in how Elasticsearch parses timestamps. This will be fixed in other PRs.

TODO

Get rid of both sets of major/minor/patch fields (ua.* and ua.os.*) in fields.yml
- delete the fields
- delete the aliases to these fields
- ecs-migration.yml entries now refer to the corresponding .version field, with alias: false
Correct a few mistakes in ecs-migration.yml
- Add missing entries
- Correctly point the device field to ua.device.name
- Remove incorrect entries (user_name under user_agent)
- Remove obsolete comment
Reformat each set of entries to be in the same order
Rebase over Remove accidental overwrite of user_agent.device.name by user_agent.device #10472 once merged
Confirm build is green, once CI runs on Elasticsearch that includes this PR Ingest node - user_agent, move device parsing to an object elasticsearch#38115

ruflin · 2019-01-31T14:35:07Z

filebeat/module/apache/access/ingest/default.json

      "ignore_failure": true
    }
-  }, {


Nice to see that some much code can be removed.

webmat · 2019-01-31T14:46:27Z

I'm trying to make this PR work for the user_agent without this fix elastic/elasticsearch#38094.

Not sure if CI incorporates yesterday's change, which causes a mapping error. If it does, I want this PR to be merged to resolve the issue, and I'll do a follow-up PR once the final fix is in, to remove the last rename.

webmat · 2019-01-31T17:46:21Z

The fun meter raises by another level: I just noticed that the ECS definition of user_agent.device.name was imported, but then was later overridden by a "temporary" field definition, defining user_agent.device as keyword...

- Device now aliased to ua.device.name - Version breakdown fields all relate to .version field, with alias:false

… to go.

…test files

ruflin

LGTM. Should we add a changelog entry for this?

For the timestamp changes I would suggest we ignore it for now to make sure CI is green as we already work on fixing it. @ycombinator objections?

ruflin · 2019-02-04T17:29:07Z

filebeat/module/iis/access/test/test-iis-7.5.log-expected.json

        "user_agent.name": "IE",
        "user_agent.original": "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR[ 2.0.50727](tel: 2050727); .NET CLR 3.0.30729)",
-        "user_agent.os.full_name": "Windows 8.1",
-        "user_agent.os.name": "Windows 8.1"
+        "user_agent.os.name": "Windows 8.1",


Is this expected? I see below this goes under full?

ES bug. Will file this soon.

ruflin · 2019-02-04T17:31:48Z

Canceled Windows jobs can be ignored.

ycombinator · 2019-02-04T17:32:45Z

For the timestamp changes I would suggest we ignore it for now to make sure CI is green as we already work on fixing it. @ycombinator objections?

No objections from me. I'm +1 to deal with those changes in separate PR(s).

ruflin · 2019-02-04T18:35:27Z

@webmat @ycombinator Let's make sure that any follow up PR to fix the timestamp changes / fixes the timestamps we have remove / changed here.

webmat · 2019-02-04T18:39:17Z

It's mostly on the ES side, though.

We may proactively be able to circumvent some of parsing problems in our pipelines. But since they're regressions in ts parsing, they should also be fixed in ES.

ycombinator · 2019-02-04T18:56:23Z

We may proactively be able to circumvent some of parsing problems in our pipelines. But since they're regressions in ts parsing, they should also be fixed in ES.

++ to trying to fix on the ES side. I've been trying to adapt our pipelines in a way that would work with . or , as the delimiter between seconds and milliseconds as well as with timestamps that do or do not contain time zone offsets in them. I haven't found a way to make all 4 of those cases work as expected yet. I'll put up a PR to demonstrate the challenges.

webmat · 2019-02-04T19:01:41Z

@ycombinator Yes, even if never merged, would make it easy for me to 👀 .

Otherwise I doubt I'll get to that in time :-\

ycombinator · 2019-02-04T19:21:48Z

@webmat #10543

There was a regression in ES when parsing timezones and we adapted the expected results to the incorrect values in elastic#10441. Revert these changes when the fixed ES images are ready.

There was a regression in ES when parsing timezones and we adapted the expected results to the incorrect values in #10441. Revert these changes when the fixed ES images are ready.

There was a regression in ES when parsing timezones and we adapted the expected results to the incorrect values in elastic#10441. Revert these changes when the fixed ES images are ready. (cherry picked from commit f9be490)

…0678) There was a regression in ES when parsing timezones and we adapted the expected results to the incorrect values in #10441. Revert these changes when the fixed ES images are ready. (cherry picked from commit f9be490)

This adjusts to the updated user_agent parser, which is now in line with ECS schema. - major/minor/patch fields have been removed from the field definitions - multiple field renames in Filebeat module ingest node pipelines are now removed - remove Fb module test skipping which was added due to the latest ES build - delete the `user_agent` field nested in the Suricata EVE events, as it's now an alias - Caveat to this PR: the latest ES build has issues with timestamp parsing, so `@timestamp` is sometimes missing, or ignoring the timezone. This will be resolved in ES, and potentially a separate Beat PR.

webmat requested review from a team as code owners January 30, 2019 20:19

webmat self-assigned this Jan 30, 2019

webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Jan 30, 2019

ruflin mentioned this pull request Jan 30, 2019

Transition Beats to ECS #8655

Closed

webmat changed the title ~~WIP Clean up user_agent field renames, now that its output matches ECS~~ WIP Remove user_agent field renames Jan 30, 2019

webmat force-pushed the ecs-user-agent-cleanup branch from f0c91d4 to 78b36d4 Compare January 31, 2019 13:39

ruflin reviewed Jan 31, 2019

View reviewed changes

webmat changed the title ~~WIP Remove user_agent field renames~~ WIP Finalize user_agent migration to ECS Jan 31, 2019

webmat mentioned this pull request Jan 31, 2019

Remove accidental overwrite of user_agent.device.name by user_agent.device #10472

Merged

webmat force-pushed the ecs-user-agent-cleanup branch from ab773d8 to 7942a0b Compare January 31, 2019 21:19

webmat requested review from a team as code owners January 31, 2019 21:19

webmat added review in progress Pull request is currently in progress. and removed in progress Pull request is currently in progress. review labels Feb 1, 2019

Mathieu Martin added 6 commits February 4, 2019 08:52

Remove unnecessary field renames, after user_agent adjustment to ECS

0c839c9

Remove user agent's old major/minor/patch fields

e373c62

Remove small mistake in ecs-migration

183984e

Remove obsolete comment

192d2d2

Correct inconsistencies in ecs-migration wrt user_agent field migration.

7d3cd94

- Device now aliased to ua.device.name - Version breakdown fields all relate to .version field, with alias:false

Unskip the Filebeat integration tests. User_agent parsing is now good…

3c0af2d

… to go.

webmat force-pushed the ecs-user-agent-cleanup branch from 7942a0b to 3c0af2d Compare February 4, 2019 14:10

webmat requested a review from a team as a code owner February 4, 2019 14:10

webmat mentioned this pull request Feb 4, 2019

Reenable modules tests #10522

Closed

Remove original suricata user_agent field (now alias), update x-pack …

9b418df

…test files

webmat changed the title ~~WIP Finalize user_agent migration to ECS~~ Finalize user_agent migration to ECS Feb 4, 2019

webmat added review and removed in progress Pull request is currently in progress. labels Feb 4, 2019

ruflin approved these changes Feb 4, 2019

View reviewed changes

webmat merged commit 0fd59ee into elastic:master Feb 4, 2019

webmat deleted the ecs-user-agent-cleanup branch February 4, 2019 18:34

jsoriano mentioned this pull request Feb 11, 2019

Revert changes in timezones in filebeat expected results #10663

Merged

This was referenced Feb 11, 2019

@timestamp fields are missing from expected test files in filebeat modules #10668

Closed

Adding back missing @timestamp fields to golden files #10669

Merged

jsoriano mentioned this pull request Feb 11, 2019

Cherry-pick #10663 to 7.0: Revert changes in timezones in filebeat expected results #10678

Merged

ycombinator mentioned this pull request Feb 11, 2019

Cherry-pick #10663 to 7.x: Revert changes in timezones in filebeat expected results #10685

Closed

ycombinator mentioned this pull request Feb 12, 2019

Cherry-pick #10669 to 7.0: Adding back missing @timestamp fields to golden files #10686

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finalize user_agent migration to ECS #10441

Finalize user_agent migration to ECS #10441

webmat commented Jan 30, 2019 •

edited

Loading

ruflin Jan 31, 2019

webmat commented Jan 31, 2019 •

edited

Loading

webmat commented Jan 31, 2019

ruflin left a comment

ruflin Feb 4, 2019

webmat Feb 4, 2019

ruflin commented Feb 4, 2019

ycombinator commented Feb 4, 2019

ruflin commented Feb 4, 2019

webmat commented Feb 4, 2019

ycombinator commented Feb 4, 2019

webmat commented Feb 4, 2019

ycombinator commented Feb 4, 2019

Finalize user_agent migration to ECS #10441

Finalize user_agent migration to ECS #10441

Conversation

webmat commented Jan 30, 2019 • edited Loading

ruflin Jan 31, 2019

Choose a reason for hiding this comment

webmat commented Jan 31, 2019 • edited Loading

webmat commented Jan 31, 2019

ruflin left a comment

Choose a reason for hiding this comment

ruflin Feb 4, 2019

Choose a reason for hiding this comment

webmat Feb 4, 2019

Choose a reason for hiding this comment

ruflin commented Feb 4, 2019

ycombinator commented Feb 4, 2019

ruflin commented Feb 4, 2019

webmat commented Feb 4, 2019

ycombinator commented Feb 4, 2019

webmat commented Feb 4, 2019

ycombinator commented Feb 4, 2019

webmat commented Jan 30, 2019 •

edited

Loading

webmat commented Jan 31, 2019 •

edited

Loading