Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Apache] Update grok pattern for accepting user-identity #9632

Merged
merged 6 commits into from
Apr 29, 2024
Merged

[Apache] Update grok pattern for accepting user-identity #9632

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7a73f07
update grok pattern
niraj-elastic Apr 17, 2024
7afce89
update changelog
niraj-elastic Apr 17, 2024
929449d
address review comments
niraj-elastic Apr 26, 2024
759f95c
address review comments
niraj-elastic Apr 29, 2024
ca985ba
address review comments
niraj-elastic Apr 29, 2024
de4fa0d
address review comment
niraj-elastic Apr 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/apache/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.17.1"
changes:
- description: Update grok for accepting user-identity.
type: bugfix
link: https://github.com/elastic/integrations/pull/9632
- version: "1.17.0"
changes:
- description: Limit request tracer log count to five.
Expand Down
3 changes: 2 additions & 1 deletion packages/apache/data_stream/access/_dev/test/pipeline/test-access-basic.log
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ monitoring-server - - [29/May/2017:19:02:48 +0000] "GET /status HTTP/1.1" 200 61
monitoring-server - - [29/May/2017:19:02:48 +0000] "GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" X-Forwarded-For="-"
89.160.20.112 - - [29/May/2017:19:02:48 +0000] "GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" X-Forwarded-For="10.0.0.2,10.0.0.1"
2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6 - - [29/May/2017:19:02:48 +0000] "GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2" X-Forwarded-For="10.225.192.17, 10.2.2.121"
monitoring-server - - [17/May/2022:21:41:43 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.79.1" X-Forwarded-For="192.168.0.2"
monitoring-server - - [17/May/2022:21:41:43 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.79.1" X-Forwarded-For="192.168.0.2"
127.0.0.1 user-identity frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
79 changes: 69 additions & 10 deletions packages/apache/data_stream/access/_dev/test/pipeline/test-access-basic.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"@timestamp": "2016-12-26T14:16:29.000Z",
"apache": {
"access": {
"identity": "-",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • In most cases the identity may not exist and this can only be available when the IdentityCheck flag is set.
  • Can we drop the identity fields with - (hyphen) values?
  • Considering that this field will always have hyphen value if the flag is off.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, I thought of doing the same but there are some fields which have same kind of behavior and those fields include - as value. So to maintain consistency across integration i have not dropped -. Still, we can drop - if that seems right. let me know your thoughts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@niraj-elastic - The main reason to drop this field is that the identity can be either turned on or off most of the users may not be enabling this. If applicable you can make changes to the other fields to make it consistent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muthu-mps I think we can not drop all the - values from response since some of them are meaningful. here is one example.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then lets drop the - value for identity field alone.

"remote_addresses": [
"::1"
]
Expand All @@ -15,7 +16,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409634501Z",
"ingested": "2024-04-09T06:11:58.419585881Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:29 +0200] \"GET /favicon.ico HTTP/1.1\" 404 209",
"outcome": "failure"
Expand Down Expand Up @@ -52,6 +53,7 @@
"@timestamp": "2016-12-26T16:22:13.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"192.168.33.1"
]
Expand All @@ -63,7 +65,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409644668Z",
"ingested": "2024-04-09T06:11:58.419621774Z",
"kind": "event",
"original": "192.168.33.1 - - [26/Dec/2016:16:22:13 +0000] \"GET /hello HTTP/1.1\" 404 499 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0\"",
"outcome": "failure"
Expand Down Expand Up @@ -113,6 +115,7 @@
"@timestamp": "2016-12-26T14:16:48.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"::1"
]
Expand All @@ -124,7 +127,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409645876Z",
"ingested": "2024-04-09T06:11:58.419628526Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:48 +0200] \"-\" 408 -",
"outcome": "failure"
Expand All @@ -149,6 +152,7 @@
"@timestamp": "2017-05-29T19:02:48.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"172.17.0.1"
]
Expand All @@ -160,7 +164,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409646876Z",
"ingested": "2024-04-09T06:11:58.419633501Z",
"kind": "event",
"original": "172.17.0.1 - - [29/May/2017:19:02:48 +0000] \"GET /stringpatch HTTP/1.1\" 404 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" \"-\"",
"outcome": "failure"
Expand Down Expand Up @@ -210,6 +214,7 @@
"@timestamp": "2017-05-29T19:02:48.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"monitoring-server"
]
Expand All @@ -221,7 +226,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409647793Z",
"ingested": "2024-04-09T06:11:58.419638188Z",
"kind": "event",
"original": "monitoring-server - - [29/May/2017:19:02:48 +0000] \"GET /status HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" \"-\"",
"outcome": "success"
Expand Down Expand Up @@ -271,6 +276,7 @@
"@timestamp": "2019-02-02T04:38:45.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"127.0.0.1"
]
Expand All @@ -282,7 +288,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409648793Z",
"ingested": "2024-04-09T06:11:58.419642845Z",
"kind": "event",
"original": "127.0.0.1 - - [02/Feb/2019:05:38:45 +0100] \"-\" 408 152 \"-\" \"-\"",
"outcome": "failure"
Expand Down Expand Up @@ -320,6 +326,7 @@
"@timestamp": "2017-05-29T19:02:48.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"monitoring-server"
]
Expand All @@ -331,7 +338,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409649793Z",
"ingested": "2024-04-09T06:11:58.419647444Z",
"kind": "event",
"original": "monitoring-server - - [29/May/2017:19:02:48 +0000] \"GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" X-Forwarded-For=\"-\"",
"outcome": "success"
Expand Down Expand Up @@ -382,6 +389,7 @@
"@timestamp": "2017-05-29T19:02:48.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"10.0.0.2",
"10.0.0.1",
Expand All @@ -398,7 +406,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409650668Z",
"ingested": "2024-04-09T06:11:58.419652022Z",
"kind": "event",
"original": "89.160.20.112 - - [29/May/2017:19:02:48 +0000] \"GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" X-Forwarded-For=\"10.0.0.2,10.0.0.1\"",
"outcome": "success"
Expand Down Expand Up @@ -470,6 +478,7 @@
"@timestamp": "2017-05-29T19:02:48.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"10.225.192.17",
"10.2.2.121",
Expand All @@ -486,7 +495,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409651543Z",
"ingested": "2024-04-09T06:11:58.419656560Z",
"kind": "event",
"original": "2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6 - - [29/May/2017:19:02:48 +0000] \"GET /A%20Beka%20G1%20Howe/029_AND_30/15%20reading%20elephants.mp4 HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2\" X-Forwarded-For=\"10.225.192.17, 10.2.2.121\"",
"outcome": "success"
Expand Down Expand Up @@ -549,6 +558,7 @@
"@timestamp": "2022-05-17T21:41:43.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"192.168.0.2",
"monitoring-server"
Expand All @@ -564,7 +574,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.409652876Z",
"ingested": "2024-04-09T06:11:58.419661112Z",
"kind": "event",
"original": "monitoring-server - - [17/May/2022:21:41:43 +0000] \"GET / HTTP/1.1\" 200 45 \"-\" \"curl/7.79.1\" X-Forwarded-For=\"192.168.0.2\"",
"outcome": "success"
Expand Down Expand Up @@ -607,6 +617,55 @@
"original": "curl/7.79.1",
"version": "7.79.1"
}
},
{
"@timestamp": "2000-10-10T20:55:36.000Z",
"apache": {
"access": {
"identity": "user-identity",
"remote_addresses": [
"127.0.0.1"
]
}
},
"ecs": {
"version": "8.5.1"
},
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2024-04-09T06:11:58.419665782Z",
"kind": "event",
"original": "127.0.0.1 user-identity frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
"outcome": "success"
},
"http": {
"request": {
"method": "GET"
},
"response": {
"body": {
"bytes": 2326
},
"status_code": 200
},
"version": "1.0"
},
"source": {
"address": "127.0.0.1",
"ip": "127.0.0.1"
},
"tags": [
"preserve_original_event"
],
"url": {
"extension": "gif",
"original": "/apache_pb.gif",
"path": "/apache_pb.gif"
},
"user": {
"name": "frank"
}
}
]
}
18 changes: 12 additions & 6 deletions packages/apache/data_stream/access/_dev/test/pipeline/test-access-darwin.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"@timestamp": "2016-12-26T14:16:28.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"::1"
]
Expand All @@ -15,7 +16,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.483539043Z",
"ingested": "2024-04-09T06:11:58.640870130Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:28 +0200] \"GET / HTTP/1.1\" 200 45",
"outcome": "success"
Expand Down Expand Up @@ -51,6 +52,7 @@
"@timestamp": "2016-12-26T14:16:29.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"::1"
]
Expand All @@ -62,7 +64,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.483550209Z",
"ingested": "2024-04-09T06:11:58.640922672Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:29 +0200] \"GET /favicon.ico HTTP/1.1\" 404 209",
"outcome": "failure"
Expand Down Expand Up @@ -99,6 +101,7 @@
"@timestamp": "2016-12-26T14:16:48.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"::1"
]
Expand All @@ -110,7 +113,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.483551501Z",
"ingested": "2024-04-09T06:11:58.640930188Z",
"kind": "event",
"original": "::1 - - [26/Dec/2016:16:16:48 +0200] \"-\" 408 -",
"outcome": "failure"
Expand All @@ -135,6 +138,7 @@
"@timestamp": "2016-12-26T16:23:35.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"89.160.20.156"
]
Expand All @@ -146,7 +150,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.483552501Z",
"ingested": "2024-04-09T06:11:58.640936194Z",
"kind": "event",
"original": "89.160.20.156 - - [26/Dec/2016:18:23:35 +0200] \"GET / HTTP/1.1\" 200 45",
"outcome": "success"
Expand Down Expand Up @@ -200,6 +204,7 @@
"@timestamp": "2016-12-26T16:23:41.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"89.160.20.156"
]
Expand All @@ -211,7 +216,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.483553418Z",
"ingested": "2024-04-09T06:11:58.640941558Z",
"kind": "event",
"original": "89.160.20.156 - - [26/Dec/2016:18:23:41 +0200] \"GET /notfound HTTP/1.1\" 404 206",
"outcome": "failure"
Expand Down Expand Up @@ -265,6 +270,7 @@
"@timestamp": "2016-12-26T16:23:45.000Z",
"apache": {
"access": {
"identity": "-",
"remote_addresses": [
"89.160.20.156"
]
Expand All @@ -276,7 +282,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.483554501Z",
"ingested": "2024-04-09T06:11:58.640946731Z",
"kind": "event",
"original": "89.160.20.156 - - [26/Dec/2016:18:23:45 +0200] \"GET /hmm HTTP/1.1\" 404 201",
"outcome": "failure"
Expand Down
4 changes: 2 additions & 2 deletions ...es/apache/data_stream/access/_dev/test/pipeline/test-access-ssl-request.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.533303168Z",
"ingested": "2024-04-09T06:11:58.788662911Z",
"kind": "event",
"original": "[10/Aug/2018:09:45:56 +0200] 172.30.0.119 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 \"GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax\u0026amp;opts=%7B%22func%22%3A%22get_admin_tasks_html%22%2C%22args%22%3A%22%22%7D\u0026amp;nsp=b5c7d5d4b6f7d0cf0c92f9cbdf737f6a5c838218425e6ae21 HTTP/1.1\" 1375"
},
Expand Down Expand Up @@ -72,7 +72,7 @@
"event": {
"category": "web",
"created": "2020-04-28T11:07:58.223Z",
"ingested": "2022-12-08T15:09:52.533318376Z",
"ingested": "2024-04-09T06:11:58.788703961Z",
"kind": "event",
"original": "[16/Oct/2019:11:53:47 +0200] 89.160.20.156 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 \"GET /appl/ajaxhelper.php?cmd=getxicoreajax\u0026opts=%7B%22func%22%3A%22get_pagetop_alert_content_html%22%2C%22args%22%3A%22%22%7D\u0026nsp=c2700eab9797eda8a9f65a3ab17a6adbceccd60a6cca7708650a5923950d HTTP/1.1\" -"
},
Expand Down
Loading