[Apache] Update grok pattern for accepting user-identity #9632

niraj-elastic · 2024-04-17T12:55:37Z

Changes you made on the PR.

The Apache HTTP Server integration currently does not collect identity. Currently - is hard coded in a grok pattern, so whenever any value will be present in log for identity pipeline will fail. In this PR a new field is introduced to fix the existing grok

Related issue

[Apache HTTP Server] Validate GROK failures with remote login name (%l) #9144

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

elasticmachine · 2024-04-17T13:13:12Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

kush-elastic

LGTM!

muthu-mps · 2024-04-18T08:53:27Z

packages/apache/data_stream/access/_dev/test/pipeline/test-access-basic.log-expected.json

@@ -4,6 +4,7 @@
            "@timestamp": "2016-12-26T14:16:29.000Z",
            "apache": {
                "access": {
+                    "identity": "-",


In most cases the identity may not exist and this can only be available when the IdentityCheck flag is set.

Can we drop the identity fields with - (hyphen) values?

Considering that this field will always have hyphen value if the flag is off.

Agree, I thought of doing the same but there are some fields which have same kind of behavior and those fields include - as value. So to maintain consistency across integration i have not dropped -. Still, we can drop - if that seems right. let me know your thoughts.

@niraj-elastic - The main reason to drop this field is that the identity can be either turned on or off most of the users may not be enabling this. If applicable you can make changes to the other fields to make it consistent.

@muthu-mps I think we can not drop all the - values from response since some of them are meaningful. here is one example.

Then lets drop the - value for identity field alone.

packages/apache/data_stream/access/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: muthu-mps <[email protected]>

shmsr

Protocol is ident not identd?

shmsr · 2024-04-29T05:39:50Z

packages/apache/docs/README.md

@@ -19,6 +19,7 @@ Access logs collects the Apache access logs.
 | Field | Description | Type |
 |---|---|---|
 | @timestamp | Event timestamp. | date |
+| apache.access.identity | The user identity associated with the event, as determined by RFC 1413 identd protocol on the client's machine. | keyword |


Suggested change

| apache.access.identity | The user identity associated with the event, as determined by RFC 1413 identd protocol on the client's machine. | keyword |

| apache.access.identity | The user identity associated with the event, as determined by RFC 1413 ident protocol on the client's machine. | keyword |

We took this description from official document here.

Ref:

https://httpd.apache.org/docs/2.4/logs.html#:~:text=client%20determined%20by-,identd,-on%20the%20clients

https://en.wikipedia.org/wiki/Ident_protocol

In this case, the information that is not available is the RFC 1413 identity of the client determined by identd on the clients machine.

Then the sentence is incorrect here in our README. ident is the protocol. identd is a daemon for providing the ident service.

Remove the protocol from the sentence then.

So either make it identd or make it ident protocol. identd internally would use ident protocol only.

shmsr · 2024-04-29T05:40:05Z

packages/apache/data_stream/access/fields/fields.yml

+    - name: identity
+      type: keyword
+      description: |
+        The user identity associated with the event, as determined by RFC 1413 identd protocol on the client's machine.


Suggested change

The user identity associated with the event, as determined by RFC 1413 identd protocol on the client's machine.

The user identity associated with the event, as determined by RFC 1413 ident protocol on the client's machine.

As mentioned above we took this description from official documentation.

muthu-mps

LGTM!

shmsr · 2024-04-29T07:09:13Z

packages/apache/docs/README.md

@@ -19,6 +19,7 @@ Access logs collects the Apache access logs.
 | Field | Description | Type |
 |---|---|---|
 | @timestamp | Event timestamp. | date |
+| apache.access.identity | The RFC 1413 identity of the client determined by identd on the clients machine. | keyword |


elasticmachine · 2024-04-29T07:26:09Z

💚 Build Succeeded

Buildkite Build
Commit: de4fa0d

History

💚 Build #10882 succeeded 759f95c
💚 Build #10824 succeeded 929449d
💚 Build #10527 succeeded 7afce89

cc @niraj-elastic

elastic-sonarqube · 2024-04-29T07:26:12Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-04-29T09:29:36Z

Package apache - 1.17.1 containing this change is available at https://epr.elastic.co/search?package=apache

commit e2a688fbb1c8712ba0cad243713146867ac2f986 Author: milan-elastic <[email protected]> Date: Wed May 1 15:43:52 2024 +0530 Squashed commit of the following: commit a17de73aa84608f67a1baca4c094819b562e42e0 Author: milan-elastic <“[email protected]”> Date: Wed May 1 15:29:41 2024 +0530 Squashed commit of the following: commit fccdb1f83f0048b07df6ee82fbd91ca432c799b9 Author: milan-elastic <[email protected]> Date: Wed May 1 14:58:41 2024 +0530 add global filter on dashboard level for hadoop commit 686e49be78dc980b2f12d365580cb800fd7cf330 Merge: 024d864b4 01201a7 Author: “milan-elastic” <“[email protected]”> Date: Wed May 1 11:38:59 2024 +0530 Merge branch 'main' of github.com:milan-elastic/integrations into mongodb-atlas-database-logs commit 01201a7 Author: Eric Forte <[email protected]> Date: Tue Apr 30 10:46:55 2024 -0400 [Security Rules] Update security rules package to v8.13.5 (elastic#9762) * [Security Rules] Update security rules package to v8.13.5 * Add changelog entry for 8.13.5 --------- Co-authored-by: protectionsmachine <[email protected]> commit c9d1f1b Author: Eric Forte <[email protected]> Date: Tue Apr 30 09:30:30 2024 -0400 [Security Rules] Update security rules package to v8.13.5-beta.1 (elastic#9758) * [Security Rules] Update security rules package to v8.13.5-beta.1 * Add changelog entry for 8.13.5-beta.1 --------- Co-authored-by: protectionsmachine <[email protected]> commit a79f813 Author: Tetiana Kravchenko <[email protected]> Date: Tue Apr 30 11:32:37 2024 +0200 [kubernetes] Remove deprecated fields, add missing status.last_terminated_reason metric (elastic#9736) * remove deprecated fields Signed-off-by: Tetiana Kravchenko <[email protected]> * Update changelog.yml * add missing metric: last_terminated_reason; update description of the status.reason field Signed-off-by: Tetiana Kravchenko <[email protected]> --------- Signed-off-by: Tetiana Kravchenko <[email protected]> commit b1627a3 Author: ShourieG <[email protected]> Date: Tue Apr 30 13:03:29 2024 +0530 [integrations][http_endpoint] - Converted HTTP Endpoint Integration to input type (elastic#9732) * converted http_endpoint to input package type * updated changelog * updated original event in sample event commit 3a9b508 Author: Lalit Satapathy <[email protected]> Date: Tue Apr 30 11:49:09 2024 +0530 Remove separate codeowners for system package kibana paths. (elastic#9731) commit c90e817 Author: Krishna Chaitanya Reddy Burri <[email protected]> Date: Tue Apr 30 11:32:17 2024 +0530 [Crowdstrike,Azure] Fix flaky tests with ECS fields (elastic#9738) * Fix flaky pipeline tests. * `azure.graphactivitylogs`: Add missing ECS field definitions. * `crowdstrike.falcon`: Update `geoip` processor to `destination` instead of `source`. commit ace8fb4 Author: Aliabbas Attarwala <[email protected]> Date: Mon Apr 29 16:37:23 2024 +0530 [O11y][AWS] Rally benchmark `aws.cloudtrail` (elastic#9448) commit d4e4aa4 Author: niraj-elastic <[email protected]> Date: Mon Apr 29 14:45:46 2024 +0530 [Apache] Update grok pattern for accepting user-identity (elastic#9632) * update grok pattern * update changelog * address review comments * address review comments Co-authored-by: muthu-mps <[email protected]> * address review comments * address review comment --------- Co-authored-by: muthu-mps <[email protected]> commit dce5699 Author: Mario Rodriguez Molins <[email protected]> Date: Mon Apr 29 10:33:19 2024 +0200 Enable publishing packages from integrations-publish pipeline (elastic#9712) Enable publishing packages from integrations-publish pipeline, and remove corresponding step from the main pipeline. commit c7bc530 Author: Chema Martínez <[email protected]> Date: Sat Apr 27 08:57:55 2024 +0200 [zscaler_zia] Fix mapping of source.ip and source.nat.ip (elastic#9727) * Fix mapping of source.ip and source.nat.ip * Update changelog * updated web datastream pipeline tests --------- Co-authored-by: Shourie Ganguly <[email protected]> commit 4750ea8 Author: Mario Rodriguez Molins <[email protected]> Date: Fri Apr 26 13:09:53 2024 +0200 [nginx] Update nginx config to listen in ipv6 too (elastic#9720) commit 25b0988 Author: Mario Rodriguez Molins <[email protected]> Date: Fri Apr 26 10:45:03 2024 +0200 [Buildkite] Update filter to use api source (elastic#9717) commit 45327cf Author: Mario Rodriguez Molins <[email protected]> Date: Fri Apr 26 10:13:22 2024 +0200 [Buildkite] Update filter condition to allow just from webhook source (elastic#9714) commit 024d864b49f1dd333529f96e06de6dec15aac703 Author: milan-elastic <[email protected]> Date: Fri Apr 26 13:00:47 2024 +0530 add dashboard level filter for apache tomcat commit 1cb5fad Author: Dan Kortschak <[email protected]> Date: Fri Apr 26 16:23:35 2024 +0930 entityanalytics_ad: new package for Active Directory user collection (elastic#9485) commit 37c598f Author: CarsonHrusovsky <[email protected]> Date: Thu Apr 25 18:13:26 2024 -0500 [BBOT] New integration for Black Lantern Security scanner (elastic#9651) commit d13e474 Author: Mario Rodriguez Molins <[email protected]> Date: Thu Apr 25 11:55:39 2024 +0200 [Buildkite] Skip install package command in serverless builds for some packages (elastic#9686) commit 0c2198b Author: Mario Rodriguez Molins <[email protected]> Date: Thu Apr 25 11:41:42 2024 +0200 [Buildkite] Add retry suffix for logs (elastic#9703) commit d932e79 Author: Simon Kötting <[email protected]> Date: Thu Apr 25 07:35:45 2024 +0200 [Exchange Server] GA of Integration, Add Dashbord Panel Titles & System Tests (elastic#9560) * Add Dashboard Titles * Add Dashboard Titles * Change Version to GA * adjust PR in Changelog * Add System Tests to all datstreams * fix imap system test config * remove Folder structure out of system tests sample logs * Fix mapping * Add convert for inode field * specify numeric_keyword_fields in system tests commit dba2901 Author: Dan Kortschak <[email protected]> Date: Thu Apr 25 10:21:30 2024 +0930 rapid7_insightvm: canonicalize host.name to lower case and map subdomain to host.hostname (elastic#9665) commit 4284262 Author: Panos Koutsovasilis <[email protected]> Date: Wed Apr 24 20:34:13 2024 +0300 fix(fim): add auto option for backend and make it the default one (elastic#9702) commit c563bb3 Author: Panos Koutsovasilis <[email protected]> Date: Wed Apr 24 19:40:04 2024 +0300 [juniper_netscreen]: include log.file.device_id and log.file.inode in base-fields (elastic#9658) * fix(juniper_netscreen): include log.file.device_id and log.file.inode in base-fields.yml * fix(juniper_netscreen): update README.md commit f187d0d Author: Panos Koutsovasilis <[email protected]> Date: Wed Apr 24 19:11:28 2024 +0300 [juniper_junos]: include log.file.device_id and log.file.inode in base-fields (elastic#9657) * fix(juniper_junos): include log.file.device_id and log.file.inode in base-fields.yml * fix(juniper_junos): update README.md

update grok pattern

7a73f07

niraj-elastic requested a review from a team as a code owner April 17, 2024 12:55

niraj-elastic mentioned this pull request Apr 17, 2024

[Apache HTTP Server] Validate GROK failures with remote login name (%l) #9144

Closed

niraj-elastic self-assigned this Apr 17, 2024

update changelog

7afce89

niraj-elastic added the Integration:apache Apache HTTP Server label Apr 17, 2024

niraj-elastic requested review from kush-elastic and muthu-mps April 17, 2024 13:00

kush-elastic approved these changes Apr 18, 2024

View reviewed changes

muthu-mps reviewed Apr 18, 2024

View reviewed changes

address review comments

929449d

niraj-elastic requested a review from muthu-mps April 26, 2024 05:56

muthu-mps reviewed Apr 26, 2024

View reviewed changes

packages/apache/data_stream/access/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

address review comments

759f95c

Co-authored-by: muthu-mps <[email protected]>

shmsr added the bug Something isn't working, use only for issues label Apr 29, 2024

shmsr requested changes Apr 29, 2024

View reviewed changes

niraj-elastic requested review from shmsr and muthu-mps April 29, 2024 05:54

muthu-mps approved these changes Apr 29, 2024

View reviewed changes

address review comments

ca985ba

shmsr reviewed Apr 29, 2024

View reviewed changes

shmsr changed the title ~~[Apache] Update grok pattern~~ [Apache] Update grok pattern for accepting user-identity Apr 29, 2024

address review comment

de4fa0d

shmsr approved these changes Apr 29, 2024

View reviewed changes

niraj-elastic merged commit d4e4aa4 into elastic:main Apr 29, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Apache] Update grok pattern for accepting user-identity #9632

[Apache] Update grok pattern for accepting user-identity #9632

niraj-elastic commented Apr 17, 2024 •

edited by shmsr

Loading

elasticmachine commented Apr 17, 2024 •

edited

Loading

kush-elastic left a comment

muthu-mps Apr 18, 2024

niraj-elastic Apr 19, 2024

muthu-mps Apr 19, 2024

niraj-elastic Apr 23, 2024

muthu-mps Apr 25, 2024

shmsr left a comment

shmsr Apr 29, 2024

niraj-elastic Apr 29, 2024 •

edited

Loading

shmsr Apr 29, 2024 •

edited

Loading

shmsr Apr 29, 2024 •

edited

Loading

shmsr Apr 29, 2024

niraj-elastic Apr 29, 2024

muthu-mps left a comment

shmsr Apr 29, 2024

elasticmachine commented Apr 29, 2024

elastic-sonarqube bot commented Apr 29, 2024

elasticmachine commented Apr 29, 2024

	\| apache.access.identity \| The user identity associated with the event, as determined by RFC 1413 identd protocol on the client's machine. \| keyword \|
	\| apache.access.identity \| The user identity associated with the event, as determined by RFC 1413 ident protocol on the client's machine. \| keyword \|

	The user identity associated with the event, as determined by RFC 1413 identd protocol on the client's machine.
	The user identity associated with the event, as determined by RFC 1413 ident protocol on the client's machine.

	\| apache.access.identity \| The RFC 1413 identity of the client determined by identd on the clients machine. \| keyword \|
	\| apache.access.identity \| The client's identity, as specified in RFC 1413, determined by the identd on the client's machine. \| keyword \|

[Apache] Update grok pattern for accepting user-identity #9632

[Apache] Update grok pattern for accepting user-identity #9632

Conversation

niraj-elastic commented Apr 17, 2024 • edited by shmsr Loading

Changes you made on the PR.

Related issue

Checklist

elasticmachine commented Apr 17, 2024 • edited Loading

🚀 Benchmarks report

kush-elastic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmsr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niraj-elastic Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

shmsr Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

shmsr Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muthu-mps left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Apr 29, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented Apr 29, 2024

Quality Gate passed

elasticmachine commented Apr 29, 2024

niraj-elastic commented Apr 17, 2024 •

edited by shmsr

Loading

elasticmachine commented Apr 17, 2024 •

edited

Loading

niraj-elastic Apr 29, 2024 •

edited

Loading

shmsr Apr 29, 2024 •

edited

Loading

shmsr Apr 29, 2024 •

edited

Loading