[MongoDB Atlas] mongod audit datastream (#9020)

* add mongod audit datastream * MongoDB Atlas integration package with mongod_audit data stream * update dashboard and system test * add forwarded tag * update codeowners * update pr link in changelog * resolve review comments * update groupid description Co-authored-by: Ishleen Kaur <[email protected]> * resolve review comments * make secret parameter true * update readme * update system test * resolve review comments * update description in fields.yml * update redact field value * add more events for pipeline test --------- Co-authored-by: Ishleen Kaur <[email protected]>
elastic · Apr 16, 2024 · 4b43599 · 4b43599
1 parent 7da891c
commit 4b43599
Show file tree

Hide file tree

Showing 19 changed files with 2,479 additions and 23 deletions.
diff --git a/packages/mongodb_atlas/_dev/build/docs/README.md b/packages/mongodb_atlas/_dev/build/docs/README.md
@@ -6,48 +6,52 @@
 
 Use the MongoDB Atlas integration to:
 
-- Collect metrics related to process.
+- Collect MongoDB Audit logs and Process metrics for comprehensive monitoring and analysis.
 - Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights.
 - Set up alerts to minimize Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) by quickly referencing relevant logs during troubleshooting.
 
 ## Data streams
 
-The MongoDB Atlas integration collects metrics.
+The MongoDB Atlas integration collects logs and metrics.
+
+Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by MongoDB Atlas integration is `mongod_audit`.
 
 Metrics give you insight into the statistics of the MongoDB Atlas. The `Metric` data stream collected by the MongoDB Atlas integration is `process` so that the user can monitor and troubleshoot the performance of the MongoDB Atlas instance.
 
 Data streams:
+- `mongod_audit`: The auditing facility allows administrators and users to track system activity for deployments with multiple users and applications. Mongod Audit logs capture events related to database operations such as insertions, updates, deletions, user authentication, etc., occurring within the mongod instances.
 
 - `process` : This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream.
 
 Note:
-- Users can monitor and see the metrics inside the ingested documents for MongoDB Atlas in the `logs-*` index pattern from `Discover`.
+- Users can monitor and see the log inside the ingested documents for MongoDB Atlas in the `logs-*` index pattern from `Discover`.
 
 ## Prerequisites
 
 You can store and search your data using Elasticsearch and visualize and manage it with Kibana. We recommend using our hosted Elasticsearch Service on Elastic Cloud or self-managing the Elastic Stack on your own hardware.
 
 ## Setup
 
-### To collect data from MongoDB Atlas, the following parameters from your MongoDB Atlas instance are required:
+### To collect data from MongoDB Atlas, the following parameters from your MongoDB Atlas instance are required
 
 1. Public Key
 2. Private Key
 3. GroupId
 
-### Steps to obtain Public Key, Private Key and GroupId:
+### Steps to obtain Public Key, Private Key and GroupId
 
 1. Generate programmatic API Keys with project owner permissions using the instructions in the Atlas [documentation](https://www.mongodb.com/docs/atlas/configure-api-access/#create-an-api-key-for-a-project). Then, copy the public key and private key. These serve the same function as a username and API Key respectively.
-2. You can find your Project ID (Group ID) in the Atlas UI. To do this, navigate to your project, click on Settings, and copy the Project ID (Group ID). You can also programmatically find it using the Atlas Admin API or Atlas CLI as described in this Atlas [document](https://www.mongodb.com/docs/atlas/app-services/apps/metadata/#find-a-project-id).
+2. Enable Database Auditing for the Atlas project you want to monitor logs. You can follow the instructions provided in this Atlas [document](https://www.mongodb.com/docs/atlas/database-auditing/#procedure).
+3. You can find your Project ID (Group ID) in the Atlas UI. To do this, navigate to your project, click on Settings, and copy the Project ID (Group ID). You can also programmatically find it using the Atlas Admin API or Atlas CLI as described in this Atlas [document](https://www.mongodb.com/docs/atlas/app-services/apps/metadata/#find-a-project-id).
 
-### Important terms of MongoDB Atlas API:
+### Important terms of MongoDB Atlas API
 
 1. Granularity: Duration that specifies the interval at which Atlas reports the metrics.
 2. Period: Duration over which Atlas reports the metrics.
 
 Note: Both of above attributes can be set by using `period` in configuration parameters.
 
-### Enabling the integration in Elastic:
+### Steps to enable Integration in Elastic
 
 1. In Kibana go to Management > Integrations
 2. In "Search for integrations" search bar, type MongoDB Atlas
@@ -56,6 +60,31 @@ Note: Both of above attributes can be set by using `period` in configuration par
 5. Enter all the necessary configuration parameters, including Public Key, Private Key, and GroupId.
 6. Finally, save the integration.
 
+Note:
+- The `mongod_audit` data stream gathers historical data spanning the previous 30 minutes.
+- Mongod: Mongod is the primary daemon method for the MongoDB system. It helps in handling the data requests, managing the data access, performing background management operations, and other core database operations.
+
+## Troubleshooting
+
+If you encounter an error while ingesting data, it might be due to the data collected over a long time span. Generating a response in such cases may take longer and might cause a request timeout if the `HTTP Client Timeout` parameter is set to a small duration. To avoid this error, it is recommended to adjust the `HTTP Client Timeout` and `Interval` parameters based on the duration of data collection.
+```
+{
+  "error": {
+    "message": "failed eval: net/http: request canceled (Client.Timeout or context cancellation while reading body)"
+  }
+}
+```
+
+## Logs reference
+
+### Mongod Audit
+
+This is the `mongod_audit` data stream. This data stream allows administrators and users to track system activity for deployments with multiple users and applications.
+
+{{event "mongod_audit"}}
+
+{{fields "mongod_audit"}}
+
 ## Metrics reference
 
 ### Process

diff --git a/packages/mongodb_atlas/_dev/deploy/docker/Dockerfile b/packages/mongodb_atlas/_dev/deploy/docker/Dockerfile
@@ -1,6 +1,6 @@
 FROM golang:1.19
 
-COPY ./test /go
+COPY ./mongodb_atlas /go
 CMD ["./test"]
 
 HEALTHCHECK --interval=1s --retries=90 CMD curl localhost:7780/api/atlas/v2/groups/mongodb-group1/processes --digest -u admin:MongoDB@123
diff --git a/packages/mongodb_atlas/_dev/deploy/docker/mongodb_atlas/mongod_audit_data.log b/packages/mongodb_atlas/_dev/deploy/docker/mongodb_atlas/mongod_audit_data.log
@@ -0,0 +1,5 @@
+{ "atype": "authenticate", "ts": { "$date": "2023-04-01T12:00:00.000Z" }, "uuid": { "$binary": "some-unique-identifier", "$type": "04" }, "local": { "ip": "127.0.0.1", "port": 27017 }, "remote": { "ip": "192.168.1.100", "port": 54320 }, "users": [{ "user": "auditUser", "db": "admin" }], "roles": [{ "role": "dbAdmin", "db": "admin" }], "result": 0 }
+{ "atype": "authCheck", "ts": { "$date": "2023-04-01T12:05:00.000Z" }, "uuid": { "$binary": "another-unique-identifier", "$type": "04" }, "local": { "ip": "127.0.0.1", "port": 27017 }, "remote": { "ip": "192.168.1.101", "port": 54321 }, "users": [{ "user": "userTest", "db": "test" }], "roles": [{ "role": "read", "db": "test" }], "result": 13 }
+{ "atype": "createIndex", "ts": { "$date": "2023-04-01T12:10:00.000Z" }, "uuid": { "$binary": "yet-another-unique-identifier", "$type": "04" }, "local": { "ip": "127.0.0.1", "port": 27017 }, "remote": { "ip": "192.168.1.102", "port": 54322 }, "users": [{ "user": "indexManager", "db": "test" }], "roles": [{ "role": "dbOwner", "db": "test" }], "result": 0 }
+{ "atype": "dropCollection", "ts": { "$date": "2023-04-01T12:15:00.000Z" }, "uuid": { "$binary": "unique-identifier-drop-coll", "$type": "04" }, "local": { "ip": "127.0.0.1", "port": 27017 }, "remote": { "ip": "192.168.1.103", "port": 54323 }, "users": [{ "user": "adminUser", "db": "test" }], "roles": [{ "role": "dbAdmin", "db": "test" }], "result": 0 }
+{ "atype": "createUser", "ts": { "$date": "2023-04-01T12:20:00.000Z" }, "uuid": { "$binary": "unique-identifier-create-user", "$type": "04" }, "local": { "ip": "127.0.0.1", "port": 27017 }, "remote": { "ip": "192.168.1.104", "port": 54324 }, "users": [{ "user": "admin", "db": "admin" }], "roles": [{ "role": "userAdmin", "db": "admin" }], "result": 0 }
diff --git a/...ges/mongodb_atlas/_dev/deploy/docker/test → ...las/_dev/deploy/docker/mongodb_atlas/test b/...ges/mongodb_atlas/_dev/deploy/docker/test → ...las/_dev/deploy/docker/mongodb_atlas/test
diff --git a/packages/mongodb_atlas/changelog.yml b/packages/mongodb_atlas/changelog.yml
@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "0.0.2"
+  changes:
+    - description: MongoDB Atlas integration package with "mongod_audit" data stream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/9020
 - version: "0.0.1"
   changes:
     - description: MongoDB Atlas integration package with "process" data stream.

diff --git a/packages/mongodb_atlas/data_stream/mongod_audit/_dev/test/pipeline/test-common-config.yml b/packages/mongodb_atlas/data_stream/mongod_audit/_dev/test/pipeline/test-common-config.yml
@@ -0,0 +1,5 @@
+dynamic_fields:
+  "event.ingested": ".*"
+fields:
+  tags:
+    - preserve_original_event
diff --git a/packages/mongodb_atlas/data_stream/mongod_audit/_dev/test/pipeline/test-mongod-audit.json b/packages/mongodb_atlas/data_stream/mongod_audit/_dev/test/pipeline/test-mongod-audit.json
@@ -0,0 +1,20 @@
+{
+    "events": [
+        {
+            "@timestamp": "2024-02-08T06:20:49.729Z",
+            "message": "No data for given time period or host is unreachable"
+        },
+        {
+            "@timestamp": "2024-02-08T06:20:56.621Z",
+            "message": "{ \"atype\": \"logout\", \"ts\": { \"$date\": \"2024-01-29T06:57:15.366+00:00\" }, \"uuid\": { \"$binary\": \"bY/PMV8IR36q+hmAJZYyfw==\", \"$type\": \"04\" }, \"local\": { \"ip\": \"127.0.0.1\", \"port\": 27017 }, \"remote\": { \"ip\": \"127.0.0.1\", \"port\": 43714 }, \"users\":[ { \"user\":\"mms-monitoring-agent\", \"db\":\"admin\" } ], \"roles\": [ { \"role\": \"backup\", \"db\": \"admin\" }, { \"role\": \"clusterAdmin\", \"db\": \"admin\" }, { \"role\": \"dbAdminAnyDatabase\", \"db\": \"admin\" }, { \"role\": \"readWriteAnyDatabase\", \"db\": \"admin\" }, { \"role\": \"restore\", \"db\": \"admin\" }, { \"role\": \"userAdminAnyDatabase\", \"db\": \"admin\" } ], \"result\": 0 }"
+        },
+        {
+            "@timestamp": "2024-02-08T07:20:00.123Z",
+            "message": "{\"atype\":\"authenticate\",\"ts\":{\"$date\":\"2023-04-01T12:00:00.000Z\"},\"uuid\":{\"$binary\":\"some-unique-identifier\",\"$type\":\"04\"},\"local\":{\"ip\":\"127.0.0.1\",\"port\":27017},\"remote\":{\"ip\":\"192.168.1.100\",\"port\":54320},\"users\":[{\"user\":\"auditUser\",\"db\":\"admin\"}],\"roles\":[{\"role\":\"dbAdmin\",\"db\":\"admin\"}],\"result\":0}"
+        },
+        {
+            "@timestamp": "2024-02-08T07:30:56.234Z",
+            "message": "{\"atype\":\"authCheck\",\"ts\":{\"$date\":\"2023-04-01T12:05:00.000Z\"},\"uuid\":{\"$binary\":\"another-unique-identifier\",\"$type\":\"04\"},\"local\":{\"ip\":\"127.0.0.1\",\"port\":27017},\"remote\":{\"ip\":\"192.168.1.101\",\"port\":54321},\"users\":[{\"user\":\"userTest\",\"db\":\"test\"}],\"roles\":[{\"role\":\"read\",\"db\":\"test\"}],\"result\":13}"
+        }
+    ]
+}
diff --git a/...db_atlas/data_stream/mongod_audit/_dev/test/pipeline/test-mongod-audit.json-expected.json b/...db_atlas/data_stream/mongod_audit/_dev/test/pipeline/test-mongod-audit.json-expected.json
@@ -0,0 +1,204 @@
+{
+    "expected": [
+        null,
+        {
+            "@timestamp": "2024-01-29T06:57:15.366Z",
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "action": "logout",
+                "category": [
+                    "network",
+                    "authentication"
+                ],
+                "kind": "event",
+                "module": "mongodb_atlas",
+                "original": "{ \"atype\": \"logout\", \"ts\": { \"$date\": \"2024-01-29T06:57:15.366+00:00\" }, \"uuid\": { \"$binary\": \"bY/PMV8IR36q+hmAJZYyfw==\", \"$type\": \"04\" }, \"local\": { \"ip\": \"127.0.0.1\", \"port\": 27017 }, \"remote\": { \"ip\": \"127.0.0.1\", \"port\": 43714 }, \"users\":[ { \"user\":\"mms-monitoring-agent\", \"db\":\"admin\" } ], \"roles\": [ { \"role\": \"backup\", \"db\": \"admin\" }, { \"role\": \"clusterAdmin\", \"db\": \"admin\" }, { \"role\": \"dbAdminAnyDatabase\", \"db\": \"admin\" }, { \"role\": \"readWriteAnyDatabase\", \"db\": \"admin\" }, { \"role\": \"restore\", \"db\": \"admin\" }, { \"role\": \"userAdminAnyDatabase\", \"db\": \"admin\" } ], \"result\": 0 }",
+                "type": [
+                    "access",
+                    "info"
+                ]
+            },
+            "mongodb_atlas": {
+                "mongod_audit": {
+                    "local": {
+                        "ip": "127.0.0.1",
+                        "port": 27017
+                    },
+                    "remote": {
+                        "ip": "127.0.0.1",
+                        "port": 43714
+                    },
+                    "result": "Success",
+                    "user": {
+                        "names": [
+                            {
+                                "db": "admin",
+                                "user": "mms-monitoring-agent"
+                            }
+                        ],
+                        "roles": [
+                            {
+                                "db": "admin",
+                                "role": "backup"
+                            },
+                            {
+                                "db": "admin",
+                                "role": "clusterAdmin"
+                            },
+                            {
+                                "db": "admin",
+                                "role": "dbAdminAnyDatabase"
+                            },
+                            {
+                                "db": "admin",
+                                "role": "readWriteAnyDatabase"
+                            },
+                            {
+                                "db": "admin",
+                                "role": "restore"
+                            },
+                            {
+                                "db": "admin",
+                                "role": "userAdminAnyDatabase"
+                            }
+                        ]
+                    },
+                    "uuid": {
+                        "binary": "bY/PMV8IR36q+hmAJZYyfw==",
+                        "type": "04"
+                    }
+                }
+            },
+            "related": {
+                "ip": [
+                    "127.0.0.1"
+                ]
+            },
+            "tags": [
+                "preserve_original_event"
+            ]
+        },
+        {
+            "@timestamp": "2023-04-01T12:00:00.000Z",
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "action": "authenticate",
+                "category": [
+                    "network",
+                    "authentication"
+                ],
+                "kind": "event",
+                "module": "mongodb_atlas",
+                "original": "{\"atype\":\"authenticate\",\"ts\":{\"$date\":\"2023-04-01T12:00:00.000Z\"},\"uuid\":{\"$binary\":\"some-unique-identifier\",\"$type\":\"04\"},\"local\":{\"ip\":\"127.0.0.1\",\"port\":27017},\"remote\":{\"ip\":\"192.168.1.100\",\"port\":54320},\"users\":[{\"user\":\"auditUser\",\"db\":\"admin\"}],\"roles\":[{\"role\":\"dbAdmin\",\"db\":\"admin\"}],\"result\":0}",
+                "type": [
+                    "access",
+                    "info"
+                ]
+            },
+            "mongodb_atlas": {
+                "mongod_audit": {
+                    "local": {
+                        "ip": "127.0.0.1",
+                        "port": 27017
+                    },
+                    "remote": {
+                        "ip": "192.168.1.100",
+                        "port": 54320
+                    },
+                    "result": "Success",
+                    "user": {
+                        "names": [
+                            {
+                                "db": "admin",
+                                "user": "auditUser"
+                            }
+                        ],
+                        "roles": [
+                            {
+                                "db": "admin",
+                                "role": "dbAdmin"
+                            }
+                        ]
+                    },
+                    "uuid": {
+                        "binary": "some-unique-identifier",
+                        "type": "04"
+                    }
+                }
+            },
+            "related": {
+                "ip": [
+                    "127.0.0.1",
+                    "192.168.1.100"
+                ]
+            },
+            "tags": [
+                "preserve_original_event"
+            ]
+        },
+        {
+            "@timestamp": "2023-04-01T12:05:00.000Z",
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "action": "authCheck",
+                "category": [
+                    "network",
+                    "authentication"
+                ],
+                "kind": "event",
+                "module": "mongodb_atlas",
+                "original": "{\"atype\":\"authCheck\",\"ts\":{\"$date\":\"2023-04-01T12:05:00.000Z\"},\"uuid\":{\"$binary\":\"another-unique-identifier\",\"$type\":\"04\"},\"local\":{\"ip\":\"127.0.0.1\",\"port\":27017},\"remote\":{\"ip\":\"192.168.1.101\",\"port\":54321},\"users\":[{\"user\":\"userTest\",\"db\":\"test\"}],\"roles\":[{\"role\":\"read\",\"db\":\"test\"}],\"result\":13}",
+                "type": [
+                    "access",
+                    "info"
+                ]
+            },
+            "mongodb_atlas": {
+                "mongod_audit": {
+                    "local": {
+                        "ip": "127.0.0.1",
+                        "port": 27017
+                    },
+                    "remote": {
+                        "ip": "192.168.1.101",
+                        "port": 54321
+                    },
+                    "result": "Unauthorized to perform the operation",
+                    "user": {
+                        "names": [
+                            {
+                                "db": "test",
+                                "user": "userTest"
+                            }
+                        ],
+                        "roles": [
+                            {
+                                "db": "test",
+                                "role": "read"
+                            }
+                        ]
+                    },
+                    "uuid": {
+                        "binary": "another-unique-identifier",
+                        "type": "04"
+                    }
+                }
+            },
+            "related": {
+                "ip": [
+                    "127.0.0.1",
+                    "192.168.1.101"
+                ]
+            },
+            "tags": [
+                "preserve_original_event"
+            ]
+        }
+    ]
+}
diff --git a/packages/mongodb_atlas/data_stream/mongod_audit/_dev/test/system/test-default-config.yml b/packages/mongodb_atlas/data_stream/mongod_audit/_dev/test/system/test-default-config.yml
@@ -0,0 +1,13 @@
+vars:
+  url:
+    - http://{{Hostname}}:{{Port}}
+  public_key:
+    - admin
+  private_key:
+    - MongoDB@123
+data_stream:
+  vars:
+    groupId:
+      - mongodb-group1
+input: cel
+service: mongodbatlas