[MongoDB Atlas] Add support of mongodb database datastream

.github/CODEOWNERS

@@ -236,6 +236,7 @@
 /packages/mimecast @elastic/security-service-integrations
 /packages/modsecurity @elastic/sec-deployment-and-devices
 /packages/mongodb @elastic/obs-infraobs-integrations
+/packages/mongodb_atlas @elastic/obs-infraobs-integrations
 /packages/mysql @elastic/obs-infraobs-integrations
 /packages/mysql_enterprise @elastic/sec-windows-platform
 /packages/nagios_xi @elastic/obs-infraobs-integrations

packages/mongodb_atlas/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]

packages/mongodb_atlas/_dev/build/docs/README.md

@@ -0,0 +1,75 @@
+# MongoDB Atlas Integration
+
+## Overview
+
+[MongoDB Atlas](https://www.mongodb.com/atlas), the leading multi-cloud developer data platform, offers the easiest way to run MongoDB, enabling you to work with your code's objects directly through its document-based data model, which allows for flexible schema and easy scalability.
+
+Use the MongoDB Atlas integration to:
+
+- Collect MongoDB database logs for comprehensive monitoring and analysis.
+- Create informative visualizations to track usage trends, measure key metrics, and derive actionable business insights.
+- Set up alerts to minimize Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) by quickly referencing relevant logs during troubleshooting.
+
+## Data streams
+
+The MongoDB Atlas integration collects logs.
+
+Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by MongoDB Atlas integration is `mongod_database`.
+
+Data stream:
+- `mongod_database`: This datastream collects a running log of events, including entries such as incoming connections, commands run, and issues encountered. Generally, database log messages are useful for diagnosing issues, monitoring your deployment, and tuning performance.
+
+Note:
+- Users can monitor and see the log inside the ingested documents for MongoDB Atlas in the `logs-*` index pattern from `Discover`.
+
+## Prerequisites
+
+You can store and search your data using Elasticsearch and visualize and manage it with Kibana. We recommend using our hosted Elasticsearch Service on Elastic Cloud or self-managing the Elastic Stack on your own hardware.
+
+## Setup
+
+### To collect data from MongoDB Atlas, the following parameters from your MongoDB Atlas instance are required:
+
+1. Public Key
+2. Private Key
+3. GroupId
+
+### Steps to obtain Public Key, Private Key and GroupId:
+
+1. Generate programmatic API Keys with project owner permissions using the instructions in the Atlas [documentation](https://www.mongodb.com/docs/atlas/configure-api-access/#create-an-api-key-for-a-project). Then, copy the public key and private key. These serve the same function as a username and API Key respectively.
+2. Enable Database Auditing for the Atlas project you want to monitor logs. You can follow the instructions provided in this Atlas [document](https://www.mongodb.com/docs/atlas/database-auditing/#procedure).
+3. You can find your Project ID (Group ID) in the Atlas UI. To do this, navigate to your project, click on Settings, and copy the Project ID (Group ID). You can also programmatically find it using the Atlas Admin API or Atlas CLI as described in this Atlas [document](https://www.mongodb.com/docs/atlas/app-services/apps/metadata/#find-a-project-id).
+
+### Steps to enable Integration in Elastic
+
+1. In Kibana go to Management > Integrations
+2. In "Search for integrations" search bar, type MongoDB Atlas
+3. Click on the "MongoDB Atlas" integration from the search results.
+4. To add the integration, click on the "Add MongoDB Atlas" button.
+5. Enter all the necessary configuration parameters, including Public Key, Private Key, and GroupId.
+6. Finally, save the integration.
+
+Note:
+- The `mongod_database` data streams gather historical data spanning the previous 30 minutes.
+- Mongod: Mongod is the primary daemon method for the MongoDB system. It helps in handling the data requests, managing the data access, performing background management operations, and other core database operations.
+
+## Troubleshooting
+
+If you encounter an error while ingesting data, it might be due to the data collected over a long time span. Generating a response in such cases may take longer and might cause a request timeout if the `HTTP Client Timeout` parameter is set to a small duration. To avoid this error, it is recommended to adjust the `HTTP Client Timeout` and `Interval` parameters based on the duration of data collection.
+```
+{
+  "error": {
+    "message": "failed eval: net/http: request canceled (Client.Timeout or context cancellation while reading body)"
+  }
+}
+```
+
+## Logs reference
+
+### Mongod Database
+
+This is the `mongod_database` data stream. This datastream collects a running log of events, including entries such as incoming connections, commands run, monitoring deployment, tuning performance and issues encountered.
+
+{{event "mongod_database"}}
+
+{{fields "mongod_database"}}

packages/mongodb_atlas/_dev/deploy/docker/Dockerfile

@@ -0,0 +1,9 @@
+FROM golang:1.20
+
+WORKDIR /app
+
+COPY ./mongodb_atlas .
+
+RUN go mod download
+
+CMD ["go","run","main.go"]

packages/mongodb_atlas/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,8 @@
+version: '3.0'
+services:
+  mongodbatlas:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    ports:
+      - 7780

packages/mongodb_atlas/_dev/deploy/docker/mongodb_atlas/go.mod

@@ -0,0 +1,3 @@
+module main.go
+
+go 1.20

packages/mongodb_atlas/_dev/deploy/docker/mongodb_atlas/main.go

@@ -0,0 +1,165 @@
+package main
+
+import (
+	"compress/gzip"
+	"crypto/md5"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+)
+
+// Replace these values with your desired username and password
+const (
+	username = "admin"
+	password = "MongoDB@123"
+)
+
+func generateDigest(username, realm, password, method, uri, nonce, nc, cnonce, qop string) string {
+	ha1 := md5.New()
+	fmt.Fprintf(ha1, "%s:%s:%s", username, realm, password)
+	ha1Hex := fmt.Sprintf("%x", ha1.Sum(nil))
+
+	ha2 := md5.New()
+	fmt.Fprintf(ha2, "%s:%s", method, uri)
+	ha2Hex := fmt.Sprintf("%x", ha2.Sum(nil))
+
+	response := md5.New()
+	fmt.Fprintf(response, "%s:%s:%s:%s:%s:%s", ha1Hex, nonce, nc, cnonce, qop, ha2Hex)
+	return fmt.Sprintf("%x", response.Sum(nil))
+}
+
+func digestAuth(w http.ResponseWriter, r *http.Request, realm, nonce, qop string) bool {
+	authHeader := r.Header.Get("Authorization")
+	if authHeader == "" {
+		w.Header().Set("WWW-Authenticate", fmt.Sprintf(`Digest realm="%s", qop="%s", nonce="%s"`, realm, qop, nonce))
+		http.Error(w, "Unauthorized", http.StatusUnauthorized)
+		return false
+	}
+
+	parts := strings.SplitN(authHeader, " ", 2)
+	if len(parts) != 2 || parts[0] != "Digest" {
+		http.Error(w, "Bad Request", http.StatusBadRequest)
+		return false
+	}
+
+	params := make(map[string]string)
+	for _, param := range strings.Split(parts[1], ",") {
+		kv := strings.SplitN(strings.TrimSpace(param), "=", 2)
+		if len(kv) != 2 {
+			http.Error(w, "Bad Request", http.StatusBadRequest)
+			return false
+		}
+		params[kv[0]] = strings.Trim(kv[1], `"`)
+	}
+
+	calculatedResponse := generateDigest(params["username"], realm, password, r.Method, r.RequestURI, nonce, params["nc"], params["cnonce"], params["qop"])
+
+	if calculatedResponse != params["response"] {
+		http.Error(w, "Unauthorized", http.StatusUnauthorized)
+		return false
+	}
+
+	return true
+}
+
+func hostHandler(w http.ResponseWriter, r *http.Request) {
+
+	realm := "MyRealm"
+	nonce := "123456"
+	qop := "auth"
+
+	authorized := digestAuth(w, r, realm, nonce, qop)
+	if !authorized {
+		return
+	}
+	// Create the response
+	type Link struct {
+		Rel string `json:"rel"`
+	}
+	type Result struct {
+		Hostname string `json:"hostname"`
+	}
+	type Response struct {
+		Links   []Link   `json:"links"`
+		Results []Result `json:"results"`
+	}
+
+	// Create the response
+	response := Response{
+		Links: []Link{
+			{
+				Rel: "self",
+			},
+		},
+		Results: []Result{
+			{
+				Hostname: "hostname1",
+			},
+		},
+	}
+
+	// Set the content type to application/json
+	w.Header().Set("Content-Type", "application/vnd.atlas.2023-02-01+json")
+
+	// Encode and send the response
+	err := json.NewEncoder(w).Encode(response)
+	if err != nil {
+		http.Error(w, fmt.Sprintf("Error encoding JSON: %v", err), http.StatusInternalServerError)
+		return
+	}
+}
+
+func handler(w http.ResponseWriter, r *http.Request) {
+	// Digest Authentication
+	realm := "MyRealm"
+	nonce := "123456"
+	qop := "auth"
+
+	authorized := digestAuth(w, r, realm, nonce, qop)
+	if !authorized {
+		return
+	}
+
+	fileName := strings.TrimPrefix(r.URL.Path, "/api/atlas/v2/groups/mongodb-group1/clusters/hostname1/logs/")
+	// Replace "path/to/your/log/file.log" with the actual path to your log file
+	var logFilePath string
+	if fileName == "mongodb.gz" {
+		logFilePath = "mongod_database_data.log"
+	}
+
+	// Open the log file
+	logFile, err := os.Open(logFilePath)
+	if err != nil {
+		http.Error(w, fmt.Sprintf("Error opening log file: %v", err), http.StatusInternalServerError)
+		return
+	}
+	defer logFile.Close()
+
+	// Create a gzip writer for the response
+	w.Header().Set("Content-Type", "application/vnd.atlas.2023-02-01+gzip")
+	w.Header().Set("Content-Disposition", `attachment; filename="mongodb-log.gz"`)
+
+	gzipWriter := gzip.NewWriter(w)
+	defer gzipWriter.Close()
+
+	// Copy the log file content to the gzip writer
+	_, err = io.Copy(gzipWriter, logFile)
+	if err != nil {
+		http.Error(w, fmt.Sprintf("Error reading log file: %v", err), http.StatusInternalServerError)
+		return
+	}
+}
+
+func main() {
+	// Existing handler for serving the compressed file
+	http.HandleFunc("/api/atlas/v2/groups/mongodb-group1/clusters/hostname1/logs/mongodb.gz", handler)
+
+	// New handler for returning the hostname
+	http.HandleFunc("/api/atlas/v2/groups/mongodb-group1/processes", hostHandler)
+
+	// Start the server
+	http.ListenAndServe(":7780", nil)
+}

packages/mongodb_atlas/_dev/deploy/docker/mongodb_atlas/mongod_database_data.log

@@ -0,0 +1,5 @@
+{ "t": { "$date": "2024-03-20T19:17:06.188+00:00" }, "s": "W", "c": "CONTROL", "id": 22120, "ctx": "initandlisten", "msg": "Access control is not enabled for the database. Read and write access to data and configuration is unrestricted", "tags": [ "startupWarnings" ] }
+{ "t": { "$date": "2024-02-18T14:45:23.512+00:00" }, "s": "I", "c": "NETWORK", "id": 67890, "ctx": "conn123", "msg": "Client connection accepted", "tags": [ "connection" ] }
+{ "t": { "$date": "2024-02-22T10:20:05.933+00:00" }, "s": "E", "c": "STORAGE", "id": 13579, "ctx": "journal", "msg": "Journal file not found", "tags": [ "journalError" ] }
+{ "t": { "$date": "2024-02-25T16:55:36.124+00:00" }, "s": "I", "c": "NETWORK", "id": 24680, "ctx": "conn456", "msg": "Client disconnected", "tags": [ "connection" ] }
+{ "t": { "$date": "2024-02-28T09:12:50.007+00:00" }, "s": "E", "c": "QUERY", "id": 98765, "ctx": "queryExecutor", "msg": "Query execution failed", "tags": [ "queryError" ] }

packages/mongodb_atlas/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.0.1"
+  changes:
+    - description: MongoDB Atlas integration package with "mongod_database" data stream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/9539

packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,5 @@
+dynamic_fields:
+  "event.ingested": ".*"
+fields:
+  tags:
+    - preserve_original_event

packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-mongod-database.json

@@ -0,0 +1,12 @@
+{
+    "events": [
+        {
+            "@timestamp": "2024-02-08T06:20:49.729Z",
+            "message": "No data for given time period or host is unreachable"
+        },
+        {
+            "@timestamp": "2024-02-08T06:20:56.621Z",
+            "message": "{ \"t\": { \"$date\": \"2020-05-20T19:17:06.188+00:00\" }, \"s\": \"W\", \"c\": \"CONTROL\", \"id\": 22120, \"ctx\": \"initandlisten\", \"msg\": \"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted\", \"tags\": [ \"startupWarnings\" ] }"
+        }
+    ]
+}

packages/mongodb_atlas/data_stream/mongod_database/_dev/test/pipeline/test-mongod-database.json-expected.json

@@ -0,0 +1,43 @@
+{
+    "expected": [
+        null,
+        {
+            "@timestamp": "2020-05-20T19:17:06.188Z",
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "category": [
+                    "network",
+                    "database"
+                ],
+                "kind": "event",
+                "module": "mongodb_atlas",
+                "original": "{ \"t\": { \"$date\": \"2020-05-20T19:17:06.188+00:00\" }, \"s\": \"W\", \"c\": \"CONTROL\", \"id\": 22120, \"ctx\": \"initandlisten\", \"msg\": \"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted\", \"tags\": [ \"startupWarnings\" ] }",
+                "type": [
+                    "access",
+                    "info"
+                ]
+            },
+            "log": {
+                "level": "warning"
+            },
+            "mongodb_atlas": {
+                "mongod_database": {
+                    "component": "CONTROL",
+                    "id": 22120,
+                    "message": "Access control is not enabled for the database. Read and write access to data and configuration is unrestricted",
+                    "tags": [
+                        "startupWarnings"
+                    ],
+                    "thread": {
+                        "name": "initandlisten"
+                    }
+                }
+            },
+            "tags": [
+                "preserve_original_event"
+            ]
+        }
+    ]
+}

packages/mongodb_atlas/data_stream/mongod_database/_dev/test/system/test-default-config.yml

@@ -0,0 +1,13 @@
+vars:
+  url:
+    - http://{{Hostname}}:{{Port}}
+  public_key:
+    - admin
+  private_key:
+    - MongoDB@123
+data_stream:
+  vars:
+    groupId:
+      - mongodb-group1
+input: cel
+service: mongodbatlas