[FEATURE] Add First Integration into Observability

[YANG-DB]

Is your feature request related to a problem?
As part of the Integration RFC we would like to add the first Integration use case for NginX pre-canned dashboards including the logs & metrics observability schema supported signals

New repo structure

Proposal for Observability repo new structure structure

- build-tools
- config/checkstyle
- gradle/wrapper
- release-notes
- schema
    - observability
       - logs
       - traces
       - metrics
- scripts
- src
- integrations
   - observability
       - nginX

What solution would you like?

Integrating Component Structure

The following section details the structure and composition of an integration component and how it may be utilized for the Observability use-cases.

Structure

As mentioned above, integration is a collection of elements that formulate how to observe a specific data emitting resource - in our case a telemetry data producer.

A typical Observability Integration consists of the following parts:

Metadata

* Observability data producer resource
* Supplement Indices (mapping & naming)
* Collection Agent Version
* Transformation schema 
* Optional test harnesses repository
* Verified version and documentation 
* Category & classification (logs/traces/alerts/metrics)

Display components

* Dashboards 
* Maps
* Applications
* Notebooks
* Operations Panels
* Saved PPL/SQL/DQL Queries
* Alerts

A major factor in the following RFC is that structured data has an enormous contribution to the understanding of the system behaviour.
Once input content has form and shape - it can and will be used to calculate and correlate different pieces of data.

The next parts of this document will present Integrations For Observability which has a key concept of Observability schema.

It will overview the concepts of observability, will describe the current issues customers are facing with observability and continue to elaborate on how to mitigate them using Integrations and structured schemas.

---

Creating An Integration

Integrations are an encapsulated collection of elements and a such have a specific structure.

NginX

Lets examine the next NginX integration component:

nginX
    config.json
    display`
        ApplicationA.json
        Maps.json
        DashboardB.json
    queries
      QueryA.json
    schemas
      transformation.json
    samples
      nginx.access logs
      nginx.error logs
      nginx.stats metrics
      expected transformation results
    metadata
      ...
    info  
      documentation

Definitions

• config.json defines the general configuration for the entire integration component.
• display this is the folder in which the actual visualization components are stored
• queries this is the folder in which the actual PPL queries are stored
• schemas this is the folder in which the schemas are stored - schema for mapping translations or index mapping.
• samples this folder contains sample logs and translated logs are present
• metadata this folder contains additional metadata definitions such as security and policies
• info this folder contains documentations, licences and external references

Config.json file includes the following Integration configuration

{
  "name": "nginx",
  "version": {
        "integ": "0.1.0",
        "schema": "1.0.0",
        "resource": "^1.23.0",
   }
  "description": "Nginx HTTP server collector",
  "identification": "instrumentationScope.attributes.identification",
  "categories": [
    "web",
  ],
  "collection":[
    {
       "logs": [{
                    "info": "access logs",
                    "input_type":"logfile",
                    "dataset":"nginx.access",
                    "labels" :["`nginx"``,``"access"``],`
                    "schema": "./schema/logs/access.json"
                },
                {
                    "info": "error logs",
                    "input_type":"logfile",
                    "labels" :["nginx","error"],
                    "dataset":"nginx.error",
                    "schema": "./schema/logs/error.json"
                }]
    },
    {
        "metrics": [{
                    "info": "`status metrics`",
                    "input_type":"`metrics`",
                    "dataset":"nginx.status",
                    "labels" :["nginx","status"],
                    "schema": "./schema/metrics/status.json"
                }]
    }
  ],
  "repo": {
    "github": ".../"
  }
}

Definitions:

  "version": {
        "integ": "0.1.0",
        "schema": "1.0.0",
        "resource": "^1.23.0",
   }

version:
This references the next semantic versioning:

• integ version indicates the version for this specific Integration
• schema version indicates the Observability schema version
• resource version indicates the actual resource version which is being integrated.

identification:
This references the field this integration is using to explicitly identify the resource the signal is generated from

In this case the field resides in the instrumentationScope.attributes.identification path and should have a value that corresponds to the name of the integration.

"identification": "instrumentationScope.attributes.identification",

Categories:
This section defines the classification categories associated to this Integration according to ECS specification (https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-category.html)

collection:
This references the different types of collection this integration if offering. It can be one of the following
{ Traces, Logs, Metrics, Alerts, Supplements }.

Collections

Let's dive into a specific log collection:

       "logs": [{
                    "info": "access logs",
                    "input_type":"logfile",
                    "dataset":"nginx.access",
                    "labels" :["`nginx"``,``"access"``],`
                    "schema": "./schema/logs/access.json"
                },

This log collects nginx access logs as described in the info section.
The input_type is a categorical classification of the log kind which is specified in the ECS specification as well.

• dataset is defined above and indicates the target routing index.
• lables are general purpose labeling tags that allow further correlation and associations.
• schema is the location of the mapping configuration between the original log format to the Observability Log format.

---

Display:

Visualization contains the relevant visual components associated with this integration.

The visual display component will need to be validated to the schema that it is expected to work on - this may be part of the Integration validation flow...

Queries

Queries contains specific PPL queries that precisely demonstrates some common and useful use-case .

Example:

-- The visual display component will need to be validated to the schema that it is expected to work on

source = logs-*-prod | ... where ...

---

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?

• Observability Naming standards
• Integration RFC

• Traces
  • https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/otel-trace-source
  • https://github.com/open-telemetry/opentelemetry-proto/blob/v0.9.0/opentelemetry/proto/trace/v1/trace.proto
  • https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/trace/semantic_conventions
  • https://github.com/open-telemetry/opentelemetry-java/tree/0a9794ad415c87c162f518a9112a9b7849564bee/sdk/trace
  • Support otel trace mapping #1395
• Metrics
  • https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/otel-metrics-source
  • https://github.com/open-telemetry/opentelemetry-proto/blob/v0.9.0/opentelemetry/proto/metrics/v1/metrics.proto
  • https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/metrics/semantic_conventions
  • https://github.com/open-telemetry/opentelemetry-java/tree/0a9794ad415c87c162f518a9112a9b7849564bee/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics
  • Support otel metrics mapping #1397
• Logs
  • based on OTEL / ECS logs formats OTEL / ECS
  • https://github.com/open-telemetry/opentelemetry-proto/blob/v0.9.0/opentelemetry/proto/logs/v1/logs.proto
  • https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/logs/semantic_conventions
  • https://github.com/open-telemetry/opentelemetry-java/tree/0a9794ad415c87c162f518a9112a9b7849564bee/sdk/logs
  • Support SSO logs mapping #1403