Kibana Package (#1519)

packages/kibana/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]

packages/kibana/_dev/build/docs/README.md

@@ -0,0 +1,94 @@
+# Kibana
+
+## Configuration parameters
+
+If the Kibana instance is using a basepath in its URL, you must set the `basepath` setting for this integration with the same value.
+
+## Compatibility
+
+The `kibana` package works with Kibana 6.7.0 and later.
+
+## Usage for Stack Monitoring
+
+The `kibana` package can be used to collect metrics shown in our Stack Monitoring
+UI in Kibana. To enable this usage, set `xpack.enabled: true` on the package config.
+
+## Logs
+
+### Audit
+
+{{fields "audit"}}
+
+### Log
+
+{{fields "log"}}
+
+## Metrics
+
+### Stats
+
+Stats data stream uses the stats endpoint of Kibana, which is available in 6.4 by default.
+
+**Exported fields**
+
+| Field | Description | Type |
+|---|---|---|
+| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
+| data_stream.dataset | Data stream dataset. | constant_keyword |
+| data_stream.namespace | Data stream namespace. | constant_keyword |
+| data_stream.type | Data stream type. | constant_keyword |
+| kibana.stats.concurrent_connections | Number of client connections made to the server. Note that browsers can send multiple simultaneous connections to request multiple server assets at once, and they can re-use established connections. | long |
+| kibana.stats.host.name | Kibana instance hostname | keyword |
+| kibana.stats.index | Name of Kibana's internal index | keyword |
+| kibana.stats.kibana.status |  | keyword |
+| kibana.stats.name | Kibana instance name | keyword |
+| kibana.stats.os.distro |  | keyword |
+| kibana.stats.os.distroRelease |  | keyword |
+| kibana.stats.os.load.15m |  | half_float |
+| kibana.stats.os.load.1m |  | half_float |
+| kibana.stats.os.load.5m |  | half_float |
+| kibana.stats.os.memory.free_in_bytes |  | long |
+| kibana.stats.os.memory.total_in_bytes |  | long |
+| kibana.stats.os.memory.used_in_bytes |  | long |
+| kibana.stats.os.platform |  | keyword |
+| kibana.stats.os.platformRelease |  | keyword |
+| kibana.stats.process.event_loop_delay.ms | Event loop delay in milliseconds | scaled_float |
+| kibana.stats.process.memory.heap.size_limit.bytes | Max. old space size allocated to Node.js process, in bytes | long |
+| kibana.stats.process.memory.heap.total.bytes | Total heap allocated to process in bytes | long |
+| kibana.stats.process.memory.heap.uptime.ms | Uptime of process in milliseconds | long |
+| kibana.stats.process.memory.heap.used.bytes | Heap used by process in bytes | long |
+| kibana.stats.process.memory.resident_set_size.bytes |  | long |
+| kibana.stats.process.uptime.ms |  | long |
+| kibana.stats.request.disconnects | Number of requests that were disconnected | long |
+| kibana.stats.request.total | Total number of requests | long |
+| kibana.stats.response_time.avg.ms | Average response time in milliseconds | long |
+| kibana.stats.response_time.max.ms | Maximum response time in milliseconds | long |
+| kibana.stats.snapshot | Whether the Kibana build is a snapshot build | boolean |
+| kibana.stats.status | Kibana instance's health status | keyword |
+| kibana.stats.usage.index |  | keyword |
+| service.id | Unique identifier of the running service. If the service is comprised of many nodes, the `service.id` should be the same for all nodes. This id should uniquely identify the service. This makes it possible to correlate logs and metrics for one specific service, no matter which particular node emitted the event. Note that if you need to see the events from one specific host of the service, you should filter on that `host.name` or `host.id` instead. | keyword |
+| service.version | Version of the service the data was collected from. This allows to look at a data set only for a specific version of a service. | keyword |
+
+{{event "stats"}}
+
+### Status
+
+This status endpoint is available in 6.0 by default and can be enabled in Kibana >= 5.4 with the config option `status.v6ApiFormat: true`.
+
+**Exported fields**
+
+| Field | Description | Type |
+|---|---|---|
+| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
+| data_stream.dataset | Data stream dataset. | constant_keyword |
+| data_stream.namespace | Data stream namespace. | constant_keyword |
+| data_stream.type | Data stream type. | constant_keyword |
+| kibana.status.metrics.concurrent_connections | Current concurrent connections. | long |
+| kibana.status.metrics.requests.disconnects | Total number of disconnected connections. | long |
+| kibana.status.metrics.requests.total | Total number of connections. | long |
+| kibana.status.name | Kibana instance name. | keyword |
+| kibana.status.status.overall.state | Kibana overall state. | keyword |
+| service.id | Unique identifier of the running service. If the service is comprised of many nodes, the `service.id` should be the same for all nodes. This id should uniquely identify the service. This makes it possible to correlate logs and metrics for one specific service, no matter which particular node emitted the event. Note that if you need to see the events from one specific host of the service, you should filter on that `host.name` or `host.id` instead. | keyword |
+| service.version | Version of the service the data was collected from. This allows to look at a data set only for a specific version of a service. | keyword |
+
+{{event "status"}}

packages/kibana/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: 1.0.0
+  changes:
+    - description: initial release
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/1519

packages/kibana/data_stream/audit/agent/stream/log.yml.hbs

@@ -0,0 +1,14 @@
+paths:
+{{#each paths as |path i|}}
+ - {{path}}
+{{/each}}
+exclude_files: [".gz$"]
+processors:
+  - add_locale: ~
+  - add_fields:
+      target: ''
+      fields:
+        ecs.version: 1.10.0
+  - decode_json_fields:
+      fields: [message]
+      target: kibana._audit_temp

packages/kibana/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

@@ -0,0 +1,22 @@
+---
+description: Pipeline for parsing Kibana audit logs
+processors:
+- set:
+    field: event.ingested
+    value: '{{_ingest.timestamp}}'
+- rename:
+    field: '@timestamp'
+    target_field: event.created
+- pipeline:
+    name: '{{ IngestPipeline "pipeline-json" }}'
+- set:
+    field: event.kind
+    value: event
+- append:
+    field: related.user
+    value: "{{user.name}}"
+    if: "ctx?.user?.name != null"
+on_failure:
+- set:
+    field: error.message
+    value: '{{ _ingest.on_failure_message }}'

packages/kibana/data_stream/audit/elasticsearch/ingest_pipeline/pipeline-json.yml

@@ -0,0 +1,115 @@
+---
+description: Pipeline for parsing Kibana audit logs in JSON format
+processors:
+
+- set:
+    field: "@timestamp"
+    value: "{{kibana._audit_temp.@timestamp}}"
+
+- set:
+    field: message
+    value: "{{kibana._audit_temp.message}}"
+
+- set:
+    if: ctx.kibana._audit_temp.event.action != null
+    field: event.action
+    value: "{{kibana._audit_temp.event.action}}"
+- set:
+    if: ctx.kibana._audit_temp.event.category != null && ctx.kibana._audit_temp.event.category instanceof List
+    field: event.category
+    value: "{{kibana._audit_temp.event.category.0}}"
+- set:
+    if: ctx.kibana._audit_temp.event.category != null && ctx.kibana._audit_temp.event.category instanceof String
+    field: event.category
+    value: "{{kibana._audit_temp.event.category}}"
+- set:
+    if: ctx.kibana._audit_temp.event.outcome != null
+    field: event.outcome
+    value: "{{kibana._audit_temp.event.outcome}}"
+- set:
+    if: ctx.kibana._audit_temp.event.type != null && ctx.kibana._audit_temp.event.type instanceof List
+    field: event.type
+    value: "{{kibana._audit_temp.event.type.0}}"
+- set:
+    if: ctx.kibana._audit_temp.event.type != null && ctx.kibana._audit_temp.event.type instanceof String
+    field: event.type
+    value: "{{kibana._audit_temp.event.type}}"
+
+- remove:
+    field: 'ecs'
+- rename:
+    if: ctx.kibana._audit_temp.ecs != null
+    field: kibana._audit_temp.ecs
+    target_field: "ecs"
+
+- rename:
+    if: ctx.kibana._audit_temp.url != null
+    field: kibana._audit_temp.url
+    target_field: "url"
+
+- set:
+    if: ctx.url?.query == null
+    field: url.original
+    value: '{{url.path}}'
+    ignore_empty_value: true
+- set:
+    if: ctx.url?.path != null && ctx.url?.query != null
+    field: url.original
+    value: '{{url.path}}?{{url.query}}'
+
+- rename:
+    if: ctx.kibana._audit_temp.http != null
+    field: kibana._audit_temp.http
+    target_field: http
+
+- rename:
+    if: ctx.kibana._audit_temp.user != null
+    field: kibana._audit_temp.user
+    target_field: user
+
+- rename:
+    if: ctx.kibana._audit_temp.trace != null
+    field: kibana._audit_temp.trace
+    target_field: trace
+
+- rename:
+    if: ctx.kibana._audit_temp.process?.pid != null
+    target_field: process
+    field: kibana._audit_temp.process
+
+- rename:
+    if: ctx.kibana._audit_temp.error != null
+    target_field: error
+    field: kibana._audit_temp.error
+
+- rename:
+    if: ctx.kibana._audit_temp.kibana.session_id != null
+    target_field: kibana.session_id
+    field: kibana._audit_temp.kibana.session_id
+
+- rename:
+    if: ctx.kibana._audit_temp.kibana.space_id != null
+    target_field: kibana.space_id
+    field: kibana._audit_temp.kibana.space_id
+
+- rename:
+    if: ctx.kibana._audit_temp.kibana.saved_object != null
+    target_field: kibana.saved_object
+    field: kibana._audit_temp.kibana.saved_object
+
+- rename:
+    if: ctx.kibana._audit_temp.kibana.add_to_spaces != null
+    target_field: kibana.add_to_spaces
+    field: kibana._audit_temp.kibana.add_to_spaces
+
+- rename:
+    if: ctx.kibana._audit_temp.kibana.delete_from_spaces != null
+    target_field: kibana.delete_from_spaces
+    field: kibana._audit_temp.kibana.delete_from_spaces
+
+- remove:
+    field: 'kibana._audit_temp'
+on_failure:
+- set:
+    field: error.message
+    value: '{{ _ingest.on_failure_message }}'

packages/kibana/data_stream/audit/fields/base-fields.yml

@@ -0,0 +1,9 @@
+- name: data_stream.type
+  type: constant_keyword
+  description: Data stream type.
+- name: data_stream.dataset
+  type: constant_keyword
+  description: Data stream dataset.
+- name: data_stream.namespace
+  type: constant_keyword
+  description: Data stream namespace.

packages/kibana/data_stream/audit/fields/ecs.yml

@@ -0,0 +1,2 @@
+- name: '@timestamp'
+  external: ecs

packages/kibana/data_stream/audit/fields/package-fields.yml

@@ -0,0 +1,33 @@
+- name: kibana
+  type: group
+  fields:
+    - name: session_id
+      type: keyword
+      description: The ID of the user session associated with this event. Each login attempt results in a unique session id.
+    - name: space_id
+      type: keyword
+      description: The id of the space associated with this event.
+    - name: saved_object.type
+      type: keyword
+      description: The type of the saved object associated with this event.
+    - name: saved_object.id
+      type: keyword
+      description: The id of the saved object associated with this event.
+    - name: add_to_spaces
+      type: keyword
+      description: The set of space ids that a saved object was shared to.
+    - name: delete_from_spaces
+      type: keyword
+      description: The set of space ids that a saved object was removed from.
+    - name: authentication_provider
+      type: keyword
+      description: The authentication provider associated with a login event.
+    - name: authentication_type
+      type: keyword
+      description: The authentication provider type associated with a login event.
+    - name: authentication_realm
+      type: keyword
+      description: The Elasticsearch authentication realm name which fulfilled a login event.
+    - name: lookup_realm
+      type: keyword
+      description: The Elasticsearch lookup realm which fulfilled a login event.

packages/kibana/data_stream/audit/manifest.yml

@@ -0,0 +1,17 @@
+type: logs
+title: kibana audit logs
+release: experimental
+streams:
+  - input: logfile
+    vars:
+      - name: paths
+        type: text
+        title: Paths
+        multi: true
+        required: true
+        show_user: true
+        default:
+          - /var/log/kibana/*_audit.json
+    template_path: log.yml.hbs
+    title: Kibana Audit logs
+    description: Collect Kibana audit logs

packages/kibana/data_stream/log/agent/stream/log.yml.hbs

@@ -0,0 +1,12 @@
+paths:
+{{#each paths as |path i|}}
+ - {{path}}
+{{/each}}
+exclude_files: [".gz$"]
+json.keys_under_root: false
+json.add_error_key: true
+processors:
+  - add_fields:
+      target: ''
+      fields:
+        ecs.version: 1.10.0