From 50d9adc14f4d39a5606a086ed9fce50c3b3b8c86 Mon Sep 17 00:00:00 2001 From: Kush Rana Date: Thu, 21 Nov 2024 12:27:21 +0530 Subject: [PATCH] address review comments --- packages/vsphere/_dev/build/docs/README.md | 98 ++- .../data_stream/cluster/fields/fields.yml | 4 +- .../data_stream/cluster/sample_event.json | 48 +- .../data_stream/datastore/fields/fields.yml | 2 +- .../data_stream/datastore/sample_event.json | 76 +-- .../datastorecluster/fields/fields.yml | 6 +- .../datastorecluster/sample_event.json | 44 +- .../data_stream/host/fields/fields.yml | 4 +- .../data_stream/host/sample_event.json | 112 +++- .../data_stream/network/sample_event.json | 148 +++-- .../resourcepool/sample_event.json | 44 +- .../virtualmachine/fields/fields.yml | 2 +- .../virtualmachine/sample_event.json | 126 ++-- packages/vsphere/docs/README.md | 596 +++++++++--------- 14 files changed, 663 insertions(+), 647 deletions(-) diff --git a/packages/vsphere/_dev/build/docs/README.md b/packages/vsphere/_dev/build/docs/README.md index 551ef126639..7b015bd5fa4 100644 --- a/packages/vsphere/_dev/build/docs/README.md +++ b/packages/vsphere/_dev/build/docs/README.md @@ -16,89 +16,85 @@ The vSphere integration collects logs and metrics. Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by vSphere as integration is `log`. -Metrics give you insight into the statistics of the vSphere. The `Metric` data streams collected by the vSphere integration are `cluster`, `datastore`, `datastorecluster`, `host`, `network`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. +Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `datastorecluster`, `host`, `network`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. + +Data Streams: + +- **`log`**: This data stream collects logs generated by VMware vSphere using a syslog daemon. +- **`cluster`**: This data stream collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. +- **`datastore`**: This data stream gathers datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. +- **`datastorecluster`**: This data stream gathers metrics for datastore clusters from VMware vSphere, including statistics like cluster capacity and available free space. Additionally, it provides information about the individual datastores that comprise the cluster. +- **`host`**: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. +- **`resourcepool`**: This data stream collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. +- **`virtualmachine`**: This data stream gathers virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. +- **`network`**: This data stream gathers metrics and status information related to VMware vSphere networks, including network accessibility, connected hosts and virtual machines, configuration health, and network type. + +## Important Note +- Users can monitor and see the log inside the ingested documents for vSphere in the `logs-*` index pattern from `Discover`, and for metrics, the index pattern is `metrics-*`. +- **Supported Periods**: + - Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds by default. + - The Datastore and Host datastreams support performance data collection using the vSphere performance API. + - Since the performance API has usage restrictions based on data collection intervals, users should ensure that the period is configured optimally to receive real-time data. Users can still collect summary metrics if performance metrics are not supported for the configured instance. + - Period configuration can be determined based on the [Data Collection Intervals](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html) and [Data Collection Levels](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html). -Data streams: - -- **`log`**: Collects logs generated by VMware vSphere using a syslog daemon. -- **`cluster`**: Collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. -- **`datastore`**: Gathers datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. -- **`datastorecluster`**: Gathers metrics for datastore clusters from VMware vSphere, including statistics like cluster capacity and available free space. Additionally, it provides information about the individual datastores that comprise the cluster. -- **`host`**: Collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. -- **`resourcepool`**: Collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. -- **`virtualmachine`**: Gathers virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. -- **`network`**: Gathers metrics and status information related to VMware vSphere networks, including network accessibility, connected hosts and virtual machines, configuration health, and network type. - -Note: - -Users can monitor and check the log inside the ingested documents for vSphere in the `logs-*` index pattern from `Discover`, and for metrics, the index pattern is `metrics-*`. +## Prerequisites -## Supported periods - -- Real-time data collection: An ESXi Server collects data for each performance counter every 20 seconds by default. -- The `datastore` and `host` data streams support performance data collection using the vSphere performance API. -- Since the performance API has usage restrictions based on data collection intervals, make sure that the period is configured optimally to receive real-time data. You can still collect summary metrics if performance metrics are not supported for the configured instance. -- Period configuration can be determined based on the [Data Collection Intervals](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html) and [Data Collection Levels](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html). +You can store and search your data using Elasticsearch and visualize and manage it with Kibana. We recommend using our hosted Elasticsearch Service on Elastic Cloud or self-managing the Elastic Stack on your own hardware. ## Compatibility This integration has been tested and verified against VMware ESXi and vCenter version 7.0.3. -## Prerequisites - -You can store and search your data using Elasticsearch and visualize and manage it with Kibana. We recommend using our hosted Elasticsearch Service on Elastic Cloud or self-managing the Elastic Stack on your own hardware. - ## Setup -For step-by-step instructions on how to set up an integration, check the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. +For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. -## Logs +## Logs reference -The `log` data stream collects logs generated by VMware vSphere using a syslog daemon. +### Log -To collect logs, follow these steps: +This is the `log` data stream. This data stream collects logs generated by VMware vSphere using a syslog daemon. -1. Configure the listening host/IP address (default: localhost) and host port (default: 9525) in the integration. - -1. Configure vSphere to send logs to a remote syslog host and provide the configured hostname/IP and port of the Elastic Agent host. +Note: +- To collect logs, a syslog daemon is used. First, you must configure the listening host/IP address (default: localhost) and host port (default: 9525) in the integration. Then, configure vSphere to send logs to a remote syslog host and provide the configured hostname/IP and port of the Elastic Agent host. {{event "log"}} **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. {{fields "log"}} -## Metrics +## Metrics reference -To access the metrics, provide the URL in the **Add Integration** page of the vSphere package. +Note: +- To access the metrics, provide the URL in the "Add Integration" page of the vSphere package. ### Cluster - Clusters in vSphere represent a group of ESXi hosts working together to optimize resource allocation, ensure high availability, and manage workloads efficiently. {{event "cluster"}} **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. {{fields "cluster"}} ### Datastore -This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. +This is `datastore` data stream. This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. {{event "datastore"}} **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. {{fields "datastore"}} -### Datastore cluster +### Datastore Cluster Datastore clusters in vSphere group multiple datastores for optimized management and automated load balancing, ensuring efficient storage utilization and simplified administration. @@ -106,59 +102,57 @@ Datastore clusters in vSphere group multiple datastores for optimized management **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. {{fields "datastorecluster"}} ### Host -This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. +This is the `host` data stream. This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. {{event "host"}} **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. {{fields "host"}} ### Network -This data stream gathers metrics and status information related to VMware vSphere networks, including network accessibility, connected hosts and virtual machines, configuration health, and network type. +This is the `network` data stream. This data stream gathers metrics and status information related to VMware vSphere networks, including network accessibility, connected hosts and virtual machines, configuration health, and network type. {{event "network"}} **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. {{fields "network"}} ### Resourcepool - -Resource pools in vSphere enable the allocation and management of CPU and memory resources across groups of virtual machines. +Resource pools in vSphere allow for the allocation and management of CPU and memory resources across groups of virtual machines. {{event "resourcepool"}} **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. {{fields "resourcepool"}} ### Virtual Machine -This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. +This is the `virtualmachine` data stream. This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. {{event "virtualmachine"}} **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. {{fields "virtualmachine"}} ## Troubleshoot -If you encounter issues during the setup or usage of the VMware vSphere integration, in particular with Data Collection Intervals, with agent error messages such as **`Failed to query performance metrics: ServerFaultCode: A specified parameter was not correct: querySpec.interval`**, refer to -[Supported periods](#supported-periods) for guidance and resolution. \ No newline at end of file +If you encounter any issues during the setup or usage of the VMware vSphere integration, particularly with regards to Data Collection Intervals, with agent error messages such as **`Failed to query performance metrics: ServerFaultCode: A specified parameter was not correct: querySpec.interval`**, please refer to the mentioned Important Notes/Supported Periods for guidance and resolution. diff --git a/packages/vsphere/data_stream/cluster/fields/fields.yml b/packages/vsphere/data_stream/cluster/fields/fields.yml index 89b2ffb29f9..3121224ef9c 100644 --- a/packages/vsphere/data_stream/cluster/fields/fields.yml +++ b/packages/vsphere/data_stream/cluster/fields/fields.yml @@ -5,7 +5,7 @@ - name: alert.names type: keyword description: > - List of all the alerts on this Cluster. + List of all the alerts on this cluster. - name: das_config type: group fields: @@ -91,4 +91,4 @@ - name: warning.names type: keyword description: > - List of all the warnings on this Cluster. \ No newline at end of file + List of all the warnings on this cluster. \ No newline at end of file diff --git a/packages/vsphere/data_stream/cluster/sample_event.json b/packages/vsphere/data_stream/cluster/sample_event.json index 66baed93101..bacae3813c7 100644 --- a/packages/vsphere/data_stream/cluster/sample_event.json +++ b/packages/vsphere/data_stream/cluster/sample_event.json @@ -1,54 +1,51 @@ { - "@timestamp": "2024-09-19T05:44:00.800Z", + "@timestamp": "2024-11-21T06:39:19.834Z", "agent": { - "ephemeral_id": "676a770b-a207-4fec-99d4-e82377578711", - "id": "6b430ae3-0bdb-4d5c-b60d-a02f54e770e5", - "name": "elastic-agent-47605", + "ephemeral_id": "f41470c3-9bf0-4217-baf6-73383f35fcff", + "id": "6b90cdc8-92f4-486f-9b8a-02a0aca316a6", + "name": "elastic-agent-81320", "type": "metricbeat", - "version": "8.15.2" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.cluster", - "namespace": "93141", + "namespace": "35689", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "6b430ae3-0bdb-4d5c-b60d-a02f54e770e5", + "id": "6b90cdc8-92f4-486f-9b8a-02a0aca316a6", "snapshot": true, - "version": "8.15.2" + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.cluster", - "duration": 17059144, - "ingested": "2024-09-19T05:44:03Z", + "duration": 33840300, + "ingested": "2024-11-21T06:39:22Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-47605", - "id": "57723763cd1b4ff48e54a505de4ebe6c", + "hostname": "elastic-agent-81320", "ip": [ - "192.168.244.4", - "192.168.245.2" + "192.168.242.4", + "192.168.243.2" ], "mac": [ - "02-42-C0-A8-F4-04", - "02-42-C0-A8-F5-02" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F3-02" ], - "name": "elastic-agent-47605", + "name": "elastic-agent-81320", "os": { - "codename": "focal", - "family": "debian", - "kernel": "4.18.0-348.7.1.el8_5.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "kernel": "3.10.0-1160.118.1.el7.x86_64", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { @@ -76,15 +73,16 @@ "DC0_C0_H2" ] }, + "id": "domain-c28", "name": "DC0_C0", "network": { "count": 3, "names": [ "DC0_DVPG0", - "DVS0-DVUplinks-9", + "DVS0-DVUplinks-10", "VM Network" ] } } } -} \ No newline at end of file +} diff --git a/packages/vsphere/data_stream/datastore/fields/fields.yml b/packages/vsphere/data_stream/datastore/fields/fields.yml index 381cc2d5dbc..445e6356825 100644 --- a/packages/vsphere/data_stream/datastore/fields/fields.yml +++ b/packages/vsphere/data_stream/datastore/fields/fields.yml @@ -5,7 +5,7 @@ - name: alert.names type: keyword description: > - List of all the alerts on this Datastore. + List of all the alerts on this datastore. - name: name type: keyword # Reason to add as a dimension field: to uniquely identify the datastore diff --git a/packages/vsphere/data_stream/datastore/sample_event.json b/packages/vsphere/data_stream/datastore/sample_event.json index f48886186e3..7b68c593e3b 100644 --- a/packages/vsphere/data_stream/datastore/sample_event.json +++ b/packages/vsphere/data_stream/datastore/sample_event.json @@ -1,87 +1,76 @@ { - "@timestamp": "2024-09-02T10:04:25.122Z", + "@timestamp": "2024-11-21T06:40:14.950Z", "agent": { - "ephemeral_id": "4da294a3-ad54-47f4-92c7-544e1356a0d8", - "id": "b01ab3cf-51ad-4c4d-87bd-fc2d4aa59d8a", - "name": "elastic-agent-76236", + "ephemeral_id": "80421da7-699d-4ad1-96a3-56d7aa430190", + "id": "482796dd-401d-4568-8c52-7f9036322873", + "name": "elastic-agent-36929", "type": "metricbeat", - "version": "8.16.0" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.datastore", - "namespace": "86691", + "namespace": "35376", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "b01ab3cf-51ad-4c4d-87bd-fc2d4aa59d8a", + "id": "482796dd-401d-4568-8c52-7f9036322873", "snapshot": true, - "version": "8.16.0" + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.datastore", - "duration": 190147614, - "ingested": "2024-09-02T10:04:27Z", + "duration": 48985279, + "ingested": "2024-11-21T06:40:17Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-76236", - "id": "e744630f9d4f43dc818e497d221bd0b2", + "hostname": "elastic-agent-36929", "ip": [ - "172.18.0.4", - "172.21.0.2" + "192.168.242.4", + "192.168.245.2" ], "mac": [ - "02-42-AC-12-00-04", - "02-42-AC-15-00-02" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F5-02" ], - "name": "elastic-agent-76236", + "name": "elastic-agent-36929", "os": { - "codename": "focal", - "family": "debian", - "kernel": "5.15.153.1-microsoft-standard-WSL2", - "name": "Ubuntu", - "platform": "ubuntu", + "kernel": "3.10.0-1160.118.1.el7.x86_64", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { "name": "datastore", - "period": 10000 + "period": 20000 }, "service": { "address": "https://svc-vsphere-metrics:8989/sdk", "type": "vsphere" }, + "tags": [ + "vsphere-datastore" + ], "vsphere": { "datastore": { - "disk": { - "capacity": { - "usage": { - "bytes": 520505786368 - }, - "bytes": 1610344300544 - }, - "provisioned": { - "bytes": 520505786368 - } - }, "capacity": { "free": { - "bytes": 37120094208 + "bytes": 10952166604800 }, "total": { - "bytes": 74686664704 + "bytes": 10995116277760 }, "used": { - "bytes": 37566570496, - "pct": 0.502988996026061 + "bytes": 42949672960, + "pct": 0.004 } }, "fstype": "OTHER", @@ -89,10 +78,8 @@ "count": 1, "names": "DC0_H0" }, + "id": "datastore-53", "name": "LocalDS_0", - "read": { - "bytes": 1024 - }, "status": "green", "vm": { "count": 4, @@ -102,10 +89,7 @@ "DC0_H0_VM0", "DC0_H0_VM1" ] - }, - "write": { - "bytes": 450560 } } } -} \ No newline at end of file +} diff --git a/packages/vsphere/data_stream/datastorecluster/fields/fields.yml b/packages/vsphere/data_stream/datastorecluster/fields/fields.yml index dc41e67cfa8..1737f64ae63 100644 --- a/packages/vsphere/data_stream/datastorecluster/fields/fields.yml +++ b/packages/vsphere/data_stream/datastorecluster/fields/fields.yml @@ -1,12 +1,12 @@ - name: vsphere.datastorecluster type: group description: > - Datastore Cluster + Datastore cluster fields: - name: alert.names type: keyword description: > - List of all the alerts on this Datastore Cluster. + List of all the alerts on this datastore cluster. - name: capacity.bytes type: long metric_type: gauge @@ -68,4 +68,4 @@ - name: warning.names type: keyword description: > - List of all the warnings on this Datastore Cluster. + List of all the warnings on this datastore cluster. diff --git a/packages/vsphere/data_stream/datastorecluster/sample_event.json b/packages/vsphere/data_stream/datastorecluster/sample_event.json index f524e63af28..d85e6f7ce8d 100644 --- a/packages/vsphere/data_stream/datastorecluster/sample_event.json +++ b/packages/vsphere/data_stream/datastorecluster/sample_event.json @@ -1,54 +1,51 @@ { - "@timestamp": "2024-09-22T05:28:46.315Z", + "@timestamp": "2024-11-21T06:41:10.164Z", "agent": { - "ephemeral_id": "8b7e9ea4-0517-4e98-a795-b6fe529f4a2f", - "id": "7737279e-51e9-4d90-a0d0-2c12dc4446bf", - "name": "elastic-agent-23128", + "ephemeral_id": "dca752ad-1ce6-40f5-8a40-c25947763083", + "id": "cd777970-1a7d-4e59-99bf-7201c2c21f9e", + "name": "elastic-agent-72401", "type": "metricbeat", - "version": "8.15.2" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.datastorecluster", - "namespace": "65218", + "namespace": "74413", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "7737279e-51e9-4d90-a0d0-2c12dc4446bf", + "id": "cd777970-1a7d-4e59-99bf-7201c2c21f9e", "snapshot": true, - "version": "8.15.2" + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.datastorecluster", - "duration": 10772332, - "ingested": "2024-09-22T05:28:49Z", + "duration": 9107809, + "ingested": "2024-11-21T06:41:13Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-23128", - "id": "57723763cd1b4ff48e54a505de4ebe6c", + "hostname": "elastic-agent-72401", "ip": [ - "192.168.240.2", - "192.168.255.5" + "192.168.242.4", + "192.168.247.2" ], "mac": [ - "02-42-C0-A8-F0-02", - "02-42-C0-A8-FF-05" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F7-02" ], - "name": "elastic-agent-23128", + "name": "elastic-agent-72401", "os": { - "codename": "focal", - "family": "debian", "kernel": "3.10.0-1160.118.1.el7.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { @@ -73,7 +70,8 @@ "free_space": { "bytes": 0 }, + "id": "group-p8", "name": "DC0_POD0" } } -} \ No newline at end of file +} diff --git a/packages/vsphere/data_stream/host/fields/fields.yml b/packages/vsphere/data_stream/host/fields/fields.yml index 978b63902ce..dc3e7ce00c6 100644 --- a/packages/vsphere/data_stream/host/fields/fields.yml +++ b/packages/vsphere/data_stream/host/fields/fields.yml @@ -6,7 +6,7 @@ - name: alert.names type: keyword description: > - List of all the alerts on this Host. + List of all the alerts on this host. - name: name type: keyword # Reason to add as a dimension field: unique identifier of the host @@ -212,4 +212,4 @@ - name: warning.names type: keyword description: > - List of all the warnings on this Host. + List of all the warnings on this host. diff --git a/packages/vsphere/data_stream/host/sample_event.json b/packages/vsphere/data_stream/host/sample_event.json index 0a2c9c9e515..fe8e271eed8 100644 --- a/packages/vsphere/data_stream/host/sample_event.json +++ b/packages/vsphere/data_stream/host/sample_event.json @@ -1,62 +1,64 @@ { - "@timestamp": "2024-09-02T06:04:26.077Z", + "@timestamp": "2024-11-21T06:42:10.396Z", "agent": { - "ephemeral_id": "1acd79f0-0361-47dd-8d47-32064268713f", - "id": "958125fc-776b-4be3-b0ce-2762148f5f6d", - "name": "docker-fleet-agent", + "ephemeral_id": "347fdd67-dd63-4c29-a676-fa2c8a1ca1b2", + "id": "b307ccbf-41ad-4a93-88ee-2fb6914576bb", + "name": "elastic-agent-78670", "type": "metricbeat", - "version": "8.13.0" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.host", - "namespace": "50835", + "namespace": "36483", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "958125fc-776b-4be3-b0ce-2762148f5f6d", - "snapshot": false, - "version": "8.13.0" + "id": "b307ccbf-41ad-4a93-88ee-2fb6914576bb", + "snapshot": true, + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.host", - "duration": 32951837, - "ingested": "2024-09-02T06:04:38Z", + "duration": 1974395500, + "ingested": "2024-11-21T06:42:13Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "docker-fleet-agent", - "id": "8259e024976a406e8a54cdbffeb84fec", + "hostname": "elastic-agent-78670", "ip": [ - "192.168.253.6" + "192.168.242.4", + "192.168.249.2" ], "mac": [ - "02-42-C0-A8-FD-06" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F9-02" ], - "name": "docker-fleet-agent", + "name": "elastic-agent-78670", "os": { - "codename": "focal", - "family": "debian", "kernel": "3.10.0-1160.118.1.el7.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { "name": "host", - "period": 10000 + "period": 20000 }, "service": { "address": "https://svc-vsphere-metrics:8989/sdk", "type": "vsphere" }, + "tags": [ + "vsphere-host" + ], "vsphere": { "host": { "cpu": { @@ -71,6 +73,22 @@ "pct": 0.015 } }, + "datastore": { + "count": 1, + "names": "LocalDS_0" + }, + "disk": { + "read": { + "bytes": 7168 + }, + "total": { + "bytes": 4290560 + }, + "write": { + "bytes": 2251776 + } + }, + "id": "host-51", "memory": { "free": { "bytes": 2822230016 @@ -83,8 +101,52 @@ "pct": 0.343 } }, - "name": "DC0_H0", - "network_names": "VM Network" + "name": "DC0_C0_H2", + "network": { + "bandwidth": { + "received": { + "bytes": 1014784 + }, + "total": { + "bytes": 1775616 + }, + "transmitted": { + "bytes": 642048 + } + }, + "count": 3, + "names": [ + "DC0_DVPG0", + "DVS0-DVUplinks-10", + "VM Network" + ], + "packets": { + "multicast": { + "received": { + "count": 143 + }, + "total": { + "count": 148 + }, + "transmitted": { + "count": 5 + } + }, + "received": { + "count": 18207 + }, + "transmitted": { + "count": 8118 + } + } + }, + "network_names": [ + "DC0_DVPG0", + "DVS0-DVUplinks-10", + "VM Network" + ], + "status": "gray", + "uptime": 77229 } } -} \ No newline at end of file +} diff --git a/packages/vsphere/data_stream/network/sample_event.json b/packages/vsphere/data_stream/network/sample_event.json index fb7ac5ff7cb..946d6bc7aff 100644 --- a/packages/vsphere/data_stream/network/sample_event.json +++ b/packages/vsphere/data_stream/network/sample_event.json @@ -1,76 +1,74 @@ { - "@timestamp":"2024-09-22T21:01:42.635Z", - "agent":{ - "ephemeral_id":"b4116483-d4c6-4860-b93d-f0d8091cc838", - "id":"ff0ab35a-1abe-47a1-aee7-6d70362e4335", - "name":"docker-fleet-agent", - "type":"metricbeat", - "version":"8.15.2" - }, - "data_stream":{ - "dataset":"vsphere.network", - "namespace":"default", - "type":"metrics" - }, - "ecs":{ - "version":"8.11.0" - }, - "elastic_agent":{ - "id":"ff0ab35a-1abe-47a1-aee7-6d70362e4335", - "snapshot":true, - "version":"8.15.2" - }, - "event":{ - "agent_id_status":"verified", - "dataset":"vsphere.network", - "duration":10826519, - "ingested":"2024-09-22T21:01:43Z", - "module":"vsphere" - }, - "host":{ - "architecture":"x86_64", - "containerized":true, - "hostname":"elastic-agent-23128", - "id":"57723763cd1b4ff48e54a505de4ebe6c", - "ip":[ - "192.168.240.2", - "192.168.255.5" - ], - "mac":[ - "02-42-C0-A8-F0-02", - "02-42-C0-A8-FF-05" - ], - "name":"elastic-agent-23128", - "os":{ - "codename":"focal", - "family":"debian", - "kernel":"3.10.0-1160.118.1.el7.x86_64", - "name":"Ubuntu", - "platform":"ubuntu", - "type":"linux", - "version":"20.04.6 LTS (Focal Fossa)" - } - }, - "metricset":{ - "name":"network", - "period":20000 - }, - "service":{ - "address":"https://172.18.0.4:8989/sdk", - "type":"vsphere" - }, - "tags":[ - "vsphere-network" - ], - "vsphere":{ - "network":{ - "accessible":true, - "config":{ - "status":"green" - }, - "name":"VM Network", - "status":"green", - "type":"Network" - } - } -} \ No newline at end of file + "@timestamp": "2024-11-21T06:44:58.755Z", + "agent": { + "ephemeral_id": "5fb45395-31b5-476b-bc9f-ff885d033b7b", + "id": "aafddff2-0ad6-4601-b5be-bfacb507cb11", + "name": "elastic-agent-31986", + "type": "metricbeat", + "version": "8.17.0" + }, + "data_stream": { + "dataset": "vsphere.network", + "namespace": "97009", + "type": "metrics" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "aafddff2-0ad6-4601-b5be-bfacb507cb11", + "snapshot": true, + "version": "8.17.0" + }, + "event": { + "agent_id_status": "verified", + "dataset": "vsphere.network", + "duration": 10166398, + "ingested": "2024-11-21T06:45:01Z", + "module": "vsphere" + }, + "host": { + "architecture": "x86_64", + "containerized": true, + "hostname": "elastic-agent-31986", + "ip": [ + "192.168.240.2", + "192.168.242.4" + ], + "mac": [ + "02-42-C0-A8-F0-02", + "02-42-C0-A8-F2-04" + ], + "name": "elastic-agent-31986", + "os": { + "kernel": "3.10.0-1160.118.1.el7.x86_64", + "name": "Wolfi", + "platform": "wolfi", + "type": "linux", + "version": "20230201" + } + }, + "metricset": { + "name": "network", + "period": 20000 + }, + "service": { + "address": "https://svc-vsphere-metrics:8989/sdk", + "type": "vsphere" + }, + "tags": [ + "vsphere-network" + ], + "vsphere": { + "network": { + "accessible": true, + "config": { + "status": "green" + }, + "id": "network-7", + "name": "VM Network", + "status": "green", + "type": "Network" + } + } +} diff --git a/packages/vsphere/data_stream/resourcepool/sample_event.json b/packages/vsphere/data_stream/resourcepool/sample_event.json index 617c41fc650..b52d451befd 100644 --- a/packages/vsphere/data_stream/resourcepool/sample_event.json +++ b/packages/vsphere/data_stream/resourcepool/sample_event.json @@ -1,54 +1,51 @@ { - "@timestamp": "2024-10-21T09:31:54.201Z", + "@timestamp": "2024-11-21T06:45:47.037Z", "agent": { - "ephemeral_id": "f0138ef6-8fb9-4fb0-bf8f-002379e3e876", - "id": "93deaee2-10b7-426a-a4b2-e4aca31a244f", - "name": "elastic-agent-36652", + "ephemeral_id": "cf95a7c8-48b9-4810-8b93-63fece417dcb", + "id": "9fa1eac1-283d-4285-80d4-cb764c577733", + "name": "elastic-agent-76769", "type": "metricbeat", - "version": "8.15.2" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.resourcepool", - "namespace": "65031", + "namespace": "29674", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "93deaee2-10b7-426a-a4b2-e4aca31a244f", - "snapshot": false, - "version": "8.15.2" + "id": "9fa1eac1-283d-4285-80d4-cb764c577733", + "snapshot": true, + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.resourcepool", - "duration": 9922374, - "ingested": "2024-10-21T09:31:57Z", + "duration": 11393478, + "ingested": "2024-11-21T06:45:50Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-36652", - "id": "93db770e92a444c98362aee1860ae326", + "hostname": "elastic-agent-76769", "ip": [ - "192.168.243.2", - "192.168.247.4" + "192.168.242.4", + "192.168.243.2" ], "mac": [ - "02-42-C0-A8-F3-02", - "02-42-C0-A8-F7-04" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F3-02" ], - "name": "elastic-agent-36652", + "name": "elastic-agent-76769", "os": { - "codename": "focal", - "family": "debian", "kernel": "3.10.0-1160.118.1.el7.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { @@ -64,6 +61,7 @@ ], "vsphere": { "resourcepool": { + "id": "resgroup-27", "name": "Resources", "status": "green" } diff --git a/packages/vsphere/data_stream/virtualmachine/fields/fields.yml b/packages/vsphere/data_stream/virtualmachine/fields/fields.yml index d5dae125b79..25dfc578ac2 100644 --- a/packages/vsphere/data_stream/virtualmachine/fields/fields.yml +++ b/packages/vsphere/data_stream/virtualmachine/fields/fields.yml @@ -34,7 +34,7 @@ - name: os type: keyword description: > - Virtual machine Operating System name. + Virtual machine operating system name. - name: cpu.used.mhz type: long metric_type: gauge diff --git a/packages/vsphere/data_stream/virtualmachine/sample_event.json b/packages/vsphere/data_stream/virtualmachine/sample_event.json index 3910868bfda..003d1565acf 100644 --- a/packages/vsphere/data_stream/virtualmachine/sample_event.json +++ b/packages/vsphere/data_stream/virtualmachine/sample_event.json @@ -1,139 +1,115 @@ { - "@timestamp": "2024-09-02T09:44:14.128Z", + "@timestamp": "2024-11-21T06:46:37.132Z", "agent": { - "ephemeral_id": "2d60906e-b972-4981-a356-c57ccb79108b", - "id": "8ea73fb3-a0a0-4270-aad6-e88edb8b385a", - "name": "elastic-agent-55444", + "ephemeral_id": "b00e4654-522f-487c-b2fa-497e7e86e4a4", + "id": "ccc02bb5-23d2-4c33-88c5-04f806c20f18", + "name": "elastic-agent-81451", "type": "metricbeat", - "version": "8.16.0" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.virtualmachine", - "namespace": "64133", + "namespace": "25029", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "8ea73fb3-a0a0-4270-aad6-e88edb8b385a", + "id": "ccc02bb5-23d2-4c33-88c5-04f806c20f18", "snapshot": true, - "version": "8.16.0" + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.virtualmachine", - "duration": 52726937, - "ingested": "2024-09-02T09:44:17Z", + "duration": 46619423, + "ingested": "2024-11-21T06:46:40Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-55444", - "id": "e744630f9d4f43dc818e497d221bd0b2", + "hostname": "elastic-agent-81451", "ip": [ - "127.28.12.4", - "127.28.12.2" + "192.168.242.4", + "192.168.245.2" ], "mac": [ - "02-42-AC-12-00-04", - "02-42-AC-14-00-02" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F5-02" ], - "name": "elastic-agent-55444", + "name": "elastic-agent-81451", "os": { - "codename": "focal", - "family": "debian", - "kernel": "127.15.128.1-microsoft-standard-WSL2", - "name": "Ubuntu", - "platform": "ubuntu", + "kernel": "3.10.0-1160.118.1.el7.x86_64", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { "name": "virtualmachine", - "period": 10000 + "period": 20000 }, "service": { - "address": "https://localhost:8989/sdk", + "address": "https://svc-vsphere-metrics:8989/sdk", "type": "vsphere" }, + "tags": [ + "vsphere-virtualmachine" + ], "vsphere": { "virtualmachine": { - "name": "xt0nmfpv9", - "uptime": 5348978, - "status": "green", - "host": { - "id": "host-32", - "hostname": "localhost.com" - }, "cpu": { "free": { "mhz": 0 }, - "used": { - "mhz": 161 - }, "total": { "mhz": 0 + }, + "used": { + "mhz": 0 } }, - "network": { - "names": [ - "PROD_VCF_VMS" - ], - "count": 1 + "datastore": { + "count": 1, + "names": "LocalDS_0" + }, + "host": { + "hostname": "DC0_C0_H1", + "id": "host-43" }, + "id": "vm-62", "memory": { - "used": { + "free": { "guest": { - "bytes": 686817280 - }, - "host": { - "bytes": 29027729408 + "bytes": 33554432 } }, "total": { "guest": { - "bytes": 68719476736 + "bytes": 33554432 } }, - "free": { + "used": { "guest": { - "bytes": 68032659456 + "bytes": 0 + }, + "host": { + "bytes": 0 } } }, - "network_names": [ - "PROD_VCF_VMS" - ], - "datastore": { + "name": "DC0_C0_RP0_VM0", + "network": { "count": 1, - "names": [ - "VxRailtoup-Virtual-Datastore-bc1d-5aa310fb" - ] + "names": "DC0_DVPG0" }, - "os": "CentOS 4/5/6/7 (64-bit)", - "snapshot": { - "info": [ - { - "id": 1, - "name": "VM Snapshot 7%2f3%2f2024, 4:01:21 PM", - "description": "Created to demo", - "createtime": "2024-07-03T20:01:34.329Z", - "state": "poweredOn" - }, - { - "createtime": "2024-07-05T23:35:40.859Z", - "state": "poweredOn", - "id": 2, - "name": "VM Snapshot 7%2f5%2f2024, 7:35:37 PM", - "description": "backup" - } - ], - "count": 2 - } + "network_names": "DC0_DVPG0", + "os": "otherGuest", + "status": "green", + "uptime": 0 } } } diff --git a/packages/vsphere/docs/README.md b/packages/vsphere/docs/README.md index c4ef5e6e28b..d86b886042a 100644 --- a/packages/vsphere/docs/README.md +++ b/packages/vsphere/docs/README.md @@ -16,51 +16,47 @@ The vSphere integration collects logs and metrics. Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by vSphere as integration is `log`. -Metrics give you insight into the statistics of the vSphere. The `Metric` data streams collected by the vSphere integration are `cluster`, `datastore`, `datastorecluster`, `host`, `network`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. +Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `datastorecluster`, `host`, `network`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. + +Data Streams: + +- **`log`**: This data stream collects logs generated by VMware vSphere using a syslog daemon. +- **`cluster`**: This data stream collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. +- **`datastore`**: This data stream gathers datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. +- **`datastorecluster`**: This data stream gathers metrics for datastore clusters from VMware vSphere, including statistics like cluster capacity and available free space. Additionally, it provides information about the individual datastores that comprise the cluster. +- **`host`**: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. +- **`resourcepool`**: This data stream collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. +- **`virtualmachine`**: This data stream gathers virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. +- **`network`**: This data stream gathers metrics and status information related to VMware vSphere networks, including network accessibility, connected hosts and virtual machines, configuration health, and network type. + +## Important Note +- Users can monitor and see the log inside the ingested documents for vSphere in the `logs-*` index pattern from `Discover`, and for metrics, the index pattern is `metrics-*`. +- **Supported Periods**: + - Real-time data collection – An ESXi Server collects data for each performance counter every 20 seconds by default. + - The Datastore and Host datastreams support performance data collection using the vSphere performance API. + - Since the performance API has usage restrictions based on data collection intervals, users should ensure that the period is configured optimally to receive real-time data. Users can still collect summary metrics if performance metrics are not supported for the configured instance. + - Period configuration can be determined based on the [Data Collection Intervals](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html) and [Data Collection Levels](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html). -Data streams: - -- **`log`**: Collects logs generated by VMware vSphere using a syslog daemon. -- **`cluster`**: Collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. -- **`datastore`**: Gathers datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. -- **`datastorecluster`**: Gathers metrics for datastore clusters from VMware vSphere, including statistics like cluster capacity and available free space. Additionally, it provides information about the individual datastores that comprise the cluster. -- **`host`**: Collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. -- **`resourcepool`**: Collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. -- **`virtualmachine`**: Gathers virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. -- **`network`**: Gathers metrics and status information related to VMware vSphere networks, including network accessibility, connected hosts and virtual machines, configuration health, and network type. - -Note: - -Users can monitor and check the log inside the ingested documents for vSphere in the `logs-*` index pattern from `Discover`, and for metrics, the index pattern is `metrics-*`. +## Prerequisites -## Supported periods - -- Real-time data collection: An ESXi Server collects data for each performance counter every 20 seconds by default. -- The `datastore` and `host` data streams support performance data collection using the vSphere performance API. -- Since the performance API has usage restrictions based on data collection intervals, make sure that the period is configured optimally to receive real-time data. You can still collect summary metrics if performance metrics are not supported for the configured instance. -- Period configuration can be determined based on the [Data Collection Intervals](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html) and [Data Collection Levels](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html). +You can store and search your data using Elasticsearch and visualize and manage it with Kibana. We recommend using our hosted Elasticsearch Service on Elastic Cloud or self-managing the Elastic Stack on your own hardware. ## Compatibility This integration has been tested and verified against VMware ESXi and vCenter version 7.0.3. -## Prerequisites - -You can store and search your data using Elasticsearch and visualize and manage it with Kibana. We recommend using our hosted Elasticsearch Service on Elastic Cloud or self-managing the Elastic Stack on your own hardware. - ## Setup -For step-by-step instructions on how to set up an integration, check the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. +For step-by-step instructions on how to set up an integration, see the [Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide. -## Logs +## Logs reference -The `log` data stream collects logs generated by VMware vSphere using a syslog daemon. +### Log -To collect logs, follow these steps: +This is the `log` data stream. This data stream collects logs generated by VMware vSphere using a syslog daemon. -1. Configure the listening host/IP address (default: localhost) and host port (default: 9525) in the integration. - -1. Configure vSphere to send logs to a remote syslog host and provide the configured hostname/IP and port of the Elastic Agent host. +Note: +- To collect logs, a syslog daemon is used. First, you must configure the listening host/IP address (default: localhost) and host port (default: 9525) in the integration. Then, configure vSphere to send logs to a remote syslog host and provide the configured hostname/IP and port of the Elastic Agent host. An example event for `log` looks as following: @@ -141,7 +137,7 @@ An example event for `log` looks as following: **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. **Exported fields** @@ -165,68 +161,65 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | vsphere.log.file.path | | keyword | -## Metrics +## Metrics reference -To access the metrics, provide the URL in the **Add Integration** page of the vSphere package. +Note: +- To access the metrics, provide the URL in the "Add Integration" page of the vSphere package. ### Cluster - Clusters in vSphere represent a group of ESXi hosts working together to optimize resource allocation, ensure high availability, and manage workloads efficiently. An example event for `cluster` looks as following: ```json { - "@timestamp": "2024-09-19T05:44:00.800Z", + "@timestamp": "2024-11-21T06:39:19.834Z", "agent": { - "ephemeral_id": "676a770b-a207-4fec-99d4-e82377578711", - "id": "6b430ae3-0bdb-4d5c-b60d-a02f54e770e5", - "name": "elastic-agent-47605", + "ephemeral_id": "f41470c3-9bf0-4217-baf6-73383f35fcff", + "id": "6b90cdc8-92f4-486f-9b8a-02a0aca316a6", + "name": "elastic-agent-81320", "type": "metricbeat", - "version": "8.15.2" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.cluster", - "namespace": "93141", + "namespace": "35689", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "6b430ae3-0bdb-4d5c-b60d-a02f54e770e5", + "id": "6b90cdc8-92f4-486f-9b8a-02a0aca316a6", "snapshot": true, - "version": "8.15.2" + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.cluster", - "duration": 17059144, - "ingested": "2024-09-19T05:44:03Z", + "duration": 33840300, + "ingested": "2024-11-21T06:39:22Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-47605", - "id": "57723763cd1b4ff48e54a505de4ebe6c", + "hostname": "elastic-agent-81320", "ip": [ - "192.168.244.4", - "192.168.245.2" + "192.168.242.4", + "192.168.243.2" ], "mac": [ - "02-42-C0-A8-F4-04", - "02-42-C0-A8-F5-02" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F3-02" ], - "name": "elastic-agent-47605", + "name": "elastic-agent-81320", "os": { - "codename": "focal", - "family": "debian", - "kernel": "4.18.0-348.7.1.el8_5.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "kernel": "3.10.0-1160.118.1.el7.x86_64", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { @@ -254,12 +247,13 @@ An example event for `cluster` looks as following: "DC0_C0_H2" ] }, + "id": "domain-c28", "name": "DC0_C0", "network": { "count": 3, "names": [ "DC0_DVPG0", - "DVS0-DVUplinks-9", + "DVS0-DVUplinks-10", "VM Network" ] } @@ -270,7 +264,7 @@ An example event for `cluster` looks as following: **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. **Exported fields** @@ -289,7 +283,7 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | data_stream.type | Data stream type. | constant_keyword | | | host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | -| vsphere.cluster.alert.names | List of all the alerts on this Cluster. | keyword | | +| vsphere.cluster.alert.names | List of all the alerts on this cluster. | keyword | | | vsphere.cluster.das_config.admission.control.enabled | Indicates whether strict admission control is enabled. | boolean | | | vsphere.cluster.das_config.enabled | Indicates whether vSphere HA feature is enabled. | boolean | | | vsphere.cluster.datastore.count | Number of datastores associated with the cluster. | long | gauge | @@ -306,100 +300,89 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | vsphere.cluster.triggered_alarms.name | Name of the alarm. | keyword | | | vsphere.cluster.triggered_alarms.status | Status of the alarm. | keyword | | | vsphere.cluster.triggered_alarms.triggered_time | Time when the alarm was triggered. | date | | -| vsphere.cluster.warning.names | List of all the warnings on this Cluster. | keyword | | +| vsphere.cluster.warning.names | List of all the warnings on this cluster. | keyword | | ### Datastore -This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. +This is `datastore` data stream. This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. An example event for `datastore` looks as following: ```json { - "@timestamp": "2024-09-02T10:04:25.122Z", + "@timestamp": "2024-11-21T06:40:14.950Z", "agent": { - "ephemeral_id": "4da294a3-ad54-47f4-92c7-544e1356a0d8", - "id": "b01ab3cf-51ad-4c4d-87bd-fc2d4aa59d8a", - "name": "elastic-agent-76236", + "ephemeral_id": "80421da7-699d-4ad1-96a3-56d7aa430190", + "id": "482796dd-401d-4568-8c52-7f9036322873", + "name": "elastic-agent-36929", "type": "metricbeat", - "version": "8.16.0" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.datastore", - "namespace": "86691", + "namespace": "35376", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "b01ab3cf-51ad-4c4d-87bd-fc2d4aa59d8a", + "id": "482796dd-401d-4568-8c52-7f9036322873", "snapshot": true, - "version": "8.16.0" + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.datastore", - "duration": 190147614, - "ingested": "2024-09-02T10:04:27Z", + "duration": 48985279, + "ingested": "2024-11-21T06:40:17Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-76236", - "id": "e744630f9d4f43dc818e497d221bd0b2", + "hostname": "elastic-agent-36929", "ip": [ - "172.18.0.4", - "172.21.0.2" + "192.168.242.4", + "192.168.245.2" ], "mac": [ - "02-42-AC-12-00-04", - "02-42-AC-15-00-02" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F5-02" ], - "name": "elastic-agent-76236", + "name": "elastic-agent-36929", "os": { - "codename": "focal", - "family": "debian", - "kernel": "5.15.153.1-microsoft-standard-WSL2", - "name": "Ubuntu", - "platform": "ubuntu", + "kernel": "3.10.0-1160.118.1.el7.x86_64", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { "name": "datastore", - "period": 10000 + "period": 20000 }, "service": { "address": "https://svc-vsphere-metrics:8989/sdk", "type": "vsphere" }, + "tags": [ + "vsphere-datastore" + ], "vsphere": { "datastore": { - "disk": { - "capacity": { - "usage": { - "bytes": 520505786368 - }, - "bytes": 1610344300544 - }, - "provisioned": { - "bytes": 520505786368 - } - }, "capacity": { "free": { - "bytes": 37120094208 + "bytes": 10952166604800 }, "total": { - "bytes": 74686664704 + "bytes": 10995116277760 }, "used": { - "bytes": 37566570496, - "pct": 0.502988996026061 + "bytes": 42949672960, + "pct": 0.004 } }, "fstype": "OTHER", @@ -407,10 +390,8 @@ An example event for `datastore` looks as following: "count": 1, "names": "DC0_H0" }, + "id": "datastore-53", "name": "LocalDS_0", - "read": { - "bytes": 1024 - }, "status": "green", "vm": { "count": 4, @@ -420,9 +401,6 @@ An example event for `datastore` looks as following: "DC0_H0_VM0", "DC0_H0_VM1" ] - }, - "write": { - "bytes": 450560 } } } @@ -431,7 +409,7 @@ An example event for `datastore` looks as following: **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. **Exported fields** @@ -455,7 +433,7 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | host.os.build | OS build information. | keyword | | | | host.os.codename | OS codename, if any. | keyword | | | | service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | | -| vsphere.datastore.alert.names | List of all the alerts on this Datastore. | keyword | | | +| vsphere.datastore.alert.names | List of all the alerts on this datastore. | keyword | | | | vsphere.datastore.capacity.free.bytes | Free bytes of the datastore. | long | byte | gauge | | vsphere.datastore.capacity.total.bytes | Total bytes of the datastore. | long | byte | gauge | | vsphere.datastore.capacity.used.bytes | Used bytes of the datastore. | long | byte | gauge | @@ -482,7 +460,7 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | vsphere.datastore.write.bytes | Rate of writing data to the datastore. | long | byte | gauge | -### Datastore cluster +### Datastore Cluster Datastore clusters in vSphere group multiple datastores for optimized management and automated load balancing, ensuring efficient storage utilization and simplified administration. @@ -490,56 +468,53 @@ An example event for `datastorecluster` looks as following: ```json { - "@timestamp": "2024-09-22T05:28:46.315Z", + "@timestamp": "2024-11-21T06:41:10.164Z", "agent": { - "ephemeral_id": "8b7e9ea4-0517-4e98-a795-b6fe529f4a2f", - "id": "7737279e-51e9-4d90-a0d0-2c12dc4446bf", - "name": "elastic-agent-23128", + "ephemeral_id": "dca752ad-1ce6-40f5-8a40-c25947763083", + "id": "cd777970-1a7d-4e59-99bf-7201c2c21f9e", + "name": "elastic-agent-72401", "type": "metricbeat", - "version": "8.15.2" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.datastorecluster", - "namespace": "65218", + "namespace": "74413", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "7737279e-51e9-4d90-a0d0-2c12dc4446bf", + "id": "cd777970-1a7d-4e59-99bf-7201c2c21f9e", "snapshot": true, - "version": "8.15.2" + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.datastorecluster", - "duration": 10772332, - "ingested": "2024-09-22T05:28:49Z", + "duration": 9107809, + "ingested": "2024-11-21T06:41:13Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-23128", - "id": "57723763cd1b4ff48e54a505de4ebe6c", + "hostname": "elastic-agent-72401", "ip": [ - "192.168.240.2", - "192.168.255.5" + "192.168.242.4", + "192.168.247.2" ], "mac": [ - "02-42-C0-A8-F0-02", - "02-42-C0-A8-FF-05" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F7-02" ], - "name": "elastic-agent-23128", + "name": "elastic-agent-72401", "os": { - "codename": "focal", - "family": "debian", "kernel": "3.10.0-1160.118.1.el7.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { @@ -564,6 +539,7 @@ An example event for `datastorecluster` looks as following: "free_space": { "bytes": 0 }, + "id": "group-p8", "name": "DC0_POD0" } } @@ -572,7 +548,7 @@ An example event for `datastorecluster` looks as following: **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. **Exported fields** @@ -593,7 +569,7 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | event.module | Event module | constant_keyword | | | | host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | | service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | | -| vsphere.datastorecluster.alert.names | List of all the alerts on this Datastore Cluster. | keyword | | | +| vsphere.datastorecluster.alert.names | List of all the alerts on this datastore cluster. | keyword | | | | vsphere.datastorecluster.capacity.bytes | Total capacity of this storage pod, in bytes. | long | byte | gauge | | vsphere.datastorecluster.datastore.count | Number of datastores in the datastore cluster. | long | | gauge | | vsphere.datastorecluster.datastore.names | List of all the datastore names associated with the datastore cluster. | keyword | | | @@ -606,75 +582,77 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | vsphere.datastorecluster.triggered_alarms.name | Name of the alarm. | keyword | | | | vsphere.datastorecluster.triggered_alarms.status | Status of the alarm. | keyword | | | | vsphere.datastorecluster.triggered_alarms.triggered_time | Time when the alarm was triggered. | date | | | -| vsphere.datastorecluster.warning.names | List of all the warnings on this Datastore Cluster. | keyword | | | +| vsphere.datastorecluster.warning.names | List of all the warnings on this datastore cluster. | keyword | | | ### Host -This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. +This is the `host` data stream. This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. An example event for `host` looks as following: ```json { - "@timestamp": "2024-09-02T06:04:26.077Z", + "@timestamp": "2024-11-21T06:42:10.396Z", "agent": { - "ephemeral_id": "1acd79f0-0361-47dd-8d47-32064268713f", - "id": "958125fc-776b-4be3-b0ce-2762148f5f6d", - "name": "docker-fleet-agent", + "ephemeral_id": "347fdd67-dd63-4c29-a676-fa2c8a1ca1b2", + "id": "b307ccbf-41ad-4a93-88ee-2fb6914576bb", + "name": "elastic-agent-78670", "type": "metricbeat", - "version": "8.13.0" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.host", - "namespace": "50835", + "namespace": "36483", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "958125fc-776b-4be3-b0ce-2762148f5f6d", - "snapshot": false, - "version": "8.13.0" + "id": "b307ccbf-41ad-4a93-88ee-2fb6914576bb", + "snapshot": true, + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.host", - "duration": 32951837, - "ingested": "2024-09-02T06:04:38Z", + "duration": 1974395500, + "ingested": "2024-11-21T06:42:13Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "docker-fleet-agent", - "id": "8259e024976a406e8a54cdbffeb84fec", + "hostname": "elastic-agent-78670", "ip": [ - "192.168.253.6" + "192.168.242.4", + "192.168.249.2" ], "mac": [ - "02-42-C0-A8-FD-06" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F9-02" ], - "name": "docker-fleet-agent", + "name": "elastic-agent-78670", "os": { - "codename": "focal", - "family": "debian", "kernel": "3.10.0-1160.118.1.el7.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { "name": "host", - "period": 10000 + "period": 20000 }, "service": { "address": "https://svc-vsphere-metrics:8989/sdk", "type": "vsphere" }, + "tags": [ + "vsphere-host" + ], "vsphere": { "host": { "cpu": { @@ -689,6 +667,22 @@ An example event for `host` looks as following: "pct": 0.015 } }, + "datastore": { + "count": 1, + "names": "LocalDS_0" + }, + "disk": { + "read": { + "bytes": 7168 + }, + "total": { + "bytes": 4290560 + }, + "write": { + "bytes": 2251776 + } + }, + "id": "host-51", "memory": { "free": { "bytes": 2822230016 @@ -701,8 +695,52 @@ An example event for `host` looks as following: "pct": 0.343 } }, - "name": "DC0_H0", - "network_names": "VM Network" + "name": "DC0_C0_H2", + "network": { + "bandwidth": { + "received": { + "bytes": 1014784 + }, + "total": { + "bytes": 1775616 + }, + "transmitted": { + "bytes": 642048 + } + }, + "count": 3, + "names": [ + "DC0_DVPG0", + "DVS0-DVUplinks-10", + "VM Network" + ], + "packets": { + "multicast": { + "received": { + "count": 143 + }, + "total": { + "count": 148 + }, + "transmitted": { + "count": 5 + } + }, + "received": { + "count": 18207 + }, + "transmitted": { + "count": 8118 + } + } + }, + "network_names": [ + "DC0_DVPG0", + "DVS0-DVUplinks-10", + "VM Network" + ], + "status": "gray", + "uptime": 77229 } } } @@ -710,7 +748,7 @@ An example event for `host` looks as following: **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. **Exported fields** @@ -731,7 +769,7 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | event.module | Event module | constant_keyword | | | | host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | | service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | | -| vsphere.host.alert.names | List of all the alerts on this Host. | keyword | | | +| vsphere.host.alert.names | List of all the alerts on this host. | keyword | | | | vsphere.host.cpu.free.mhz | Free CPU in MHz. | long | | gauge | | vsphere.host.cpu.total.mhz | Total CPU in MHz. | long | | counter | | vsphere.host.cpu.used.mhz | Used CPU in MHz. | long | | gauge | @@ -777,67 +815,64 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | vsphere.host.uptime | The total uptime of a host in seconds within the vSphere environment. | long | | gauge | | vsphere.host.vm.count | Number of virtual machines on the host. | long | | gauge | | vsphere.host.vm.names | List of all the VM names. | keyword | | | -| vsphere.host.warning.names | List of all the warnings on this Host. | keyword | | | +| vsphere.host.warning.names | List of all the warnings on this host. | keyword | | | ### Network -This data stream gathers metrics and status information related to VMware vSphere networks, including network accessibility, connected hosts and virtual machines, configuration health, and network type. +This is the `network` data stream. This data stream gathers metrics and status information related to VMware vSphere networks, including network accessibility, connected hosts and virtual machines, configuration health, and network type. An example event for `network` looks as following: ```json { - "@timestamp": "2024-09-22T21:01:42.635Z", + "@timestamp": "2024-11-21T06:44:58.755Z", "agent": { - "ephemeral_id": "b4116483-d4c6-4860-b93d-f0d8091cc838", - "id": "ff0ab35a-1abe-47a1-aee7-6d70362e4335", - "name": "docker-fleet-agent", + "ephemeral_id": "5fb45395-31b5-476b-bc9f-ff885d033b7b", + "id": "aafddff2-0ad6-4601-b5be-bfacb507cb11", + "name": "elastic-agent-31986", "type": "metricbeat", - "version": "8.15.2" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.network", - "namespace": "default", + "namespace": "97009", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "ff0ab35a-1abe-47a1-aee7-6d70362e4335", + "id": "aafddff2-0ad6-4601-b5be-bfacb507cb11", "snapshot": true, - "version": "8.15.2" + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.network", - "duration": 10826519, - "ingested": "2024-09-22T21:01:43Z", + "duration": 10166398, + "ingested": "2024-11-21T06:45:01Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-23128", - "id": "57723763cd1b4ff48e54a505de4ebe6c", + "hostname": "elastic-agent-31986", "ip": [ "192.168.240.2", - "192.168.255.5" + "192.168.242.4" ], "mac": [ "02-42-C0-A8-F0-02", - "02-42-C0-A8-FF-05" + "02-42-C0-A8-F2-04" ], - "name": "elastic-agent-23128", + "name": "elastic-agent-31986", "os": { - "codename": "focal", - "family": "debian", "kernel": "3.10.0-1160.118.1.el7.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { @@ -845,7 +880,7 @@ An example event for `network` looks as following: "period": 20000 }, "service": { - "address": "https://172.18.0.4:8989/sdk", + "address": "https://svc-vsphere-metrics:8989/sdk", "type": "vsphere" }, "tags": [ @@ -857,6 +892,7 @@ An example event for `network` looks as following: "config": { "status": "green" }, + "id": "network-7", "name": "VM Network", "status": "green", "type": "Network" @@ -867,7 +903,7 @@ An example event for `network` looks as following: **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. **Exported fields** @@ -910,63 +946,59 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e ### Resourcepool - -Resource pools in vSphere enable the allocation and management of CPU and memory resources across groups of virtual machines. +Resource pools in vSphere allow for the allocation and management of CPU and memory resources across groups of virtual machines. An example event for `resourcepool` looks as following: ```json { - "@timestamp": "2024-10-21T09:31:54.201Z", + "@timestamp": "2024-11-21T06:45:47.037Z", "agent": { - "ephemeral_id": "f0138ef6-8fb9-4fb0-bf8f-002379e3e876", - "id": "93deaee2-10b7-426a-a4b2-e4aca31a244f", - "name": "elastic-agent-36652", + "ephemeral_id": "cf95a7c8-48b9-4810-8b93-63fece417dcb", + "id": "9fa1eac1-283d-4285-80d4-cb764c577733", + "name": "elastic-agent-76769", "type": "metricbeat", - "version": "8.15.2" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.resourcepool", - "namespace": "65031", + "namespace": "29674", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "93deaee2-10b7-426a-a4b2-e4aca31a244f", - "snapshot": false, - "version": "8.15.2" + "id": "9fa1eac1-283d-4285-80d4-cb764c577733", + "snapshot": true, + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.resourcepool", - "duration": 9922374, - "ingested": "2024-10-21T09:31:57Z", + "duration": 11393478, + "ingested": "2024-11-21T06:45:50Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-36652", - "id": "93db770e92a444c98362aee1860ae326", + "hostname": "elastic-agent-76769", "ip": [ - "192.168.243.2", - "192.168.247.4" + "192.168.242.4", + "192.168.243.2" ], "mac": [ - "02-42-C0-A8-F3-02", - "02-42-C0-A8-F7-04" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F3-02" ], - "name": "elastic-agent-36652", + "name": "elastic-agent-76769", "os": { - "codename": "focal", - "family": "debian", "kernel": "3.10.0-1160.118.1.el7.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { @@ -982,6 +1014,7 @@ An example event for `resourcepool` looks as following: ], "vsphere": { "resourcepool": { + "id": "resgroup-27", "name": "Resources", "status": "green" } @@ -991,7 +1024,7 @@ An example event for `resourcepool` looks as following: **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. **Exported fields** @@ -1042,147 +1075,123 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e ### Virtual Machine -This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. +This is the `virtualmachine` data stream. This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. An example event for `virtualmachine` looks as following: ```json { - "@timestamp": "2024-09-02T09:44:14.128Z", + "@timestamp": "2024-11-21T06:46:37.132Z", "agent": { - "ephemeral_id": "2d60906e-b972-4981-a356-c57ccb79108b", - "id": "8ea73fb3-a0a0-4270-aad6-e88edb8b385a", - "name": "elastic-agent-55444", + "ephemeral_id": "b00e4654-522f-487c-b2fa-497e7e86e4a4", + "id": "ccc02bb5-23d2-4c33-88c5-04f806c20f18", + "name": "elastic-agent-81451", "type": "metricbeat", - "version": "8.16.0" + "version": "8.17.0" }, "data_stream": { "dataset": "vsphere.virtualmachine", - "namespace": "64133", + "namespace": "25029", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "8ea73fb3-a0a0-4270-aad6-e88edb8b385a", + "id": "ccc02bb5-23d2-4c33-88c5-04f806c20f18", "snapshot": true, - "version": "8.16.0" + "version": "8.17.0" }, "event": { "agent_id_status": "verified", "dataset": "vsphere.virtualmachine", - "duration": 52726937, - "ingested": "2024-09-02T09:44:17Z", + "duration": 46619423, + "ingested": "2024-11-21T06:46:40Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-55444", - "id": "e744630f9d4f43dc818e497d221bd0b2", + "hostname": "elastic-agent-81451", "ip": [ - "127.28.12.4", - "127.28.12.2" + "192.168.242.4", + "192.168.245.2" ], "mac": [ - "02-42-AC-12-00-04", - "02-42-AC-14-00-02" + "02-42-C0-A8-F2-04", + "02-42-C0-A8-F5-02" ], - "name": "elastic-agent-55444", + "name": "elastic-agent-81451", "os": { - "codename": "focal", - "family": "debian", - "kernel": "127.15.128.1-microsoft-standard-WSL2", - "name": "Ubuntu", - "platform": "ubuntu", + "kernel": "3.10.0-1160.118.1.el7.x86_64", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { "name": "virtualmachine", - "period": 10000 + "period": 20000 }, "service": { - "address": "https://localhost:8989/sdk", + "address": "https://svc-vsphere-metrics:8989/sdk", "type": "vsphere" }, + "tags": [ + "vsphere-virtualmachine" + ], "vsphere": { "virtualmachine": { - "name": "xt0nmfpv9", - "uptime": 5348978, - "status": "green", - "host": { - "id": "host-32", - "hostname": "localhost.com" - }, "cpu": { "free": { "mhz": 0 }, - "used": { - "mhz": 161 - }, "total": { "mhz": 0 + }, + "used": { + "mhz": 0 } }, - "network": { - "names": [ - "PROD_VCF_VMS" - ], - "count": 1 + "datastore": { + "count": 1, + "names": "LocalDS_0" + }, + "host": { + "hostname": "DC0_C0_H1", + "id": "host-43" }, + "id": "vm-62", "memory": { - "used": { + "free": { "guest": { - "bytes": 686817280 - }, - "host": { - "bytes": 29027729408 + "bytes": 33554432 } }, "total": { "guest": { - "bytes": 68719476736 + "bytes": 33554432 } }, - "free": { + "used": { "guest": { - "bytes": 68032659456 + "bytes": 0 + }, + "host": { + "bytes": 0 } } }, - "network_names": [ - "PROD_VCF_VMS" - ], - "datastore": { + "name": "DC0_C0_RP0_VM0", + "network": { "count": 1, - "names": [ - "VxRailtoup-Virtual-Datastore-bc1d-5aa310fb" - ] + "names": "DC0_DVPG0" }, - "os": "CentOS 4/5/6/7 (64-bit)", - "snapshot": { - "info": [ - { - "id": 1, - "name": "VM Snapshot 7%2f3%2f2024, 4:01:21 PM", - "description": "Created to demo", - "createtime": "2024-07-03T20:01:34.329Z", - "state": "poweredOn" - }, - { - "createtime": "2024-07-05T23:35:40.859Z", - "state": "poweredOn", - "id": 2, - "name": "VM Snapshot 7%2f5%2f2024, 7:35:37 PM", - "description": "backup" - } - ], - "count": 2 - } + "network_names": "DC0_DVPG0", + "os": "otherGuest", + "status": "green", + "uptime": 0 } } } @@ -1190,7 +1199,7 @@ An example event for `virtualmachine` looks as following: **ECS Field Reference** -Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. **Exported fields** @@ -1232,7 +1241,7 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e | vsphere.virtualmachine.network.count | Number of networks associated to this virtualmachine. | long | | gauge | | vsphere.virtualmachine.network.names | Names of the networks associated to this virtualmachine. | keyword | | | | vsphere.virtualmachine.network_names | Network names. | keyword | | | -| vsphere.virtualmachine.os | Virtual machine Operating System name. | keyword | | | +| vsphere.virtualmachine.os | Virtual machine operating system name. | keyword | | | | vsphere.virtualmachine.snapshot.count | The number of snapshots of this virtualmachine. | long | | gauge | | vsphere.virtualmachine.snapshot.info.createtime | Snapshot creation time. | date | | | | vsphere.virtualmachine.snapshot.info.description | Snapshot description. | keyword | | | @@ -1252,5 +1261,4 @@ Refer to the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/e ## Troubleshoot -If you encounter issues during the setup or usage of the VMware vSphere integration, in particular with Data Collection Intervals, with agent error messages such as **`Failed to query performance metrics: ServerFaultCode: A specified parameter was not correct: querySpec.interval`**, refer to -[Supported periods](#supported-periods) for guidance and resolution. \ No newline at end of file +If you encounter any issues during the setup or usage of the VMware vSphere integration, particularly with regards to Data Collection Intervals, with agent error messages such as **`Failed to query performance metrics: ServerFaultCode: A specified parameter was not correct: querySpec.interval`**, please refer to the mentioned Important Notes/Supported Periods for guidance and resolution.