[Enhancement] Sensors #255

M0NsTeRRR · 2021-03-11T11:58:20Z

Metrics added

vmware_host_sensor_state
vmware_host_sensor_fan
vmware_host_sensor_temperature
vmware_host_sensor_power_voltage
vmware_host_sensor_power_current
vmware_host_sensor_power_watt
vmware_host_sensor_redundancy

Some more metrics can be added but since some informations are shown with some specific models feel free to give me an extract with this script : https://gist.github.com/M0NsTeRRR/5fe8e4e44d4a6b0c1fecf2b88bd0e67c

Breaking changes

runtime.healthSystemRuntime.systemHealthInfo.numericSensorInfo is not an alarm, i've handled it as a standard metrics
runtime.healthSystemRuntime.hardwareStatusInfo.cpuStatusInfo and runtime.healthSystemRuntime.hardwareStatusInfo.memoryStatusInfo are removed they return me empty values all the time so I can't implement it until someone give me some datas.
You can give me the data with this script : https://gist.github.com/M0NsTeRRR/5fe8e4e44d4a6b0c1fecf2b88bd0e67c
don't forget to replace line 10 with one of the two queries.

Enhancement

I think we should add in config.py under collect_only an option like sensors: False because many peoples will use sensors from ipmi-exporter and it generates a lot of datas, what do you think ?

Example

# HELP vmware_host_sensor_state VMWare sensor state value (0=red / 1=yellow / 2=green / 3=unknown) labeled by sensor name and type from the host.
# TYPE vmware_host_sensor_state gauge
vmware_host_sensor_state{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Processor 1 Temp",type="temperature"} 2.0
vmware_host_sensor_state{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="System Board 1 Pwr Consumption",type="systemBoard"} 2.0
# HELP vmware_host_sensor_fan VMWare sensor fan speed value in RPM labeled by sensor name from the host.
# TYPE vmware_host_sensor_fan gauge
vmware_host_sensor_fan{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="System Board 1 Fan5B"} 3960.0
vmware_host_sensor_fan{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="System Board 1 Fan5A"} 5280.0
# HELP vmware_host_sensor_temperature VMWare sensor temperature value in degree C labeled by sensor name from the host.
# TYPE vmware_host_sensor_temperature gauge
vmware_host_sensor_temperature{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Processor 1 Temp"} 45.0
vmware_host_sensor_temperature{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="System Board 1 Inlet Temp"} 21.0
# HELP vmware_host_sensor_power_voltage VMWare sensor power voltage value in volt labeled by sensor name from the host.
# TYPE vmware_host_sensor_power_voltage gauge
vmware_host_sensor_power_voltage{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Power Supply 2 Voltage 2"} 236.0
vmware_host_sensor_power_voltage{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Power Supply 1 Voltage 1"} 236.0
# HELP vmware_host_sensor_power_current VMWare sensor power current value in amp labeled by sensor name from the host.
# TYPE vmware_host_sensor_power_current gauge
vmware_host_sensor_power_current{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Power Supply 2 Current 2"} 0.2
vmware_host_sensor_power_current{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Power Supply 1 Current 1"} 0.2
# HELP vmware_host_sensor_power_watt VMWare sensor power watt value in watt labeled by sensor name from the host.
# TYPE vmware_host_sensor_power_watt gauge
vmware_host_sensor_power_watt{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="System Board 1 Pwr Consumption"} 70.0
vmware_host_sensor_power_watt{cluster_name="HA",dc_name="Homelab",host_name="server4.unicornafk.fr",name="Power Supply 2 PS2 Input Power"} 0.0
# HELP vmware_host_sensor_redundancy VMWare sensor redundancy value (1=ok / 0=ko) labeled by sensor name from the host.
# TYPE vmware_host_sensor_redundancy gauge
vmware_host_sensor_redundancy{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="System Board 1 Fan Redundancy 0"} 1.0
vmware_host_sensor_redundancy{cluster_name="HA",dc_name="Homelab",host_name="server4.unicornafk.fr",name="Cooling Unit 1 Fan Redundancy 0"} 1.0

Tested on Dell R320 & Intel R1208WTTGSR

vmware_host_sensor_state vmware_host_sensor_fan vmware_host_sensor_temperature vmware_host_sensor_power_voltage vmware_host_sensor_power_current vmware_host_sensor_power_watt vmware_host_sensor_redundancy

codecov-io · 2021-03-11T12:09:48Z

Codecov Report

Merging #255 (301499d) into main (802240c) will increase coverage by 1.50%.
The diff coverage is 91.30%.

@@            Coverage Diff             @@
##             main     #255      +/-   ##
==========================================
+ Coverage   80.29%   81.80%   +1.50%     
==========================================
  Files           4        4              
  Lines         873      874       +1     
==========================================
+ Hits          701      715      +14     
+ Misses        172      159      -13

Impacted Files	Coverage Δ
vmware_exporter/helpers.py	`75.00% <0.00%> (+13.80%)`	⬆️
vmware_exporter/__init__.py	`100.00% <100.00%> (ø)`
vmware_exporter/vmware_exporter.py	`81.48% <100.00%> (+0.38%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc6c807...301499d. Read the comment docs.

lgtm-com · 2021-03-11T12:22:16Z

This pull request introduces 1 alert when merging 303bf7b into dc6c807 - view on LGTM.com

new alerts:

1 for Variable defined multiple times

pryorda · 2021-03-11T18:20:11Z

Memory:

(vim.host.HardwareStatusInfo.HardwareElementInfo) [
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 12',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   },
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 13',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   },
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 14',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   },
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 11',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   },
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 0',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   }
]

pryorda · 2021-03-11T18:23:34Z

cpu:

cpuStatusInfo = (vim.host.HardwareStatusInfo.HardwareElementInfo) []

M0NsTeRRR · 2021-03-11T21:06:13Z

Memory:

(vim.host.HardwareStatusInfo.HardwareElementInfo) [
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 12',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   },
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 13',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   },
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 14',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   },
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 11',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   },
   (vim.host.HardwareStatusInfo.HardwareElementInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory 0',
      status = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'green',
         summary = 'green',
         key = 'green'
      }
   }
]

Oh strange, I got memory info on runtime.healthSystemRuntime.systemHealthInfo.numericSensorInfo not on this endpoint do you get the info here too ?

vmware_host_sensor_state{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Memory Device 1 A  6",type="memory"} 2.0
vmware_host_sensor_state{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Memory Device 1 A  5",type="memory"} 2.0
vmware_host_sensor_state{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Memory Device 1 A  4",type="memory"} 2.0
vmware_host_sensor_state{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Memory Device 1 A  3",type="memory"} 2.0
vmware_host_sensor_state{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Memory Device 1 A  2",type="memory"} 2.0
vmware_host_sensor_state{cluster_name="HA",dc_name="Homelab",host_name="server3.unicornafk.fr",name="Memory Device 1 A  1",type="memory"} 2.0

Also I got more information on memory on my endpoint

(vim.host.NumericSensorInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'Memory Device 1 A  6',
      healthState = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'Green',
         summary = 'Sensor is operating under normal conditions',
         key = 'green'
      },
      currentReading = 64,
      unitModifier = 0,
      baseUnits = 'sensor-discrete',
      rateUnits = 'none',
      sensorType = 'memory',
      id = '0.32.1.197',
      timeStamp = '2021-03-11T01:52:00Z'
   }

runtime.healthSystemRuntime.hardwareStatusInfo.cpuStatusInfo runtime.healthSystemRuntime.hardwareStatusInfo.memoryStatusInfo

M0NsTeRRR · 2021-03-12T00:46:51Z

Ok i've added the 2 others metrics if you can test it

pryorda · 2021-03-12T17:02:51Z

ill test shortly and let you know.

pryorda · 2021-03-29T16:19:28Z

@M0NsTeRRR I did some testing and after merging I get this error:

2021-03-29 16:15:01,213 ERROR:Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/vmware_exporter/vmware_exporter.py", line 1922, in _async_render_GET
    yield self.generate_latest_metrics(request)
twisted.internet.defer.FirstError: FirstError[#3, [Failure instance: Traceback: <class 'KeyError'>: 'Green'
/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py:477:callback
/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py:580:_startRunCallbacks
/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py:662:_runCallbacks
/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py:1514:gotResult
--- <exception caught here> ---
/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py:1445:_inlineCallbacks
/usr/local/lib/python3.7/site-packages/vmware_exporter/vmware_exporter.py:1705:_vmware_get_hosts
]]

2021-03-29 16:15:01,215 INFO:FIN: _vmware_get_host_perf_manager_metrics
2021-03-29 16:15:47,475 INFO:Start collecting metrics from ipx
2021-03-29 16:15:47,475 INFO:Starting vm metrics collection
2021-03-29 16:15:47,475 INFO:Fetching vim.VirtualMachine inventory
2021-03-29 16:15:47,475 INFO:Retrieving service instance content
2021-03-29 16:15:47,476 INFO:START: _vmware_get_vm_perf_manager_metrics
2021-03-29 16:15:47,477 INFO:Fetching vim.Datastore inventory
2021-03-29 16:15:47,477 INFO:Starting host metrics collection
2021-03-29 16:15:47,477 INFO:Fetching vim.HostSystem inventory
2021-03-29 16:15:47,477 INFO:START: _vmware_get_host_perf_manager_metrics
2021-03-29 16:15:49,503 INFO:Retrieved service instance content
Unhandled error in Deferred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 477, in callback
    self._startRunCallbacks(result)
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 580, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 662, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1514, in gotResult
    current_context.run(_inlineCallbacks, r, g, status)
--- <exception caught here> ---
  File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1445, in _inlineCallbacks
    result = current_context.run(g.send, result)
  File "/usr/local/lib/python3.7/site-packages/vmware_exporter/vmware_exporter.py", line 1705, in _vmware_get_hosts
    }[sensor['sensorStatus']]
builtins.KeyError: 'Green'

vmware_exporter/helpers.py

M0NsTeRRR added 2 commits March 10, 2021 23:12

[Update] Refactor alarm fetching system

1811b4c

[Update] Add sensors

0f13eb6

vmware_host_sensor_state vmware_host_sensor_fan vmware_host_sensor_temperature vmware_host_sensor_power_voltage vmware_host_sensor_power_current vmware_host_sensor_power_watt vmware_host_sensor_redundancy

M0NsTeRRR mentioned this pull request Mar 11, 2021

ESX host temperature monitoring #226

Open

M0NsTeRRR added 2 commits March 11, 2021 13:02

[Fix] Pipeline flake8

42db897

[Fix] Pipeline flake8

303bf7b

M0NsTeRRR and others added 2 commits March 11, 2021 13:36

[Fix] Pipeline LGTM

4523ab9

Merge branch 'main' into main

301499d

pryorda mentioned this pull request Mar 11, 2021

Hardware sensors Information. #241

Closed

[Update] handle 2 others metrics

929fec5

runtime.healthSystemRuntime.hardwareStatusInfo.cpuStatusInfo runtime.healthSystemRuntime.hardwareStatusInfo.memoryStatusInfo

pryorda merged commit da2f489 into pryorda:main Mar 29, 2021

M0NsTeRRR commented Mar 29, 2021

View reviewed changes

vmware_exporter/helpers.py Show resolved Hide resolved

pryorda mentioned this pull request Mar 30, 2021

fix(fix_sensor): Fixing sensor status look up #262

Merged

xkilian mentioned this pull request Sep 13, 2021

Support vmware hardware sensor metrics cblomart/vsphere-graphite#158

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Sensors #255

[Enhancement] Sensors #255

M0NsTeRRR commented Mar 11, 2021 •

edited

Loading

codecov-io commented Mar 11, 2021 •

edited

Loading

lgtm-com bot commented Mar 11, 2021

pryorda commented Mar 11, 2021

pryorda commented Mar 11, 2021

M0NsTeRRR commented Mar 11, 2021

M0NsTeRRR commented Mar 12, 2021

pryorda commented Mar 12, 2021

pryorda commented Mar 29, 2021

[Enhancement] Sensors #255

[Enhancement] Sensors #255

Conversation

M0NsTeRRR commented Mar 11, 2021 • edited Loading

Metrics added

Breaking changes

Enhancement

Example

Tested on Dell R320 & Intel R1208WTTGSR

codecov-io commented Mar 11, 2021 • edited Loading

Codecov Report

lgtm-com bot commented Mar 11, 2021

pryorda commented Mar 11, 2021

pryorda commented Mar 11, 2021

M0NsTeRRR commented Mar 11, 2021

M0NsTeRRR commented Mar 12, 2021

pryorda commented Mar 12, 2021

pryorda commented Mar 29, 2021

M0NsTeRRR commented Mar 11, 2021 •

edited

Loading

codecov-io commented Mar 11, 2021 •

edited

Loading