[receiver/vcenter] Fix multi cluster deployment resource attributes

[schmikei]

Description: First pass at better contextualizing VMs in a multicluster deployment.

The Problem

For multicluster deployments the original scraper code incorrectly interpreted assumptions on VM ownership belonging to multiple clusters.

The Solution

The VM folder in a datacenter is completely separated from a cluster. Because of this we have to determine the VM's host system or resource polo in order to appropriate set the vcenter.cluster.name appropriately.

Things that have changed

• ClusterComputes are now retrieved as a superset Computes
• For the resource attribute vcenter.cluster.name to show up, the inventory type must match the string "ClusterComputeResource"
• The receiver now maintains a dynamic map of vm Managed Object Reference Value to compute name

Customer impacts

• VM metrics will more appropriately set the vcenter.cluster.name resource attribute in multi-cluster environments. This resource attribute will not be emitted for VMs on a runtime hosts outside of a cluster.
• VM metrics will now emit with vcenter.resource_pool.name and vcenter.resource_pool.inventory_path when the VM is a member of one.

Link to tracking Issue: Resolves #30879

Testing:

Documentation:

[schmikei]

I believe this is ready to be looked at, I think I would like to write up a better doc on what exactly has changed for users before we merge though.

[schmikei]

@djaglowski I think this is an approach that can work. Happy to iterate on a featuregate if we think its appropriate for this.

[djaglowski]

This would introduce some more resource attributes to VM metrics conditionally on if they have a resource pool attached with them and or if they are running on a Compute managed reference vs a non-clustered one.

Can you detail this out for me (and eventually for users)? The code seems reasonable at a glance but I'm having a hard time understanding the net change.

[schmikei]

Yeah sure thing. Gonna write the changes here then I'll update the PR description.

The Problem

For multicluster deployments the original scraper code incorrectly interpreted assumptions on VM ownership belonging to multiple clusters.

The Solution

The VM folder in a datacenter is completely separated from a cluster. Because of this we have to determine the VM's host system or resource polo in order to appropriate set the vcenter.cluster.name appropriately.

Things that have changed

• ClusterComputes are now retrieved as a superset Computes
• For the resource attribute vcenter.cluster.name to show up, the inventory type must match the string "ClusterComputeResource"
• The receiver now maintains a dynamic map of vm Managed Object Reference Value to compute name

Customer impacts

• VM metrics will more appropriately set the vcenter.cluster.name resource attribute in multi-cluster environments. This resource attribute will not be emitted for VMs on a runtime hosts outside of a cluster.
• VM metrics will now emit with vcenter.resource_pool.name when the VM is a member of one.

If there is a better place to put this information in the changelog I can put it inside the subtext. Otherwise happy to make a feature gate or put it on the issue so it's easily traceable.

[djaglowski]

@adasescu, would you be willing to take as well?

[adasescu]

Looking at the changes, the relationship between Cluster->Host->Vm is resolved with the Compute superset.
Related to the new VM metrics which will emit vcenter.resource_pool.name, I think vcenter.resource_pool.inventory_path would be a good addition (in our case is more relevant than the name). And also an integration test that will contain VM metrics along with resource attributes including vcenter.cluster.name and vcenter.resource_pool.name might be helpful.

[schmikei]

Looking at the changes, the relationship between Cluster->Host->Vm is resolved with the Compute superset. Related to the new VM metrics which will emit vcenter.resource_pool.name, I think vcenter.resource_pool.inventory_path would be a good addition (in our case is more relevant than the name). And also an integration test that will contain VM metrics along with resource attributes including vcenter.cluster.name and vcenter.resource_pool.name might be helpful.

Thanks for the valuable feedback 🚀 I'll try and get those implemented as soon as I can sometime next week into this PR!

[schmikei]

I added the vcenter.resource_pool.inventory_path to VM metrics, I wasn't able to get the simulator under the hood to actually place them into the pre-made resource pools.

Was getting this error:

        	Error:      	Received unexpected error:
        	            	ServerFaultCode: VirtualMachine:vm-55 does not implement: MigrateVM_Task

Have to respin up my vCenter in order to remake the mocks for scraper test to properly capture. Apologies for the delay!

@adasescu I've been informed that because I can't enable DRS in our reference environment, that we cannot validate this quickly. Is there any chance you could take a pass at validating with your environment? Otherwise would you be able to mock out following these instructions https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/vcenterreceiver/internal/mockserver and I can try to encapsulate in a test case for us with that information :)

[adasescu]

I was able to take a pass on our environment. Didn't get any error but resource_pool.name and resource_pool.inventory_path values aren't issued along with other VM metrics. Even with a resource_pool metric enabled like below the VM metrics don't show resource_pool.name and resource_pool.inventory_path.

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.resource_pool.name:
          enabled: true
        vcenter.resource_pool.inventory_path:
          enabled: true
      ...
      metrics:
        vcenter.resource_pool.cpu.usage:
          enabled: true

I couldn't test the multiple cluster functionality cause I lost access on the second cluster, but for one cluster the vcenter.cluster.name value is right.

[schmikei]

@adasescu I ended up storing in a map the data retrieved from resource pools with inventory path now set. Hopefully things should be good now if you're able to try again. Sorry for the miss on the first pass!

[adasescu]

No worries.
Tested the changes. When resource_pool resource attributes are disabled, the values don't show up in VM metrics.

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.resource_pool.name:
          enabled: false
        vcenter.resource_pool.inventory_path:
          enabled: false 

When are enabled they show up with right values, but there's also an empty ResourceBuilder, like when a metric let's say vcenter.resource_pool.cpu.usage would be enabled. This behaviour isn't consistent with the Datastore resource builder. Don't know which is the desired behaviour. For Datastore with the config below no metrics are generated.

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.datastore.name:
          enabled: true

I hope I could make myself understood.
We are only using the VM metrics.

[schmikei]

No worries. Tested the changes. When resource_pool resource attributes are disabled, the values don't show up in VM metrics.

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.resource_pool.name:
          enabled: false
        vcenter.resource_pool.inventory_path:
          enabled: false 

When are enabled they show up with right values, but there's also an empty ResourceBuilder, like when a metric let's say vcenter.resource_pool.cpu.usage would be enabled. This behaviour isn't consistent with the Datastore resource builder. Don't know which is the desired behaviour. For Datastore with the config below no metrics are generated.

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.datastore.name:
          enabled: true

I hope I could make myself understood. We are only using the VM metrics.

@adasescu sorry just for my clarification, could you share a redacted with sensitive information version of your config/output that appears to be problematic? Are we just looking at the VM metrics being collected?

[adasescu]

No worries. Tested the changes. When resource_pool resource attributes are disabled, the values don't show up in VM metrics.

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.resource_pool.name:
          enabled: false
        vcenter.resource_pool.inventory_path:
          enabled: false 

When are enabled they show up with right values, but there's also an empty ResourceBuilder, like when a metric let's say vcenter.resource_pool.cpu.usage would be enabled. This behaviour isn't consistent with the Datastore resource builder. Don't know which is the desired behaviour. For Datastore with the config below no metrics are generated.

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.datastore.name:
          enabled: true

I hope I could make myself understood. We are only using the VM metrics.

@adasescu sorry just for my clarification, could you share a redacted with sensitive information version of your config/output that appears to be problematic? Are we just looking at the VM metrics being collected?

I'll give it a try at the beginning of the next week

[adasescu]

@schmikei I was wrong in my previous comment, I must have been looking at the wrong data. Apologies for that and for the delay.
I ran the following test cases and from my opinion all looks good. The generated metrics are exported into a kafkaexporter in our deployment. The configuration for resource attributes vcenter.cluster.name, vcenter.host.name, vcenter.vm.* are all enabled and metrics for VM are always enabled. Here are the config/output tested:

1. all enabled

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.resource_pool.name:
          enabled: true
        vcenter.resource_pool.inventory_path:
          enabled: true
        vcenter.datastore.name:
          enabled: true
      ....
      metrics:
        vcenter.datastore.disk.usage:
          enabled: true
        vcenter.resource_pool.memory.usage:
          enabled: true

output file 1-test-2024-03-06-all-enabled.txt

2. resource attributes enabled, metrics disabled. In this test case things wasn't clear.

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.resource_pool.name:
          enabled: true
        vcenter.resource_pool.inventory_path:
          enabled: true
        vcenter.datastore.name:
          enabled: true
      ....
      metrics:
        vcenter.datastore.disk.usage:
          enabled: false
        vcenter.resource_pool.memory.usage:
          enabled: false

output file 2-test-2024-03-06-res-att-on-metrics-off.txt

3. all disabled

config:
  receivers:
    vcenter:
      resource_attributes:
        vcenter.resource_pool.name:
          enabled: false
        vcenter.resource_pool.inventory_path:
          enabled: false
        vcenter.datastore.name:
          enabled: false
      ....
      metrics:
        vcenter.datastore.disk.usage:
          enabled: false
        vcenter.resource_pool.memory.usage:
          enabled: false

output file 3-test-2024-03-06-all-disabled.txt

[schmikei]

@adasescu thank you so much for your time in validation! 🚀

@djaglowski are you able to look over this again?