Set PV/PVC's namespace when using Observability Module for PowerStore

[jaeliang77]

How to set PV/PVC's namespace when using Observability Module for PowerStore

**Hello Support:
We encounter a namespace problem when trying to fetch powerstore PV/PVC Metrics in Prometheus :
When we try to fetch "powerstore_volume_logical_provisioned_megabytes/powerstore_volume_logical_used_megabytes/powerstore_filesystem_logical_provisioned_megabytes/powerstore_filesystem_logical_used_megabytes" metrics in Prometheus, we found that it can only fetch the values from CSM's namespace, but can not fetch values from real PV/PVC's namespace.
Hope to know how to set the correct PV/PVC namespace when fetching the metrics ?
Thanks!!

**

[jaeliang77]

Dear Support:
The customer hope PowerStore CSM can provide PV/PVC namespace in metrics as K8S volume .
For example:

1. PowerStore CSM powerstore_volume_logical_used_megabytes now only provide label "namespace" which provide CSM's namespace, but not PV's namespace.
2. K8S kubelet_volume_stats_used_bytes 's label includes both "namespace" and "exported_namespace" , the ”exported_namespace" provide PV's namespace.
3. Hope PowerStore CSM can provide the function as "exported_namespace" in K8S metrics.

[jaeliang77]

The question may be turned to: Shall PowerStore CSM PV metrics not only expose physical storage performance and capacity info, but also expose PV in K8S's information , such as PVC name , namespace, etc...
So that the customer can use Prometheus monitor to connect PV with PVC and the K8S pod which using it.
Thanks!!

[hoppea2]

Hi @jaeliang77 if I understand your question correctly you're looking for a mapping of the PowerStore volumes to the PV/PVC information in K8S. This is not exposed in Prometheus but instead via the Topology service dashboard in Grafana. See "Topology Service" information on that page.

This service provides a mapping between array volumes and k8s persistent volumes, including the namespace:

Please let us know if you have further questions!

[jaeliang77]

Thanks for your kindly reply! This is helpful!
Besides dashboard display , the customer also hope to develop monitor alert rules for PV usage. When PV usage or IOPS reach limits, alerts will be sent to appropriate application developers ( they are distinguished by namespace) automatically . So they still hope to fetch PV's PVC namespace in prometheus in order to trigger the alert .

For Example:
(1) Now they can fetch PVC namespace from K8S's PV metrics "kubelet_volume_stats_used_bytes" (exported_namespace) :
Expression: (sum(kubelet_volume_stats_used_bytes) by (persistentvolumeclaim,exported_namespace) / sum(kubelet_volume_stats_capacity_bytes) by (persistentvolumeclaim,exported_namespace))*100 >80

alertname="test-pv-usage"
alert_name="test-pvc-usage"
alert_type="metric"
cluster_name="test-cluster1 (ID: c-qlmv9)"
comparison="greater than"
duration="30s"
exported_namespace="appset-home"
expression="(sum(kubelet_volume_stats_used_bytes) by (persistentvolumeclaim,exported_namespace) / sum(kubelet_volume_stats_capacity_bytes) by (persistentvolumeclaim,exported_namespace))*100>80"
group_id="c-qlmv9:cag-lf9j9"
persistentvolumeclaim="vol1"
prometheus="cattle-prometheus/cluster-monitoring"
prometheus_from="test-cluster1"
server_url="https://k8stest.axzq.com.cn:8080 [k8stest.axzq.com.cn]"
severity="critical"
threshold_value="5"

(2) But it seems that there is no PVC namespace in powerstore metric：
Expression: (sum(powerstore_volume_logical_used_megabytes) by (PersistentVolumeName) / sum(powerstore_volume_logical_provisioned_megabytes) by (PersistentVolumeName))*100 or (sum(powerstore_filesystem_logical_used_megabytes) by (PersistentVolumeName) / sum(powerstore_filesystem_logical_provisioned_megabytes) by (PersistentVolumeName))*100>80

alertname="pv usage > 80"
alert_name="pv usage > 80"
alert_type="metric"
PersistentVolumeName="csi-lab-chengh-7a46c59316"
cluster_name="chengh-cluster1 (ID: c-ncbgw)"
comparison="greater than"
duration="30s"
expression="(sum(powerstore_volume_logical_used_megabytes) by (PersistentVolumeName) / sum(powerstore_volume_logical_provisioned_megabytes) by (PersistentVolumeName))*100 or (sum(powerstore_filesystem_logical_used_megabytes) by (PersistentVolumeName) / sum(powerstore_filesystem_logical_provisioned_megabytes) by (PersistentVolumeName))*100>80"
group_id="c-ncbgw:etcd-alert"
prometheus="cattle-prometheus/cluster-monitoring"
prometheus_from="chengh-cluster1"
server_url="https://lab.axzq.com.cn [lab.axzq.com.cn]"
severity="critical"
threshold_value="80"

The customer hope to fetch PVC namespace in PowerStore performance/capacity metrics, as what they fetch from K8S PV metrics . So they can develop monitor alert rules in Prometheus.

Many Thanks!!

---

[hoppea2]

@jaeliang77 thank you for the feedback.

We reviewed what is required to expose the namespace in the PowerStore metrics via Prometheus and put it on our roadmap for CSM Observability.

[jaeliang77]

Many Thanks!!
We are waiting for the CSM Observability update!!

[jaeliang77]

Thanks for the csm-observability for powerstore 1.2 feature update!
The customer is trying to upgrade their CSM but unfortunately they can only use offline upgrade.
We try to follow the offline upgrade guide in "https://dell.github.io/csm-docs/docs/observability/deployment/offline/ " but it seems that :
(1) There is no offline upgrade guide , but only reinstall guide for offline customers.
(2) There is still csm-metrics-powerstore:v1.0.0 in the offline installation package if we follow the offline install guide , but not new v1.2.0.

Hope to have your help to upgrade the offline CSM to update version, Many Thanks!!

[YianZong]

Hi @jaeliang77

(1) We are considering add offline upgrade for observability.
(2) Please create new offline bundle pointing to csm-metrics-powerstore:v1.2.0 by following the page here.

BTW, the PV/PVC namespace mapping is not implement in csm-metrics-powerstore:v1.2.0. It will be release in 1.3.0.

[jaeliang77]

Thanks!! Then we will wait for the new 1.3.0 release and offline upgrade update.

[jaeliang77]

Dear Sir: Hope to know when the csm-metrics-powerstore:v1.3.0 will be released? The customer hope to use the pv/pvc namespace mapping function for powerstore observability . Thanks!!

…

________________________________ 发件人: YianZong ***@***.***> 发送时间: 2022年8月1日 16:34 收件人: dell/csm ***@***.***> 抄送: jaeliang77 ***@***.***>; Mention ***@***.***> 主题: Re: [dell/csm] Set PV/PVC's namespace when using Observability Module for PowerStore (Issue #204) Hi @jaeliang77<https://github.com/jaeliang77> (1) We are considering add offline upgrade for observability. (2) Please create new offline bundle pointing to csm-metrics-powerstore:v1.2.0 by following the page here. ― Reply to this email directly, view it on GitHub<#204 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AI6VTXMNUW4N5SJMOCCHURDVW6DX3ANCNFSM5PGPJ6MQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[YianZong]

Hi @jaeliang77
The RTS is target at Sep. 28.

[jaeliang77]

Got it, Thanks !!

…

________________________________ 发件人: YianZong ***@***.***> 发送时间: 2022年8月29日 14:13 收件人: dell/csm ***@***.***> 抄送: jaeliang77 ***@***.***>; Mention ***@***.***> 主题: Re: [dell/csm] Set PV/PVC's namespace when using Observability Module for PowerStore (Issue #204) Hi @jaeliang77<https://github.com/jaeliang77> The RTS is target at Sep. 28. ― Reply to this email directly, view it on GitHub<#204 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AI6VTXLUSBNENIAYOSPBOFLV3RII7ANCNFSM5PGPJ6MQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[gallacher]

Fixed as part of #453