Update observability README + fix typos

[eero-t]

Description

This is continuation of #541:

• Updates observability README + fix typos
• Gives its image files more descriptive names

Issues

n/a.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Dependencies

n/a.

Tests

CI + PR file view functionality.

Further work

Additional improvements that could be done e.g. after v1.1 release:

• Remove serviceMonitor files (obsolete by this and earlier Helm support PR) from chatqna/
• Generalize dashboards under chatqna/dashboards/ and move them to more appropriate place
• Optimize images file sizes

[yongfengdu]

The 1.1 release are using tgi-gaudi 2.0.6, which is validated with SW stack v 1.18. Shall we update/verify the Metrics with 1.18?

[eero-t]

IMHO not worth the trouble for that dashboard. I do not see why Habana would change metric names between version upgrades, and there's a much better Gaudi HW panel in Eval repo [1], which is more worth checking.

In the long run, I think it would be better to separate dashboard for different purposes to different repos, instead of duplicating them:

• Drop Gaudi HW dashboard from here [1],
• Move PCM one to Eval repo, and
• Add k8s specific ones here, that do some assumptions about how deployments are named with current Helm charts, to allow user to select from multiple apps running in different namespaces

=> I can do that after v1.1.

---

[1] Gaudi HW one here is not very good. It's lacking most metrics, does not allow selecting a node or device for them, and as can be seen from its screenshot in the README, the metric legends are awful:

$ cat Dashboard-Gaudi-HW.json  | grep expr | cut -d'"' -f4- | sed 's/",$//'
habanalabs_temperature_onboard
habanalabs_kube_info
habanalabs_memory_free_bytes
habanalabs_power_mW
habanalabs_utilization

vs one in Eval repo:

$ cat gaudi_grafana.json | grep expr | cut -d'"' -f4- | sed 's/",$//'
habanalabs_device_config{instance=\"$node\", UUID=\"$hpu\"}
habanalabs_utilization{instance=\"$node\", UUID=\"$hpu\"}/100
habanalabs_power_mW{instance=\"$node\", UUID=\"$hpu\"}
habanalabs_pcie_receive_throughput{UUID=\"$hpu\", instance=\"$node\"}
habanalabs_pcie_transmit_throughput{UUID=\"$hpu\", instance=\"$node\"}
habanalabs_temperature_onchip{UUID=\"$hpu\", instance=\"$node\"}
habanalabs_temperature_onboard{UUID=\"$hpu\", instance=\"$node\"}
habanalabs_device_config{instance=\"$node\", UUID=\"$hpu\"}
habanalabs_device_config{instance=\"$node\", UUID=\"$hpu\"}
habanalabs_utilization{instance=\"$node\", UUID=\"$hpu\"}/100
habanalabs_temperature_onboard{UUID=\"$hpu\", instance=\"$node\"}
habanalabs_temperature_onchip{UUID=\"$hpu\", instance=\"$node\"}
habanalabs_clock_soc_mhz{UUID=\"$hpu\", instance=\"$node\"}
habanalabs_memory_used_bytes{UUID=\"$hpu\", instance=\"$node\"}
habanalabs_power_mW{instance=\"$node\", UUID=\"$hpu\"}
habanalabs_memory_used_bytes{UUID=\"$hpu\", instance=\"$node\"} / habanalabs_memory_total_bytes{UUID=\"$hpu\", instance=\"$node\"}