INCIDENT-23437 Always resolve hostname regardless of how dbm or disable_generic_tags are set

[alexandre-normand]

What does this PR do?

This change makes sure that the check's resolved_hostname always represents the database instance. The previous behavior of the resolved_hostname property was to use the agent hostname as the host when dbm was disabled or disable_generic_tags was enabled. This caused issues when, with agent 7.48.0, we started sending database_instance metadata. With the agent_hostname being used instance of each instance's host value, agents with multiple databases being monitored would end up creating and clobbering the same database_instance which, in turn, caused tag interference between database instances monitored by the same agent.

Very importantly, that problem surfaced the broken experience when dbm is disabled in regards to the host used on integration metrics: instead of having the postgres metrics tagged by the database instance's host, they were tagged by the agent hostname which would only be correct when the agent is running on the same host as the database.

Here is an example showing this deployment of a GCP cloud sql database instance with dbm disabled (stolen from the sister PR to the postgres integration but this applies to mysql as well):

  - dbms: postgres
    dbms_version: cloudsql-postgres-15-sandbox
    cloud: true
    instances:
      - host: '10.x.x.x'
        dbm: false
        port: 5432
        ssl: "allow"   
        autodiscovery_exclude:
          - cloudsqladmin

This, running in kubernetes, was deployed on a pod / host named postgres-f06e6baf-7cb8dcd888-tmds4.
Looking at the metrics emitted by the integration, we can see that the host tag is postgres-f06e6baf-7cb8dcd888-tmds4, the agent host, rather than 10.x.x.x`.

Note that this is going to be a breaking change but given the currently broken experience, this is something we want to do in order to improve the experience and to avoid potential future incidents resulting in the confusing implementation of resolved_hostname.

Motivation

Additional Notes

• We considered an alternative of creating a new property (database_instance_name) that would be used for everything except the core integration metrics. It would essentially be the same implementation as the resolved_hostname property from this PR but would leave resolved_hostname as-is. We decided to go with the better approach of fixing the broken experience at the risk of causing friction for users relying on that broken behavior.

• It's also worth noting that the experience for aws rds instances was less broken than others because of some custom inference of additional host, hostname and aws tags here. The metric points were still also tagged with the monitoring agent's hostname which was not correct but easier to deal with when filtering on aws rds instance was possible.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Changelog entries must be created for modifications to shipped code
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.

[github-actions]

Test Results

     16 files       16 suites   15m 0s ⏱️
   138 tests    138 ✔️ 0 💤 0 ❌
1 104 runs  1 095 ✔️ 9 💤 0 ❌

Results for commit 7862764.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

Merging #16201 (7862764) into master (305f739) will increase coverage by 0.10%.
Report is 11 commits behind head on master.
The diff coverage is 100.00%.

Additional details and impacted files

Flag	Coverage Δ
activemq	?
cassandra	?
confluent_platform	?
hive	?
hivemq	?
hudi	?
ignite	?
jboss_wildfly	?
kafka	?
mysql	87.39% <100.00%> (-0.04%)	⬇️
presto	?
solr	?
tomcat	?
weblogic	?

Flags with carried forward coverage won't be shown. Click here to find out more.