libbeat/monitoring/inputmon: log key, id and input type when registering/deregistering metrics

[efd6]

What does this PR do?

This notes under "metric_registry" all inputmon-handled metric registration and deregistration, linking register and deregister operations by use of a unique ID unrelated to the request.

Why is it important?

We have had a number of cases where metric registrations have been duplicated, this addition allows us to identify pairs of registration/deregistration and correlate with prior un-deregistered registrations with the same name. The logging at INFO level speeds recovery of the information with minimal additional log load.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Marked as backporting for benefits to support.

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-01T23:06:43.476+0000

• Duration: 69 min 5 sec

Test stats 🧪

Test	Results
Failed	0
Passed	26421
Skipped	1975
Total	28396

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

I assume this is related to the panics that this causes. One suggestion is to modify NewInputRegistry to return an error if the ID is already registered. The upstream elastic-agent-libs/monitoring package would need extended to offer a method that does not panic.

What's an example cause of a duplicated registration that this logging would help us trace?

[efd6]

I'm not convinced that relaxing the constraint of registration is a good idea.

[andrewkroh]

My main driver for making the change upstream is to lower the consequence of these programming mistakes on production systems. I like that the panic alerts us to these issues, but in production I would choose to have the input continue to operate and log and error about metric collection issues. Something along the lines of https://pkg.go.dev/go.uber.org/zap#Logger.DPanic behavior.

But the logging is useful in any case for debugging these problems.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)