redact secrets in diagnostics collect command

[michel-laterman]

Describe the enhancement:

The elastic-agent diagnostics collect command creates an archive that config files that have secret/sensitive data.
These fields should be redacted before being written to the archive.

api_key:  "<REDACTED>"
key_passphrase: "<REDACTED>"
access_api_key: "<REDACTED>"

Additionally the private key of a cert should be redacted if it's inlined in the config and not a path to a file.

[jlind23]

First step here, hide fields that follow some common patterns:

• key
• password
• passphrase
• token

@joshdover @ph could you please think about the long term solution and how we can mark fields as sensitive right from the integration definition?

[jlind23]

@joshdover @ph as the package spec PR is now done, do we really need to do this intermediary step #241 (comment) Shouldn't we directly rely on the secret type?

[joshdover]

We could still do the intermediate step as a stop-gap, but probably better to spend effort on the long-term solution.

[ph]

I believe we should do this, the long-term solution would only be delivered on 8.x and won't be backported to 7.x. Since we will support 7.x for quite some time and use the diagnostic subcommand time we should have that stop gap in and backported to 7.x.

[AndersonQ]

First step here, hide fields that follow some common patterns:

• key
• password
• passphrase
• token

Another pattern would be certificate or other terms we use for the ssl/certificates configuration

[ghost]

Hi @AndersonQ,

We have verified below check-points for this feature on the latest 8.4.0-SNAPSHOT build.

Build details:

Version: 8.4.0 SNAPSHOT
Build: 54585
Commit: f9e2ed4d9f38424676a558c34b74f0a031746c9e

Observations

• REDACTED value updated for below keys in the following files after using elastic-agent diagnostics collect command

Under Config folder:
1.  access_api_key is <REDACTED> in the elastic-agent-local.yaml file 
2.  api_key is <REDACTED> in the **elastic-agent-policy.yaml file , filebeat_default.yaml file , fleet_monitoring_default.yaml file and metricbeat_default.yaml file **
3. `access_api_key`, `service_token`, `certificate`, `key` are <REDACTED> in the elastic-agent-local.yaml file 

Further, for the ssl/cert configuration, we have following Queries:

Query 1: Could you please confirm if we need to validate any other check-points for the ssl/cert configuration other than elastic-agent-local.yaml file.

Query 2: Could you please confirm ca-trusted_fingerprint value is not displayed as REDACTED right now in following config files. Is it expected.

• elastic-agent-local.yaml elastic-agent-policy.yaml filebeat_default.yaml fleet_monitoring_default.yaml & metricbeat_default.yaml .

Please let us know if we are missing anything while validating this feature.

Thanks!

CC: @michel-laterman