Refactor keystore interface into 3 different interfaces

[ChrsMark]

What does this PR do?

This PR refactor keystore interface into 3 different interfaces.

This PR introduces a new Keystore backend which is ReadOnly and retrieves passwords from k8s secrets through k8s API. In order to implement this, the basic Keystore interface is split into 3 different interfaces, Keystore (readonly), WritableKeystore (Keystore + store/write/delete), ListingKeystore (WritableKeystore + list) as proposed on #5832.

The new Keystore that retrieves data from k8s secrets is of Keystore (readonly) type.

#### Autodiscover adoption
In order to support this k8s Keystore in Autodiscover, a keystore object is attached on every event. The keystore object is either created or retrieved if it already exists. We create only one keystore per namespace and we reuse it for events of the same namespace.

Why is it important?

This is important in order to support different keystore backends, like k8s keystore.

This is important in order to support autodiscover hints being able to refer to k8s secrets for password retrieval.

How to test this PR locally

Try all the set of available commands:

1. ./metricbeat keystore create
2. ./metricbeat keystore add ES_PWD
3. ./metricbeat keystore add ES_PWD --force
4. echo "passpass" | ./metricbeat keystore add ES_PWD --stdin --for
5. ./metricbeat keystore list
6. ./metricbeat keystore remove ES_PWD

Create a key and consume it (Elastic Cloud auth or any other ES setting can be used as an example):

1. ./metricbeat keystore add ES_PWD
2. ./metricbeat -e -d "*" -E "cloud.id=testkeys:xxxx" -E "cloud.auth=\${ES_PWD}"

or

1. ./metricbeat keystore add ES_HOST
2. ./metricbeat -e -d "*" -E output.elasticsearch.hosts=["\${ES_HOST}"]

Related issues

• Relates Allow the Keystore to be completely pluggable #5832, [Metricbeat] Support pulling secrets from Kubernetes/OpenShift for autodiscovery #8847

[exekias]

I would recommend splitting this PR in 2:

• refactor keystore interface into 3 different interfaces, with all the changes required to code using keystore.
• add the new k8s keystore

[ChrsMark]

Updated that, to keep only the Keystore refactoring. The implementation of k8s keystore could be retrieved by the previous commit (bd508d6).

I will open it for review to start shaping it more.
One thing that I'm not sure of is how we could better approach the Factory implementation. I thought of it like different factory methods for each different keystore backend. This is how I did it with k8s keystore at bd508d6. But proposals are more than welcome.

Also interface inheritance is something to consider, since for now I followed an approach of having "stronger" interfaces inheriting the ones with the lower privileges.

[exekias]

One thing that I'm not sure of is how we could better approach the Factory implementation. I thought of it like different factory methods for each different keystore backend. This is how I did it with k8s keystore at bd508d6. But proposals are more than welcome.

I would avoid a common Factory for now, as you don't need it yet? Left a comment about the existing one, but it looks to me that it's file_keystore specifc.

[exekias]

This is looking good, but tests seem to be failing

[ChrsMark]

Yeap @exekias , tests were failing because the refactoring was not "healthy". I had to ensure that in all places the stores are properly checked if implement the respective interfaces. Feel free to comment on it.

[ChrsMark]

@urso @exekias any extra comments on this would be much appreciated so as to move this forward 🙂

[urso]

LGTM. Let's see what CI says.

[ChrsMark]

jenkins, test this again please

[ChrsMark]

CI looks happy now. I also added manual testing notes that I used in my testing process too.