[Meta] Improving Hints Autodiscovery with Elastic Agent

[gizas]

Synopsis

Hints based Autodiscovery for Elastic Agent has been released as beta since 8.5. Information about the functionality and configuration of this feature can be found here.
Note that templates are being created through PRs from specific CI jobs.

This is the Meta issue that tries to summarise a list of enhancements, bug fixes and documentation updates we have gathered so far in order to improve the general UX and functionality of Hints based Autodiscovery.

List of Issues

• https://github.com/elastic/package-storage-infra/issues/479 @ChrsMark
• [Elastic Agent ] Support of processors functionality with hints autodiscovery #2959 @gizas
• [Hints] K8s autodiscovery does not work when data_streams are not specified #3057 @ChrsMark
• Documentation updates for Autodiscovery for Elastic Agent #3062 @gizas
• Introduce log message for not supported annotations for Hints based autodiscover of Elastic Agent #3064 @gizas
• [Elastic Agent] Add support for parsers configuration for Hints Autodiscover #3102
• Document default_container_logs setting of k8s provider #3158 @gizas
• Support hints Autodiscovery for input configs per container #3161
• Remove the need to define host in hints autodiscovery by using proper default values #1453 @constanca-m
• Support of secrets for autodiscovery scenarios in kubernetes environments  #2200

[ChrsMark]

Added #3158 after discussed at #2959 with @gsantoro.

[cmacknz]

We are starting to look at #2460 where the root problem is that the agent configurations generated by autodiscovery are gigantic when it is configured to monitor all the pods on a node or all the nodes in a cluster.

We haven't decided on a solution yet, so the exact impact isn't known. I don't expect it to affect any of the work here, but wanted to note it here for visibility anyway since it may affect how agent autodiscovery generates configurations.

[ChrsMark]

@gizas sth also related to this feature: #2200