Support hints Autodiscovery for input configs per container

[gsantoro]

Describe the enhancement:
Currently, Elastic Agent Autodiscovery with Hints supports a specific list of annotations.
The full list of both required and optional annotations supported can be found in
elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html#_required_hints

This feature will introduce an additional annotation co.elastic.hints.<container>/<configs>

Describe a specific use case for the enhancement or feature:
A user provides the manifest. Optionally the pod might have more than a single container

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        co.elastic.hints/package: "container_logs"
        co.elastic.hints.nginx/exclude_lines: '^DBG'
    spec:
      containers:
      - image: nginx
        name: nginx

Elastic-agent should generate an input definition similar to the following only for the container nginx. If there were more than a single container in the previous manifest, the input generated shouldn't contain the container-specific condition exclude_lines: '^DBG'

  - name: hints-filestream-container-logs
    id: hints-filestream-container-logs-${kubernetes.hints.container_id}
    type: filestream
    use_output: default
    streams:
      - condition: ${kubernetes.hints.container_logs.enabled} == true
        data_stream:
          dataset: kubernetes.container_logs
          type: logs
        parsers:
          - container:
              format: auto
              stream: ${kubernetes.hints.container_logs.stream|'all'}
        paths:
          - /var/log/containers/*${kubernetes.hints.container_id}.log
        prospector:
          scanner:
            symlinks: true
    data_stream.namespace: default
	exclude_lines: ['^DBG']

Similar work for supporting processors was done at this PR

What is the definition of done?

• to provide support for container-based configs similar to what filebeat provides at here
• To provide examples of working processors through annotations configuration
• Update the relevant documentation

Tasks

Preview Give feedback

1. Support Hints per container input #3416
   backport-skip +1
   
2. Updating doc for Hints Autodiscovery for multiple containers ingest-docs#503
   backport-skip +1

[gsantoro]

The line https://github.com/elastic/elastic-agent/pull/3107/files#diff-ed933ad9cf09eb9ca797c5c89b5e8991624cbe5301d88f996d9bd947f4e3ea16R512 should be modified from

hints := utils.GenerateHints(annotations, "", prefix)

to

hints := utils.GenerateHints(annotations, cID, prefix)

in order to pickup container specific annotations

[bartoszcisek]

Missing support for configuration per container is a significant problem for us. Prometheus module query each container in a pod even if only one exposes metrics. It results in 3-4 times more duplicated data than we need.