Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Hints] K8s autodiscovery does not work when data_streams are not specified #3057

Closed
ChrsMark opened this issue Jul 11, 2023 · 1 comment
Closed

[Hints] K8s autodiscovery does not work when data_streams are not specified #3057

ChrsMark opened this issue Jul 11, 2023 · 1 comment
Assignees
@ChrsMark
Labels
bug Something isn't working

Comments

@ChrsMark
Copy link
Member

Following the example from https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html#_example_hints_autodiscovery if we remove the data_stream hint from the annotation it seems that the data are not shipped any more.

We need to investigate this but I suspect there is something to do with how the id of the inputs is populated:

inputs:
- data_stream.namespace: default
  id: kubernetes-33174936-0ea7-451f-90d0-122cef6c8e0b
  name: redis-redis
  processors:
  - add_fields:
      fields:
        labels:
          app: redis-static
          k8s-app: redis
        namespace: default
        namespace_uid: 92fae074-df58-4e97-8772-a04847d2880f
        node:
          hostname: kind-control-plane
          labels:
            beta_kubernetes_io/arch: amd64
            beta_kubernetes_io/os: linux
            kubernetes_io/arch: amd64
            kubernetes_io/hostname: kind-control-plane
            kubernetes_io/os: linux
            node-role_kubernetes_io/master: ""
          name: kind-control-plane
          uid: b0bbc9db-8497-4726-8bd9-ff60616a24a6
        pod:
          ip: 10.244.0.6
          name: redis
          uid: 33174936-0ea7-451f-90d0-122cef6c8e0b
      target: kubernetes
  - add_fields:
      fields:
        cluster:
          name: kind
          url: kind-control-plane:6443
      target: orchestrator
  streams:
  - data_stream:
      dataset: redis.slowlog
      type: logs
    hosts:
    - 10.244.0.6:6379
    password: ""
  type: redis
  use_output: default
- data_stream.namespace: default
  id: kubernetes-33174936-0ea7-451f-90d0-122cef6c8e0b
  name: redis/metrics-redis
  processors:
  - add_fields:
      fields:
        labels:
          app: redis-static
          k8s-app: redis
        namespace: default
        namespace_uid: 92fae074-df58-4e97-8772-a04847d2880f
        node:
          hostname: kind-control-plane
          labels:
            beta_kubernetes_io/arch: amd64
            beta_kubernetes_io/os: linux
            kubernetes_io/arch: amd64
            kubernetes_io/hostname: kind-control-plane
            kubernetes_io/os: linux
            node-role_kubernetes_io/master: ""
          name: kind-control-plane
          uid: b0bbc9db-8497-4726-8bd9-ff60616a24a6
        pod:
          ip: 10.244.0.6
          name: redis
          uid: 33174936-0ea7-451f-90d0-122cef6c8e0b
      target: kubernetes
  - add_fields:
      fields:
        cluster:
          name: kind
          url: kind-control-plane:6443
      target: orchestrator
  streams:
  - data_stream:
      dataset: redis.info
      type: metrics
    hosts:
    - 10.244.0.6:6379
    idle_timeout: 20s
    maxconn: 10
    metricsets:
    - info
    network: tcp
    password: ""
    period: 10s
  - data_stream:
      dataset: redis.key
      type: metrics
    hosts:
    - 10.244.0.6:6379
    idle_timeout: 20s
    key.patterns:
    - limit: 20
      pattern: '*'
    maxconn: 10
    metricsets:
    - key
    network: tcp
    password: ""
    period: 10s
  - data_stream:
      dataset: redis.keyspace
      type: metrics
    hosts:
    - 10.244.0.6:6379
    idle_timeout: 20s
    maxconn: 10
    metricsets:
    - keyspace
    network: tcp
    password: ""
    period: 10s
  type: redis/metrics
  use_output: default
- data_stream.namespace: default
  id: kubernetes-33174936-0ea7-451f-90d0-122cef6c8e0b.redis42
  name: filestream-redis
  processors:
  - add_fields:
      fields:
        id: d68cd2cd67642bd467dfaebc17bbab4ed77a17b4621332b7adab2b4d2fb655ef
        image:
          name: redis
        runtime: containerd
      target: container
  - add_fields:
      fields:
        container:
          name: redis42
        labels:
          app: redis-static
          k8s-app: redis
        namespace: default
        namespace_uid: 92fae074-df58-4e97-8772-a04847d2880f
        node:
          hostname: kind-control-plane
          labels:
            beta_kubernetes_io/arch: amd64
            beta_kubernetes_io/os: linux
            kubernetes_io/arch: amd64
            kubernetes_io/hostname: kind-control-plane
            kubernetes_io/os: linux
            node-role_kubernetes_io/master: ""
          name: kind-control-plane
          uid: b0bbc9db-8497-4726-8bd9-ff60616a24a6
        pod:
          ip: 10.244.0.6
          name: redis
          uid: 33174936-0ea7-451f-90d0-122cef6c8e0b
      target: kubernetes
  - add_fields:
      fields:
        cluster:
          name: kind
          url: kind-control-plane:6443
      target: orchestrator
  streams:
  - data_stream:
      dataset: redis.log
      type: logs
    exclude_files:
    - .gz$
    exclude_lines:
    - ^\s+[\-`('.|_]
    parsers:
    - container:
        format: auto
        stream: all
    paths:
    - /var/log/containers/*d68cd2cd67642bd467dfaebc17bbab4ed77a17b4621332b7adab2b4d2fb655ef.log
    prospector:
      scanner:
        symlinks: true
    tags:
    - redis-log
  type: filestream
  use_output: default

Note that the kubernetes-33174936-0ea7-451f-90d0-122cef6c8e0b is duplicated in the above policy.
@ChrsMark ChrsMark added the bug Something isn't working label Jul 11, 2023
@ChrsMark
Copy link
Member Author

The id not being unique seems to be the issue. After fixing this I see data flowing in and the policy looks like the following:

inputs:
- data_stream.namespace: default
  id: redis-slowlog-kubernetes-33174936-0ea7-451f-90d0-122cef6c8e0b
  name: redis-redis
  original_id: redis-slowlog
  processors:
  - add_fields:
      fields:
        cluster:
          name: kind
          url: kind-control-plane:6443
      target: orchestrator
  - add_fields:
      fields:
        labels:
          app: redis-static
          k8s-app: redis42ewew
        namespace: default
        namespace_uid: 92fae074-df58-4e97-8772-a04847d2880f
        node:
          hostname: kind-control-plane
          labels:
            beta_kubernetes_io/arch: amd64
            beta_kubernetes_io/os: linux
            kubernetes_io/arch: amd64
            kubernetes_io/hostname: kind-control-plane
            kubernetes_io/os: linux
            node-role_kubernetes_io/master: ""
          name: kind-control-plane
          uid: b0bbc9db-8497-4726-8bd9-ff60616a24a6
        pod:
          ip: 10.244.0.6
          name: redis
          uid: 33174936-0ea7-451f-90d0-122cef6c8e0b
      target: kubernetes
  streams:
  - data_stream:
      dataset: redis.slowlog
      type: logs
    hosts:
    - 10.244.0.6:6379
    password: ""
  type: redis
  use_output: default
- data_stream.namespace: default
  id: redis-metrics-kubernetes-33174936-0ea7-451f-90d0-122cef6c8e0b
  name: redis/metrics-redis
  original_id: redis-metrics
  processors:
  - add_fields:
      fields:
        cluster:
          name: kind
          url: kind-control-plane:6443
      target: orchestrator
  - add_fields:
      fields:
        labels:
          app: redis-static
          k8s-app: redis42ewew
        namespace: default
        namespace_uid: 92fae074-df58-4e97-8772-a04847d2880f
        node:
          hostname: kind-control-plane
          labels:
            beta_kubernetes_io/arch: amd64
            beta_kubernetes_io/os: linux
            kubernetes_io/arch: amd64
            kubernetes_io/hostname: kind-control-plane
            kubernetes_io/os: linux
            node-role_kubernetes_io/master: ""
          name: kind-control-plane
          uid: b0bbc9db-8497-4726-8bd9-ff60616a24a6
        pod:
          ip: 10.244.0.6
          name: redis
          uid: 33174936-0ea7-451f-90d0-122cef6c8e0b
      target: kubernetes
  streams:
  - data_stream:
      dataset: redis.info
      type: metrics
    hosts:
    - 10.244.0.6:6379
    idle_timeout: 20s
    maxconn: 10
    metricsets:
    - info
    network: tcp
    password: ""
    period: 10s
  - data_stream:
      dataset: redis.key
      type: metrics
    hosts:
    - 10.244.0.6:6379
    idle_timeout: 20s
    key.patterns:
    - limit: 20
      pattern: '*'
    maxconn: 10
    metricsets:
    - key
    network: tcp
    password: ""
    period: 10s
  - data_stream:
      dataset: redis.keyspace
      type: metrics
    hosts:
    - 10.244.0.6:6379
    idle_timeout: 20s
    maxconn: 10
    metricsets:
    - keyspace
    network: tcp
    password: ""
    period: 10s
  type: redis/metrics
  use_output: default
- data_stream.namespace: default
  id: kubernetes-33174936-0ea7-451f-90d0-122cef6c8e0b.redis42
  name: filestream-redis
  processors:
  - add_fields:
      fields:
        id: d68cd2cd67642bd467dfaebc17bbab4ed77a17b4621332b7adab2b4d2fb655ef
        image:
          name: redis
        runtime: containerd
      target: container
  - add_fields:
      fields:
        cluster:
          name: kind
          url: kind-control-plane:6443
      target: orchestrator
  - add_fields:
      fields:
        container:
          name: redis42
        labels:
          app: redis-static
          k8s-app: redis42ewew
        namespace: default
        namespace_uid: 92fae074-df58-4e97-8772-a04847d2880f
        node:
          hostname: kind-control-plane
          labels:
            beta_kubernetes_io/arch: amd64
            beta_kubernetes_io/os: linux
            kubernetes_io/arch: amd64
            kubernetes_io/hostname: kind-control-plane
            kubernetes_io/os: linux
            node-role_kubernetes_io/master: ""
          name: kind-control-plane
          uid: b0bbc9db-8497-4726-8bd9-ff60616a24a6
        pod:
          ip: 10.244.0.6
          name: redis
          uid: 33174936-0ea7-451f-90d0-122cef6c8e0b
      target: kubernetes
  streams:
  - data_stream:
      dataset: redis.log
      type: logs
    exclude_files:
    - .gz$
    exclude_lines:
    - ^\s+[\-`('.|_]
    parsers:
    - container:
        format: auto
        stream: all
    paths:
    - /var/log/containers/*d68cd2cd67642bd467dfaebc17bbab4ed77a17b4621332b7adab2b4d2fb655ef.log
    prospector:
      scanner:
        symlinks: true
    tags:
    - redis-log
  type: filestream
  use_output: default
outputs:
  default:
    hosts:
    - https://test-hints.es.europe-west4.gcp.elastic-cloud.com:9243
    password: YOn55rYkH9tihOXG1lTU1p1t
    type: elasticsearch
    username: elastic

In order to update the redis template I exec inside the Agent Pod and run the following:

cat > redis.txt << 'EOL'
inputs:
    - name: filestream-redis
      type: filestream
      use_output: default
      streams:
        - condition: ${kubernetes.hints.redis.log.enabled} == true or ${kubernetes.hints.redis.enabled} == true
          data_stream:
            dataset: redis.log
            type: logs
          exclude_files:
            - .gz$
          exclude_lines:
            - ^\s+[\-`('.|_]
          parsers:
            - container:
                format: auto
                stream: ${kubernetes.hints.redis.log.stream|'all'}
          paths:
            - /var/log/containers/*${kubernetes.hints.container_id}.log
          prospector:
            scanner:
                symlinks: true
          tags:
            - redis-log
      data_stream.namespace: default
    - name: redis-redis
      type: redis
      id: redis-slowlog
      use_output: default
      streams:
        - condition: ${kubernetes.hints.redis.slowlog.enabled} == true or ${kubernetes.hints.redis.enabled} == true
          data_stream:
            dataset: redis.slowlog
            type: logs
          hosts:
            - ${kubernetes.hints.redis.slowlog.host|kubernetes.hints.redis.host|'127.0.0.1:6379'}
          password: ${kubernetes.hints.redis.slowlog.password|kubernetes.hints.redis.password|''}
      data_stream.namespace: default
    - name: redis/metrics-redis
      id: redis-metrics
      type: redis/metrics
      use_output: default
      streams:
        - condition: ${kubernetes.hints.redis.info.enabled} == true or ${kubernetes.hints.redis.enabled} == true
          data_stream:
            dataset: redis.info
            type: metrics
          hosts:
            - ${kubernetes.hints.redis.info.host|kubernetes.hints.redis.host|'127.0.0.1:6379'}
          idle_timeout: 20s
          maxconn: 10
          metricsets:
            - info
          network: tcp
          password: ${kubernetes.hints.redis.info.password|kubernetes.hints.redis.password|''}
          period: ${kubernetes.hints.redis.info.period|kubernetes.hints.redis.period|'10s'}
        - condition: ${kubernetes.hints.redis.key.enabled} == true or ${kubernetes.hints.redis.enabled} == true
          data_stream:
            dataset: redis.key
            type: metrics
          hosts:
            - ${kubernetes.hints.redis.key.host|kubernetes.hints.redis.host|'127.0.0.1:6379'}
          idle_timeout: 20s
          key.patterns:
            - limit: 20
              pattern: '*'
          maxconn: 10
          metricsets:
            - key
          network: tcp
          password: ${kubernetes.hints.redis.key.password|kubernetes.hints.redis.password|''}
          period: ${kubernetes.hints.redis.key.period|kubernetes.hints.redis.period|'10s'}
        - condition: ${kubernetes.hints.redis.keyspace.enabled} == true or ${kubernetes.hints.redis.enabled} == true
          data_stream:
            dataset: redis.keyspace
            type: metrics
          hosts:
            - ${kubernetes.hints.redis.keyspace.host|kubernetes.hints.redis.host|'127.0.0.1:6379'}
          idle_timeout: 20s
          maxconn: 10
          metricsets:
            - keyspace
          network: tcp
          password: ${kubernetes.hints.redis.keyspace.password|kubernetes.hints.redis.password|''}
          period: ${kubernetes.hints.redis.keyspace.period|kubernetes.hints.redis.period|'10s'}
      data_stream.namespace: default
EOL

and then cp redis.txt /etc/elastic-agent/inputs.d/redis.yml. Unfortunately I cannot install vi/vim into the Agent container.

cc: @gizas @LucaWintergerst @mlunadia

@gizas gizas mentioned this issue Jul 12, 2023
Open
10 tasks
@ChrsMark ChrsMark self-assigned this Jul 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ChrsMark ChrsMark

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ChrsMark