Loki + Kubernetes labels question

[uhthomas]

Hey! I've been trying to setup Vector on my cluster, but am struggling with the config as it seems various resources are outdated, or unclear as to how to approach Loki and Kubernetes.

Loki config:

data_dir = "/var/lib/vector"
[sources.kubernetes]
  type = "kubernetes"
[sinks.loki]
  type = "loki"
  inputs = ["kubernetes"]
  endpoint = "http://loki:3100"
  encoding.codec = "json"

The output of running this is:

➜  ~ kubectl -n vector logs vector-4d5np vector
May 29 18:27:28.838  INFO vector: Log level "info" is enabled.
May 29 18:27:28.838  INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
May 29 18:27:28.841 ERROR vector: Configuration error: "/etc/vector/vector.toml": missing field `labels` for key `sinks.loki`

What sort of labels should I be putting the config?

How do I extract k8s labels and ship them to Loki?

See Loki's fluent-bit config for what sorts of labels (I assume) should be added.

[pySilver]

this is confusing, im also having troubles running loki integration with grafana cloud

[MOZGIII]

From the loki docs:

Unlike other logging systems, Loki is built around the idea of only indexing labels for logs and leaving the original log message unindexed. This means that Loki is cheaper to operate and can be orders of magnitude more efficient.

We're currently working on a new integration with Kubernetes! Please take a look #2653 - it will be shipping soon.

If you can't wait, you can try a development build, for instance from here: #2653 (comment).

With our new implementation, you'll be able to configure Vector like this:

data_dir = "/var/lib/vector"

[sources.kubernetes_logs]
  type = "kubernetes_logs"

[sinks.loki]
  type = "loki"
  inputs = ["kubernetes_logs"]
  endpoint = "http://loki.loki.svc.cluster.local:3100"
  encoding.codec = "json"
  batch.max_events = 10000
  [sinks.loki.labels]
    file = "{{ file }}"
    stream = "{{ stream }}"
    source_type = "{{ source_type }}"
    k8s_pod_namespace = "{{ kubernetes.pod_namespace }}"
    k8s_pod_name = "{{ kubernetes.pod_name }}"
    k8s_pod_uid = "{{ kubernetes.pod_uid }}"

[rrichardson]

I got the deployment working using the latest from the k8s-impl branch from #2653
However, the process crashes immediately with a "capacity overflow" in the Loki sink configuration.

Here is my config:

  managed.toml: |
    # Data dir is location controlled at the `DaemonSet`.
    data_dir = "/vector-data-dir"

    # Ingest logs from Kubernetes.
    [sources.kubernetes_logs]
      type = "kubernetes_logs"

    [sinks.out]
      # General
      type = "loki"
      inputs = ["kubernetes_logs"]
      endpoint = "http://loki.loki.svc.cluster.local:3100"
      healthcheck = true
      encoding.codec = "json"
      [sinks.out.labels]
        k8s_pod_name = "{{ kubernetes.pod_name }}"
        k8s_pod_uid = "{{ kubernetes.pod_uid }}"

here is the log with backtrace:

Jul 09 21:14:07.635  INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-421-g33de5ef" released="Thu, 09 Jul 2020 20:33:25 +0000" arch="x86_64"
Jul 09 21:14:07.635  INFO vector::sources::kubernetes_logs: obtained Kubernetes Node name to collect logs for (self) self_node_name="ip-10-210-70-174.ec2.internal"
thread 'main' panicked at 'capacity overflow', src/liballoc/raw_vec.rs:622:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at ./cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at ./cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1069
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1504
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:511
  11: rust_begin_unwind
             at src/libstd/panicking.rs:419
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:111
  13: core::panicking::panic
             at src/libcore/panicking.rs:54
  14: alloc::raw_vec::capacity_overflow
             at src/liballoc/raw_vec.rs:622
  15: alloc::raw_vec::RawVec<T,A>::allocate_in::{{closure}}
  16: <vector::sinks::loki::LokiConfig as vector::topology::config::SinkConfig>::build
  17: vector::topology::builder::build_pieces
  18: vector::topology::builder::check_build
  19: vector::topology::validate
  20: vector::main::{{closure}}
  21: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  22: <futures_util::compat::compat03as01::Compat<Fut> as futures::future::Future>::poll
  23: futures::task_impl::std::set
  24: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  25: tokio::runtime::enter::Enter::block_on
  26: tokio::runtime::context::enter
  27: tokio::runtime::Runtime::block_on
  28: tokio_compat::runtime::threadpool::Runtime::block_on
  29: vector::main
  30: std::rt::lang_start::{{closure}}
  31: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  32: std::panicking::try::do_call
             at src/libstd/panicking.rs:331
  33: std::panicking::try
             at src/libstd/panicking.rs:274
  34: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  35: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  36: main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[MOZGIII]

I was able to reproduce this issue on my cluster

[MOZGIII]

It's an issue with loki sink, I created #3006 to track it.

[binarylogic]

Closing since #3006 was closed with #3016