Panic when http_server using native decoding receives metric

[dekelpilli]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Vector panics when receiving native metrics via the http_server source, because it tries to coerce the event into a log event (but can't). This is interesting, because I tried linking the http_server source to metric_to_log and got Configuration error. error=Data type mismatch between h (Log) and x (Metric), but when I changed the decoding to use native_json, it was able to start up, which indicates to me that it is intended be able to output metrics. If this is not the case, I'd be happy to open another issue to request that as a feature.

Configuration

{
  "log_schema": {
    "source_type_key": ""
  },
  "sources": {
    "h": 
    {
      "type": "http",
      "acknowledgements": false,
      "address": "0.0.0.0:${PORT:-1234}",
      "decoding": {"codec": "native_json"},
      "framing": {"method":"newline_delimited"},
      "path_key": ""
    }
  },
  "transforms": {
    "x": {
      "type": "metric_to_log",
      "inputs": ["h"]
    }
  },
  "sinks": {
    "serr": {
      "encoding": {
        "codec": "json"
      },
      "inputs": ["x"],
      "target": "stderr",
      "type": "console"
    }
  }
}

Version

vector 0.28.1 (x86_64-apple-darwin ff15924 2023-03-06)

Debug Output

thread 'vector-worker' panicked at 'Failed type coercion, Metric(Metric { series: MetricSeries { name: MetricName { name: "test_metric", namespace: Some("abc") }, tags: None }, data: MetricData { time: MetricTime { timestamp: None, interval_ms: None }, kind: Incremental, value: Counter { value: 2.0 } }, metadata: EventMetadata { value: Object({}), secrets: {}, finalizers: EventFinalizers([]), schema_definition: Definition { event_kind: Kind { bytes: Some(()), integer: Some(()), float: Some(()), boolean: Some(()), timestamp: Some(()), regex: Some(()), null: Some(()), undefined: Some(()), array: Some(Collection { known: {}, unknown: Unknown(Infinite(Infinite { bytes: Some(()), integer: Some(()), float: Some(()), boolean: Some(()), timestamp: Some(()), regex: Some(()), null: Some(()), array: Some(()), object: Some(()) })) }), object: Some(Collection { known: {}, unknown: Unknown(Infinite(Infinite { bytes: Some(()), integer: Some(()), float: Some(()), boolean: Some(()), timestamp: Some(()), regex: Some(()), null: Some(()), array: Some(()), object: Some(()) })) }) }, metadata_kind: Kind { bytes: None, integer: None, float: None, boolean: None, timestamp: None, regex: None, null: None, undefined: None, array: None, object: Some(Collection { known: {}, unknown: Unknown(Exact(Kind { bytes: None, integer: None, float: None, boolean: None, timestamp: None, regex: None, null: None, undefined: Some(()), array: None, object: None })) }) }, meaning: {}, log_namespaces: {Vector, Legacy} } } }) is not a log event', lib/vector-core/src/event/mod.rs:114:18
stack backtrace:
   0:        0x108aa808c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8d94e552d95b28cc
   1:        0x1081019ca - core::fmt::write::h421d4212716e9716
   2:        0x108a7487c - std::io::Write::write_fmt::hdc28b71c2d62dad8
   3:        0x108aabbaa - std::sys_common::backtrace::print::he11eab6b959c3b5b
   4:        0x108aab6f6 - std::panicking::default_hook::{{closure}}::ha68ba8cbe26bbbe3
   5:        0x108aac960 - std::panicking::rust_panic_with_hook::hed342721bf9addfa
   6:        0x108aac2f4 - std::panicking::begin_panic_handler::{{closure}}::h3d9af89e51f2fba9
   7:        0x108aac258 - std::sys_common::backtrace::__rust_end_short_backtrace::hfb9719355016e93f
   8:        0x108aac222 - _rust_begin_unwind
   9:        0x10c11a183 - core::panicking::panic_fmt::h1965fc2159be50bb
  10:        0x10a5a5a14 - <warp::filter::or::EitherFuture<T,U> as core::future::future::Future>::poll::he5d62b1bc725e131
  11:        0x10a598516 - <warp::filter::service::FilteredFuture<F> as core::future::future::Future>::poll::h68fe8d8310ae945b
  12:        0x10a592e97 - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_inner::hbbed59ed2f9d1028
  13:        0x10a58c129 - <hyper::server::server::new_svc::NewSvcTask<I,N,S,E,W> as core::future::future::Future>::poll::hfd06254d1dd3803e
  14:        0x10a5aa60b - tokio::runtime::task::raw::poll::h1bb8a8b08564a238
  15:        0x108af41c9 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h9ef9e5dbc7adc4bf
  16:        0x108afb943 - tokio::runtime::task::raw::poll::hef90a700f7359d7f
  17:        0x108adb9a7 - std::sys_common::backtrace::__rust_begin_short_backtrace::h5f7dbed508b337dd
  18:        0x108adb642 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hf55fbf1f56140835
  19:        0x108aaf747 - std::sys::unix::thread::Thread::new::thread_start::h1311e3ea18e7aa94
  20:     0x7ff81d0b31d3 - __pthread_start

Example Data

{"metric":{"name":"test_metric","namespace":"abc","kind":"incremental","counter":{"value":2}}}

Additional Context

No response

References

• How to increase through put for Vector consuming from kafka #12085 (this indicates to me that native_json should allow sources that are usually log-only to output metrics)

[neuronull]

Thanks for the report @dekelpilli .

I was able to reproduce that with your provided config and example data (much appreciated).

The sources that support the native and native_json codecs should be able to support metrics in that way.

Checking the source code of http_server source, I do see we are assuming the event is a log in multiple places, and that this seems to have been this way since the beginning, and the source was not adapted to work with metrics with the codec support changes.

Snippet from the full backtrace:

  15:        0x109245028 - vector_core::event::Event::as_mut_log::h581c9cc8761c5290
                               at workspace/vector/lib/vector-core/src/event/mod.rs:114:18
  16:        0x1042cdee8 - vector::sources::http_server::SimpleHttpSource::enrich_events::h582ec74859e4fa02
                               at workspace/vector/src/sources/http_server.rs:380:23
  17:        0x1042ce7e0 - <vector::sources::http_server::SimpleHttpSource as vector::sources::util::http::prelude::HttpSource>::build_events::hc481c9848c6efe06
                               at workspace/vector/src/sources/http_server.rs:456:9

[ryanartecona]

I have a use case for collecting metrics and logs in a Vector aggregator from an untrusted Vector agent, and I'd love to be able to use an http source to collect both metrics and logs (so that I can authenticate payloads). I hit this error myself, too. Any prioritization update?

[neuronull]

We have not yet queued this one into the short term roadmap, so it's hard to give an accurate timeline for it.

Do add a "👍" to the initial comment to "vote" on it, that does help.

[jszwedko]

We also accept PRs 🙂 This one should involve updating any places in the http_server source that assume the event is a log event.

[ryanartecona]

it's hard to give an accurate timeline for it

Thanks, no worries.

Do add a "👍" to the initial comment to "vote" on it, that does help.

I upvoted the PR description, assuming that's what you meant.

We also accept PRs 🙂 This one should involve updating any places in the http_server source that assume the event is a log event.

That doesn't sound too bad. I'm still in a prototype/demo stage for my project at the moment, but if I get further and still need this I may take a crack at it.

[ee07b415]

Hi here! I'm really glad we got this so fast, do we know if this fix will be include in the next release like v0.34.0?

[jszwedko]

Hi here! I'm really glad we got this so fast, do we know if this fix will be include in the next release like v0.34.0?

Yep!