Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

loki structured metadata not sent unless using snappy compression #21443

Closed
victorserbu2709 opened this issue Oct 7, 2024 · 4 comments · Fixed by #21461
Closed

loki structured metadata not sent unless using snappy compression #21443

victorserbu2709 opened this issue Oct 7, 2024 · 4 comments · Fixed by #21461
Labels
sink: loki Anything `loki` sink related type: bug A code related bug.

Comments

@victorserbu2709
Copy link

victorserbu2709 commented Oct 7, 2024 •
edited
Loading

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I tried to use structured metadata with loki sink but i observed that these are not sent.
Simple replication:

sources:
  in:
    type: "stdin"

sinks:
  loki:
    type: loki
    inputs:
      - "in"
    labels:
      l1: v1
    out_of_order_action: accept
    path: /loki/api/v1/push
    remove_timestamp: true
    remove_label_fields: true
    remove_structured_metadata_fields: true
    structured_metadata:
      sml1: smv1
    tenant_id: "100312"
    endpoint: https://loki.novalocal
    encoding:
      codec: json
    auth:
      password: pass
      user: user
      strategy: basic
    compression: none

[root@nomadtesting bin]# ./vector --config vector.yaml
2024-10-07T19:15:40.307615Z  INFO vector::app: Log level is enabled. level="info"
2024-10-07T19:15:40.307990Z  INFO vector::app: Loading configs. paths=["vector.yaml"]
2024-10-07T19:15:40.309822Z  INFO vector::sources::file_descriptors: Capturing stdin.
2024-10-07T19:15:40.325301Z  INFO vector::topology::running: Running healthchecks.
2024-10-07T19:15:40.325360Z  INFO vector: Vector has started. debug="false" version="0.42.0" arch="x86_64" revision="51dcf8d 2024-10-06 04:01:48.303006361"
2024-10-07T19:15:40.325377Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2024-10-07T19:15:40.334661Z  INFO vector::topology::builder: Healthcheck passed.
test

This is the http body that is receive by nginx proxy

[root@lbgraf nginx]# cat "/var/lib/nginx/tmp/client_body/0000027329"
{"streams":[{"stream":{"l1":"v1"},"values":[["1728328544584658415","{\"host\":\"nomadtesting.novalocal\",\"message\":\"test\",\"source_type\":\"stdin\"}"]]}]}

i tried with vector rpm 0.41.0 and also with nighly build

[root@nomadtesting bin]# ./vector --config vector.yaml  --version
vector 0.42.0 (x86_64-unknown-linux-gnu 51dcf8d 2024-10-06 04:01:48.303006361)

Configuration

sources:
  in:
    type: "stdin"

sinks:
  loki:
    type: loki
    inputs:
      - "in"
    labels:
      l1: v1
    out_of_order_action: accept
    path: /loki/api/v1/push
    remove_timestamp: true
    remove_label_fields: true
    remove_structured_metadata_fields: true
    structured_metadata:
      sml1: smv1
    tenant_id: "100312"
    endpoint: https://loki.novalocal
    encoding:
      codec: json
    auth:
      password: pass
      user: user
      strategy: basic
    compression: none

Version

vector 0.42.0 (x86_64-unknown-linux-gnu 51dcf8d 2024-10-06 04:01:48.303006361)

Debug Output

[root@nomadtesting bin]# ./vector --config vector.yaml  -vvv                                                                                  
2024-10-07T19:20:38.928394Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-10-07T19:20:38.928426Z  INFO vector::app: Log level is enabled. level="trace"
2024-10-07T19:20:38.928481Z DEBUG vector::app: messaged="Building runtime." worker_threads=2
2024-10-07T19:20:38.928817Z  INFO vector::app: Loading configs. paths=["vector.yaml"]
2024-10-07T19:20:38.929331Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2024-10-07T19:20:38.929839Z DEBUG vector::topology::builder: Building new source. component=in
2024-10-07T19:20:38.930348Z DEBUG vector::topology::builder: Building new sink. component=loki
2024-10-07T19:20:38.930707Z  INFO vector::sources::file_descriptors: Capturing stdin.
2024-10-07T19:20:38.936503Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}: vector_core::tls::settings: Fetching system root certs.
2024-10-07T19:20:38.941281Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}: vector_core::tls::settings: Fetching system root certs.
2024-10-07T19:20:38.946426Z  INFO vector::topology::running: Running healthchecks.
2024-10-07T19:20:38.946440Z DEBUG vector::topology::running: Connecting changed/added component(s).
2024-10-07T19:20:38.946446Z DEBUG vector::topology::running: Configuring outputs for source. component=in
2024-10-07T19:20:38.946458Z DEBUG vector::topology::running: Configuring output for component. component=in output_id=None
2024-10-07T19:20:38.946468Z DEBUG vector::topology::running: Connecting inputs for sink. component=loki
2024-10-07T19:20:38.946476Z DEBUG vector::topology::running: Adding component input to fanout. component=loki fanout_id=in
2024-10-07T19:20:38.946494Z DEBUG vector::topology::running: Spawning new source. key=in
2024-10-07T19:20:38.946512Z TRACE vector::topology::running: Spawning new sink. key=loki
2024-10-07T19:20:38.946533Z  INFO vector: Vector has started. debug="false" version="0.42.0" arch="x86_64" revision="51dcf8d 2024-10-06 04:01:48.303006361"
2024-10-07T19:20:38.946542Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2024-10-07T19:20:38.946545Z DEBUG source{component_kind="source" component_id=in component_type=stdin}: vector::topology::builder: Source pump supervisor starting.
2024-10-07T19:20:38.946580Z DEBUG source{component_kind="source" component_id=in component_type=stdin}: vector::topology::builder: Source pump starting.
2024-10-07T19:20:38.946594Z DEBUG source{component_kind="source" component_id=in component_type=stdin}: vector::topology::builder: Source starting.
2024-10-07T19:20:38.946650Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}: vector::topology::builder: Sink starting.
2024-10-07T19:20:38.946947Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=https://loki.novalocal/ready method=GET version=HTTP/1.1 headers={"authorization": Sensitive, "user-agent": "
Vector/0.42.0 (x86_64-unknown-linux-gnu 51dcf8d 2024-10-06 04:01:48.303006361)", "accept-encoding": "identity"} body=[empty]
2024-10-07T19:20:38.947045Z TRACE http: hyper::client::pool: checkout waiting for idle connection: ("https", loki.novalocal)
2024-10-07T19:20:38.947081Z TRACE http: hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("loki.novalocal"), port=None
2024-10-07T19:20:38.947479Z DEBUG hyper::client::connect::dns: resolving host="loki.novalocal"
2024-10-07T19:20:38.948083Z TRACE vector: Beep.
2024-10-07T19:20:38.948109Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}: vector::utilization: utilization=0.022030624255513165
2024-10-07T19:20:38.949306Z DEBUG http: hyper::client::connect::http: connecting to 192.168.10.184:443
2024-10-07T19:20:38.950222Z DEBUG http: hyper::client::connect::http: connected to 192.168.10.184:443
2024-10-07T19:20:38.952991Z TRACE http: hyper::client::conn: client handshake Http1
2024-10-07T19:20:38.953023Z TRACE http: hyper::client::client: handshake complete, spawning background dispatcher task
2024-10-07T19:20:38.953061Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
2024-10-07T19:20:38.953084Z TRACE http: hyper::client::pool: checkout dropped for ("https", loki.novalocal)
2024-10-07T19:20:38.953119Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=GET, body=None
2024-10-07T19:20:38.953166Z DEBUG hyper::proto::h1::io: flushed 213 bytes
2024-10-07T19:20:38.953179Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2024-10-07T19:20:38.953531Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-10-07T19:20:38.953584Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2024-10-07T19:20:38.954724Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-10-07T19:20:38.954753Z TRACE hyper::proto::h1::io: received 308 bytes
2024-10-07T19:20:38.954766Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=308
2024-10-07T19:20:38.954789Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(155)
2024-10-07T19:20:38.954811Z DEBUG hyper::proto::h1::io: parsed 5 headers
2024-10-07T19:20:38.954819Z DEBUG hyper::proto::h1::conn: incoming body is content-length (153 bytes)
2024-10-07T19:20:38.954834Z TRACE hyper::proto::h1::decode: decode; state=Length(153)
2024-10-07T19:20:38.954849Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-10-07T19:20:38.954865Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2024-10-07T19:20:38.954880Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-10-07T19:20:38.954891Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-10-07T19:20:38.954904Z TRACE http: hyper::client::pool: put; add idle connection for ("https", loki.novalocal)
2024-10-07T19:20:38.954918Z DEBUG http: hyper::client::pool: pooling idle connection for ("https", loki.novalocal)
2024-10-07T19:20:38.954932Z DEBUG http: vector::internal_events::http_client: HTTP response. status=404 Not Found version=HTTP/1.1 headers={"server": "nginx/1.20.1", "date": "Mon, 07 Oct 2024 19:20:38 GMT", "con
tent-type": "text/html", "content-length": "153", "connection": "keep-alive"} body=[153 bytes]
2024-10-07T19:20:38.954970Z DEBUG vector::sinks::loki::healthcheck: Endpoint `/ready` not found. Retrying healthcheck with top level query.
2024-10-07T19:20:38.954995Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=https://loki.novalocal/ method=GET version=HTTP/1.1 headers={"authorization": Sensitive, "user-agent": "Vecto
r/0.42.0 (x86_64-unknown-linux-gnu 51dcf8d 2024-10-06 04:01:48.303006361)", "accept-encoding": "identity"} body=[empty]
2024-10-07T19:20:38.954970Z DEBUG vector::sinks::loki::healthcheck: Endpoint `/ready` not found. Retrying healthcheck with top level query.
2024-10-07T19:20:38.954995Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=https://loki.novalocal/ method=GET version=HTTP/1.1 headers={"authorization": Sensitive, "user-agent": "Vecto
r/0.42.0 (x86_64-unknown-linux-gnu 51dcf8d 2024-10-06 04:01:48.303006361)", "accept-encoding": "identity"} body=[empty]
2024-10-07T19:20:38.955013Z TRACE http: hyper::client::pool: take? ("https", loki.novalocal): expiration = Some(90s)
2024-10-07T19:20:38.955027Z DEBUG http: hyper::client::pool: reuse idle connection for ("https", loki.novalocal)
2024-10-07T19:20:38.955044Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=GET, body=None
2024-10-07T19:20:38.955073Z DEBUG hyper::proto::h1::io: flushed 208 bytes
2024-10-07T19:20:38.955086Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2024-10-07T19:20:38.956043Z TRACE hyper::client::pool: idle interval checking for expired
2024-10-07T19:20:38.956108Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-10-07T19:20:38.956132Z TRACE hyper::proto::h1::io: received 163 bytes
2024-10-07T19:20:38.956143Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=163
2024-10-07T19:20:38.956167Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(161)
2024-10-07T19:20:38.956178Z DEBUG hyper::proto::h1::io: parsed 5 headers
2024-10-07T19:20:38.956186Z DEBUG hyper::proto::h1::conn: incoming body is content-length (2 bytes)
2024-10-07T19:20:38.956200Z TRACE hyper::proto::h1::decode: decode; state=Length(2)
2024-10-07T19:20:38.956210Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-10-07T19:20:38.956224Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2024-10-07T19:20:38.956235Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-10-07T19:20:38.956245Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-10-07T19:20:38.956257Z TRACE http: hyper::client::pool: put; add idle connection for ("https", loki.novalocal)
2024-10-07T19:20:38.956268Z DEBUG http: hyper::client::pool: pooling idle connection for ("https", loki.novalocal)
2024-10-07T19:20:38.956280Z DEBUG http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"server": "nginx/1.20.1", "date": "Mon, 07 Oct 2024 19:20:38 GMT", "content-ty
pe": "application/octet-stream", "content-length": "2", "connection": "keep-alive"} body=[2 bytes]
2024-10-07T19:20:38.956305Z  INFO vector::topology::builder: Healthcheck passed.
2024-10-07T19:20:38.956318Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-10-07T19:20:39.948391Z TRACE vector: Beep.
2024-10-07T19:20:40.948627Z TRACE vector: Beep.

2024-10-07T19:20:41.314118Z TRACE source{component_kind="source" component_id=in component_type=stdin}: codecs::decoding::framing::character_delimited: Decoding the frame. bytes_processed=0
2024-10-07T19:20:41.314169Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-07T19:20:41.314191Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_common::internal_event::events_received: Events received. count=1 byte_size=14
2024-10-07T19:20:41.314254Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-07T19:20:41.314272Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=115 output=_default
2024-10-07T19:20:41.314295Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_core::fanout: Processing control message outside of send: ControlMessage::Add(ComponentKey { id: "lo
ki" })
2024-10-07T19:20:41.314330Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-07T19:20:41.314350Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_core::fanout: Sent item to fanout.
2024-10-07T19:20:41.314374Z TRACE sink{component_kind="sink" component_id=loki component_type=loki}: vector_common::internal_event::events_received: Events received. count=1 byte_size=115
2024-10-07T19:20:41.948173Z TRACE vector: Beep.

2024-10-07T19:20:42.094888Z TRACE source{component_kind="source" component_id=in component_type=stdin}: codecs::decoding::framing::character_delimited: Decoding the frame. bytes_processed=0
2024-10-07T19:20:42.094927Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-07T19:20:42.094945Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_common::internal_event::events_received: Events received. count=1 byte_size=14
2024-10-07T19:20:42.095175Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-07T19:20:42.095196Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=115 output=_default
2024-10-07T19:20:42.095227Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-07T19:20:42.095248Z TRACE source{component_kind="source" component_id=in component_type=stdin}: vector_core::fanout: Sent item to fanout.
2024-10-07T19:20:42.095275Z TRACE sink{component_kind="sink" component_id=loki component_type=loki}: vector_common::internal_event::events_received: Events received. count=1 byte_size=115
2024-10-07T19:20:42.316428Z TRACE sink{component_kind="sink" component_id=loki component_type=loki}: vector_stream::driver: Submitting service request. in_flight_requests=0 request_id=1
2024-10-07T19:20:42.316495Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://loki.novalocal/loki/api/v1/push method=POST version=HTTP/1.1 headers={"content-type": "application/json", "x-scope-orgid": "100312", "authorization": Sensitive, "user-agent": "Vector/0.42.0 (x86_64-unknown-linux-gnu 51dcf8d
2024-10-06 04:01:48.303006361)", "accept-encoding": "identity"} body=[261 bytes]
2024-10-07T19:20:42.316536Z TRACE sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=1}:http: hyper::client::pool: take? ("https", loki.novalocal): expiration = Some(90s)
2024-10-07T19:20:42.316557Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=1}:http: hyper::client::pool: reuse idle connection for ("https", loki.novalocal)
2024-10-07T19:20:42.316613Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=POST, body=Some(Known(261))
2024-10-07T19:20:42.316639Z TRACE hyper::proto::h1::io: buffer.flatten self.len=301 buf.len=261
2024-10-07T19:20:42.316687Z DEBUG hyper::proto::h1::io: flushed 562 bytes
2024-10-07T19:20:42.316701Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2024-10-07T19:20:42.320908Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-10-07T19:20:42.320908Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-10-07T19:20:42.320979Z TRACE hyper::proto::h1::io: received 144 bytes
2024-10-07T19:20:42.321042Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=144
2024-10-07T19:20:42.321100Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(144)
2024-10-07T19:20:42.321177Z DEBUG hyper::proto::h1::io: parsed 4 headers
2024-10-07T19:20:42.321235Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-10-07T19:20:42.321291Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2024-10-07T19:20:42.321343Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-10-07T19:20:42.321379Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-10-07T19:20:42.321423Z TRACE sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=1}:http: hyper::client::pool: put; add idle connection for ("https", loki.novalocal)
2024-10-07T19:20:42.321457Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=1}:http: hyper::client::pool: pooling idle connection for ("https", loki.novalocal)
2024-10-07T19:20:42.321489Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=204 No Content version=
HTTP/1.1 headers={"server": "nginx/1.20.1", "date": "Mon, 07 Oct 2024 19:20:42 GMT", "connection": "keep-alive", "x-envoy-upstream-service-time": "1"} body=[empty]
2024-10-07T19:20:42.321549Z TRACE sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=1}: vector_stream::driver: Service call succeeded. request_id=1
2024-10-07T19:20:42.321588Z TRACE sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=1}: vector_common::internal_event::bytes_sent: Bytes sent. byte_size=261 protocol=https
2024-10-07T19:20:42.321622Z TRACE sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=1}: vector_common::internal_event::events_sent: Events sent. count=2 byte_size=226
2024-10-07T19:20:42.948449Z TRACE vector: Beep.
2024-10-07T19:20:43.947692Z TRACE vector: Beep.
2024-10-07T19:20:43.947706Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}: vector::utilization: utilization=0.0022231775989010045
2024-10-07T19:20:44.947959Z TRACE vector: Beep.
^C2024-10-07T19:20:45.776797Z  INFO vector::signal: Signal received. signal="SIGINT"
2024-10-07T19:20:45.776888Z  INFO vector: Vector has stopped.
2024-10-07T19:20:45.776963Z DEBUG source{component_kind="source" component_id=in component_type=stdin}: vector::sources::file_descriptors: Finished sending.
2024-10-07T19:20:45.776999Z DEBUG source{component_kind="source" component_id=in component_type=stdin}: vector::topology::builder: Source finished normally.
2024-10-07T19:20:45.777041Z DEBUG source{component_kind="source" component_id=in component_type=stdin}: vector::topology::builder: Source pump finished normally.
2024-10-07T19:20:45.777081Z DEBUG source{component_kind="source" component_id=in component_type=stdin}: vector::topology::builder: Source pump supervisor task finished normally.
2024-10-07T19:20:45.778175Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="loki" time_remaining="59 seconds left"
2024-10-07T19:20:45.948391Z TRACE vector: Beep.
2024-10-07T19:20:46.948571Z TRACE vector: Beep.
2024-10-07T19:20:47.947632Z TRACE vector: Beep.
2024-10-07T19:20:48.947821Z TRACE vector: Beep.
2024-10-07T19:20:48.947924Z TRACE hyper::proto::h1::dispatch: client tx closed
2024-10-07T19:20:48.947943Z TRACE hyper::proto::h1::conn: State::close_read()
2024-10-07T19:20:48.947957Z TRACE hyper::proto::h1::conn: State::close_write()
2024-10-07T19:20:48.947971Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Closed, writing: Closed, keep_alive: Disabled }
2024-10-07T19:20:48.948044Z TRACE hyper::proto::h1::conn: shut down IO complete
2024-10-07T19:20:48.948578Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}: vector::topology::builder: Sink finished normally.
2024-10-07T19:20:48.948628Z TRACE hyper::client::pool: pool closed, canceling idle interval



### Example Data

_No response_

### Additional Context

_No response_

### References

_No response_
@victorserbu2709 victorserbu2709 added the type: bug A code related bug. label Oct 7, 2024
@jszwedko
Copy link
Member

jszwedko commented Oct 7, 2024

Thanks for filing this @victorserbu2709 including the details about the request payload. Could you try setting compression: snappy? That will cause the loki sink to use the protobuf encoding. I think structured metadata support may have only been added to the protobuf protocol and not to the JSON one.

If that is the case, we should make that clearer in the docs and/or make it explicit error to configure structured_metadata without configuring compression to be snappy. This is, assuming, that we can't just add structured metadata support when using the JSON API.

cc/ @maxboone since you implemented this support

@victorserbu2709
Copy link
Author

Thank you for your quick response @jszwedko.
I set compression: snappy and then checked in grafana that event has structured metadata.
Thank you
image

@maxboone
Copy link
Contributor

maxboone commented Oct 7, 2024 •
edited
Loading

cc/ @maxboone since you implemented this support

I see that there is support for it with the JSON API by adding the object at the end of the log line. I'll see if I can open an MR with that tomorrow.

Gave it a shot, but building for unit tests keep running out of memory, so will have to try on another machine tomorrow.

@jszwedko jszwedko changed the title loki structured metadata not sent loki structured metadata not sent unless using snappy compression Oct 8, 2024
@jszwedko jszwedko added the sink: loki Anything `loki` sink related label Oct 8, 2024
@jszwedko
Copy link
Member

jszwedko commented Oct 8, 2024

cc/ @maxboone since you implemented this support

I see that there is support for it with the JSON API by adding the object at the end of the log line. I'll see if I can open an MR with that tomorrow.

Gave it a shot, but building for unit tests keep running out of memory, so will have to try on another machine tomorrow.

That would be great 🙏 Much simpler than trying to document that the option can only be used with snappy/protobuf. It could be useful to deprecate and remove the old JSON API though. I'm unsure it it offers anything that the protobuf API doesn't 🤔

@maxboone maxboone mentioned this issue Oct 9, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sink: loki Anything `loki` sink related type: bug A code related bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(loki): serialize the structured metadata to JSON maxboone/vector
3 participants
@jszwedko @maxboone @victorserbu2709