issue with disk buffer - data_dir filling up with ldb files and data even though data is making it upstream

[ryn9]

I am having an issue with disk buffer in an http sink.
The data_dir filling up with ldb files and data even though data is making it upstream.

Vector Version

From timberio/vector:0.11.1-alpine

vector 0.11.1 (v0.11.1 x86_64-unknown-linux-musl 2020-12-17)

Vector Configuration File

Numerous input and transforms, but everything end up going out the one sink...

sinks:
  logs_out:
    type: http
    inputs:
      - logs_remove_fields
    compression: gzip
    uri: https://url.domain.name:10010/logs
    tls:
      ca_file: /etc/vector/ssl_auth/ca-certificates.crt
      crt_file: /etc/vector/ssl_auth/client.crt
      key_file: /etc/vector/ssl_auth/client.key
      verify_hostname: true
    encoding:
      codec: ndjson
      timestamp_format: rfc3339
    buffer:
      max_size: 10737418240
      type: disk
      when_full: block

Debug Output

Not performed

Expected Behavior

The data_dir should have buffered data only that has not been flushed.

Actual Behavior

The data_dir filling up with ldb files and data even though data is making it upstream.

Example Data

N/A

Additional Context

N/A

References

N/A

Additional Notes

I started asking about this on this Discord Timber server, in the support channel.
Please feel free to reach out via that method as well if more information is needed.

[binarylogic]

Thanks @ryn9, I think #6597 should resolve this. We'll have @ktff take a look.

[lukesteensen]

@binarylogic As noted on discord, this is a related but different issue from the one you linked. We requested that this one also be opened to track both aspects of the problem.

[streeck]

I observe exactly the same issue but on a statsd sink. I'm running it under Docker and the same image version as @ryn9.

My sink configuration:

[sinks.metrics_out]
  type = "statsd"
  inputs = ["metrics_in"]
  address = "address:port"
  mode = "udp"
  path = "/tmp/"

  encoding.codec = "json"

  buffer.type = "disk"
  buffer.max_size = 1073741824

[ktff]

I'm running it under Docker

@ryn9 reported the same so it seams the original issue still happens in Docker.

[ktff]

@ryn9 @streeck the fix for this has landed on master and is currently on nightly, while v0.14 should be released soon . Could you both check it out?

[ktff]

This is fixed in v0.14 but we can reopen it if it turns out that there are more causes.