Vector events in disk buffer not preserved after pod restart

[viktordebulat]

When restarting pod with disk buffer persistence enabled vector buffer doesn't try to send already got events from kafka but starts from 0 with current consumer, so events are lost. How this issue could be resolved or I missed something?

Use 0.31.0 chart version (the same issue was on 0.26.0 version too, so bump didn't help)

Persistence is enabled:

          persistence:
            enabled: true
            storageClassName: premium-rwo
            size: 10Gi

Typical sink config.

            sinks:  
              clickhouse:
                type: clickhouse
                healthcheck: true
                acknowledgements:
                  enabled: true
                inputs: ["kafka"]
                endpoint: _lorem-ipsum_
                database: _lorem-ipsum_
                table: _lorem-ipsum_
                skip_unknown_fields: true
                auth:
                  strategy: basic
                  user: vector
                  password: _lorem-ipsum_
                buffer:
                  type: disk
                  max_size: 5368709120
                  when_full: block
                batch:
                  max_bytes: 536870912
                  timeout_secs: 5
                request:
                  timeout_secs: 120

P.S. Using Aggregator role with 1 replica

[jszwedko]

Nothing immediately jumps out. Have you verified that when the pod is restarted, the volume is reattached with all of the disk buffer files contained? These would be in /var/lib/vector in the container.

[jszwedko]

I'd suggest looking at the metrics that the disk buffer publishes like https://vector.dev/docs/reference/configuration/sources/internal_metrics/#buffer_events. I'm curious if you see them reset after the pod is restarted.

[viktordebulat]

Ok

vector_buffer_events{buffer_type="disk",component_id="clickhouse_downstream_tables",component_kind="sink",component_type="clickhouse",host="clickhouse-test-vector-0",stage="0"} 20 1708549776625

Restarted

vector_buffer_events{buffer_type="disk",component_id="clickhouse_downstream_tables",component_kind="sink",component_type="clickhouse",host="clickhouse-test-vector-0",stage="0"} 3 1708549926116

The same with vector_buffer_size.
Before:

vector_buffer_byte_size{buffer_type="disk",component_id="clickhouse_downstream_tables",component_kind="sink",component_type="clickhouse",host="clickhouse-test-vector-0",stage="0"} 5640 1708550034116

After restart statefulset

vector_buffer_byte_size{buffer_type="disk",component_id="clickhouse_downstream_tables",component_kind="sink",component_type="clickhouse",host="clickhouse-test-vector-0",stage="0"} 2424 1708550151323

But events were not fan out from sink 100% because I broke connection to clickhouse to test this

WARN sink{component_kind="sink" component_id=clickhouse_downstream_tables component_type=clickhouse}:request{request_id=1}: vector::sinks::util::retries: Retrying aft
er error. error=Failed to make HTTP(S) request: error trying to connect: error:0A000086:SSL routines:(unknown function):certificate verify failed:ssl/statem/statem_clnt.c:2092:: unable to get local issuer certificate int
ernal_log_rate_limit=true

[viktordebulat]

What I've found that persistence volume is mounted by mountPath: /vector-data-dir but buffer is stored in /var/lib/vector
So, after restart it is wiped

Any reason why default value is /vector-data-dir in pod?

      - name: data
        {{- if .Values.existingConfigMaps }}
        mountPath: "{{ if .Values.dataDir }}{{ .Values.dataDir }}{{ else }}{{ fail "Specify `dataDir` if you're using `existingConfigMaps`" }}{{ end }}"
        {{- else }}
        mountPath: "{{ .Values.customConfig.data_dir | default "/vector-data-dir" }}"
        {{- end }}

Clarifying just to understand, should I open PR for this.

[jszwedko]

Ah, you are right, I forgot the data directory is mounted in a different directory by the chart. This is to workaround the fact that /var/lib is mounted as a read-only filesystem as mentioned by #226 (comment)