fix(kafka source): fix acknowledgement handling during shutdown and rebalance events

[jches]

This adds some integration tests and a fix for the kafka source to fully drain acknowledgements during consumer group rebalancing and at shutdown, and should close #14761

Prior related work: #16928

Overview

I discussed this a bit over in #16827, the implementation here essentially follows that outline:

• Run a separate task to handle acknowledgements (additionally, handle acknowledgements for separate kafka partitions on separate streams, for cleaner handling when rebalancing partition assignments)
• Add coordination between the kafka client task and acknowledgement task, so that at rebalance and shutdown time, we have the best possible chance of processing all pending acknowledgements

Here is a visual representation of what I'm going for:

A few oddities come up around how rdkafka, and particularly the rebalance handlers work in async code. The pre_rebalance hook is called internally by the kafka client while polling for new messages on the message stream, so that task (and the rebalance process itself) will be blocked until it returns. That callback handler isn't async, so it can't yield or receive on async channels, and must run on a spawn_blocking thread. The task handling acknowledgements can be async and can run as a normal tokio::spawn task.

This just has implications for what can be used to coordinate between the two:

• the message task (on which rebalance hooks run) can send messages on a tokio mpsc channel to the acknowledgement task, but must use an Unbounded channel since sending on a bounded tokio channel requires an async context. Overlapping rebalance events will not happen, so this unbounded channel should never have more than 1 message on it.
• the message task can receive on a std::sync::mpsc::sync_channel. This uses a bound size 0, to act as a rendezvous channel, so the acknowledgement task can signal when acks have drained (or when a timeout is reached).

I also found that to properly test this, the source needed to be able to "consume until the end of the partition(s)" rather than "consume a specified number of messages". This is enabled by the enable.partition.eof flag, which I've enabled as an option (gated by cfg(test) in a couple places where it's needed outside of the actual test module).

Testing locally

When running these tests locally, I've found it is easiest to use an existing kafka topic pre-populated with, say 100k messages (for some reason, the way the tests write messages in to kafka is very slow; I haven't looked into why that is, yet). For the automated runs in CI we may need to tune the number of messages or the timings, since this is essentially testing a race condition, it can be a bit performance-dependent. I've put in assertions to avoid a false "pass" (where the first consumer takes in all the messages before shutdown, for example). I've had no trouble triggering the bug when running the tests against a pre-populated topic.

Here is how I've set this up locally:

❯ kcat -b $BROKER_ADDR -P -t TestTopic -K : -l <(for i in $(seq 1 100000); do echo "${i}:{\"value\": ${i}}"; done);

...then running the tests, targeting that topic through environment variables. This can be a bit finicky with regard to timings, so KAFKA_SHUTDOWN_DELAY controls how long to run the first consumer before shutting down (for the drain at shutdown test), and KAFKA_CONSUMER_DELAY controls the time between starting new consumers during the rebalancing test.

❯ KAFKA_SEND_COUNT=0 \
  KAFKA_EXPECT_COUNT=100000 \
  KAFKA_TEST_TOPIC=TestTopic \
  KAFKA_CONSUMER_DELAY=5000 \
  KAFKA_SHUTDOWN_DELAY=5000 \
  KAFKA_HOST=$BROKER_ADDR \
  KAFKA_PORT=9092 \
    cargo test --lib --no-default-features -F sources-kafka -F kafka-integration-tests drains_acknowledgement

Additional testing / notes

• The first pass I made at this used StreamMap from the tokio library for combining the partition-specific streams. The StreamMap docs note potential performance issues when used with a large number of streams, and indeed, when testing against a topic with 400 partitions this was significantly slower. So I refactored using forwarding tasks and channels, which need a bit more bookkeeping but scales much better
• There are some test failures around how assert_source_compliance works, and I believe it's related to some thread-local variables used to collect data from emit! macros. I've not dug deeply into this yet and might need help there

[netlify]

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	ff086d7
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/651de10fd6ce620008bdda0d

[netlify]

✅ Deploy Preview for vrl-playground canceled.

Name	Link
🔨 Latest commit	ff086d7
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/651de10fdd9c7700085d94f0

[jszwedko]

Thanks for this @jches ! Do you want reviews yet?

[jches]

Hey @jszwedko! I had left this as a draft intending to look at the failing assert_source_compliance tests in more detail before requesting a review. I didn't end up having time this week, but I might jump in to discord next week and ask around if I need more help there.

[dsmith3197]

HI @jches, I just wanted to let you know this is on my radar and I'll get you a review shortly.

[dsmith3197]

Thank you for the contribution @jches! I know this is a tricky problem to solve and we appreciate all the effort you put into a solution.

I wanted to raise some discussions before we dive into the specific code changes themselves. I took a look through the changes and your descriptions (thank you, that helped immensely), and I agree with the overall approach. Thank you also for performance testing this and fixing the issues before submitting for review!

One downside of the existing implementation (that was there before your changes) is the fact that we consume messages from all topic partitions on a single stream. While sufficient for the existing functionality, this makes it more cumbersome and difficult to manage and orchestrate state on a per topic-partition basis. For instance, in the current solution, we consume messages from a single stream but then have to fan out the BatchStatusReceivers to n acknowledgement tasks.

I'm curious if you considered a designed where we have one stream per topic partition, which can be achieved with the split_partition_queue method. This design would require more work to manage the individual topic partitions. However, it would also mean that we could have a single tokio task per topic partition that is responsible for both consuming messages from the stream and committing offsets. In conjunction with the topic partition tasks, we would still have a main consumer task that would be responsible for handling rebalance callbacks - spawning tasks for new topic partitions and sending shutdown signals for removed topic partitions / waiting for offsets to be committed. My hunch is that this design would allow us to co-locate and simplify the logic, removing some of the bookkeeping and message passing as well. If true, this would make the code easier to reason about and maintain in the future.

Other than that, I suggest that we split the kafka source up into smaller modules now that it has grown to over 2,000 lines of code. Additionally, can we declare all structs, enums, and implementations at the top-level (e.g., those declared in the handle_acks function)? All of this will help us review the code and make it more maintainable in the future.

A possible modularization of this source in its current state could look like the following:

• src/sources/kafka/mod.rs
  • re-export anything relevant from submodules
• src/sources/kafka/config.rs
  • KafkaSourceConfig and related logic
• src/sources/kafka/source.rs:
  • kafka_source and related logic
• src/sources/kafka/consumer.rs
  • create_consumer, the consumer context, etc
• src/sources/kafka/integration_tests.rs
  • All integration tests for kafka

Let's focus on the design question first before we move forward with restructuring the code.

[jches]

Thanks for taking the time to look this over @dsmith3197! Yeah, I think your hunch might be right; I will look into the stream per topic-partition design and come back soon with more thoughts and/or an implementation, and everything moved to the top level :) I will leave the restructuring into separate files for a separate PR, just to keep the diff clear here but big +1 to that from me.

[jches]

OK, finally had time to get back to this! I was unsure at first whether split_partition_queue would work for this, since the docs note that there may still be messages received on the main consumer, but if we make sure to split off the partition queues as soon as a partition is assigned, that's not a problem.

I'm much happier with this version, there are waaay fewer "gotchas" / edge cases in here, and the bookkeeping is a lot simpler thanks to messages and acknowledgements for a partition being handled on the same task, none of the ForwardedAck stuff from the initial draft is needed anymore.

[dsmith3197]

OK, finally had time to get back to this! I was unsure at first whether split_partition_queue would work for this, since the docs note that there may still be messages received on the main consumer, but if we make sure to split off the partition queues as soon as a partition is assigned, that's not a problem.

I'm much happier with this version, there are waaay fewer "gotchas" / edge cases in here, and the bookkeeping is a lot simpler thanks to messages and acknowledgements for a partition being handled on the same task, none of the ForwardedAck stuff from the initial draft is needed anymore.

@jches Great to hear! I'm excited to take a look through this. I should be able to give this a review by next week.

[bruceg]

This looks very good. I have some code comments below, mostly minor but I see a couple of issues that I would like at least clarification on.

[bruceg]

This is very nice!

[jches]

Thanks for the review and feedback @bruceg! I've merged a few of your suggestions and responded to most of the questions you had. I'll address the remaining suggestions and get those changes committed in the next day or so.

[jches]

@bruceg just an fyi, this is ready for another look when you have time (I never know what github's behavior is going to be on the re-request review button, since I've seen it dismiss other reviews in the past)

[jszwedko]

Thanks for this contribution @jches !

[github-actions]

Regression Detector Results

Run ID: 610aa4e5-7c52-4c3a-a660-82deb2d5ef25
Baseline: 9a621e3
Comparison: 92ee503
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean %	Δ mean % CI	confidence
otlp_http_to_blackhole	ingress throughput	+2.38	[+2.17, +2.59]	100.00%
syslog_loki	ingress throughput	+1.71	[+1.64, +1.77]	100.00%
http_to_http_acks	ingress throughput	+1.33	[+1.14, +1.52]	100.00%
datadog_agent_remap_datadog_logs_acks	ingress throughput	+1.19	[+1.00, +1.37]	100.00%
enterprise_http_to_http	ingress throughput	+0.93	[+0.76, +1.10]	100.00%
syslog_humio_logs	ingress throughput	+0.86	[+0.75, +0.96]	100.00%
fluent_elasticsearch	ingress throughput	+0.83	[+0.53, +1.13]	100.00%
http_text_to_http_json	ingress throughput	+0.45	[+0.29, +0.62]	100.00%
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	+0.13	[-0.01, +0.28]	86.55%
splunk_hec_indexer_ack_blackhole	ingress throughput	+0.01	[-0.16, +0.18]	9.12%
otlp_grpc_to_blackhole	ingress throughput	+0.01	[-0.12, +0.13]	5.35%
http_to_http_json	ingress throughput	-0.00	[-0.05, +0.05]	5.73%
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.09	[-0.25, +0.08]	60.00%
http_to_s3	ingress throughput	-0.25	[-0.54, +0.04]	84.08%
http_to_http_noack	ingress throughput	-0.34	[-0.49, -0.18]	99.96%
syslog_log2metric_humio_metrics	ingress throughput	-0.44	[-0.56, -0.31]	100.00%
datadog_agent_remap_blackhole	ingress throughput	-0.44	[-0.53, -0.34]	100.00%
splunk_hec_route_s3	ingress throughput	-0.52	[-1.05, +0.02]	88.62%
syslog_log2metric_splunk_hec_metrics	ingress throughput	-0.67	[-0.82, -0.52]	100.00%
datadog_agent_remap_blackhole_acks	ingress throughput	-0.98	[-1.10, -0.86]	100.00%
datadog_agent_remap_datadog_logs	ingress throughput	-0.98	[-1.10, -0.85]	100.00%
socket_to_socket_blackhole	ingress throughput	-1.09	[-1.15, -1.03]	100.00%
syslog_regex_logs2metric_ddmetrics	ingress throughput	-2.55	[-2.73, -2.37]	100.00%
syslog_splunk_hec_logs	ingress throughput	-3.03	[-3.20, -2.85]	100.00%
file_to_blackhole	egress throughput	-3.28	[-4.38, -2.17]	100.00%

[github-actions]

Regression Detector Results

Run ID: 5c4b7396-ec40-4565-b616-96bdf23ff560
Baseline: 67c4beb
Comparison: b8d9a36
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean %	Δ mean % CI	confidence
file_to_blackhole	egress throughput	+4.08	[+2.90, +5.26]	100.00%
splunk_hec_route_s3	ingress throughput	+2.77	[+2.23, +3.30]	100.00%
datadog_agent_remap_datadog_logs_acks	ingress throughput	+2.39	[+2.19, +2.59]	100.00%
socket_to_socket_blackhole	ingress throughput	+1.65	[+1.56, +1.75]	100.00%
http_to_http_noack	ingress throughput	+1.18	[+1.01, +1.34]	100.00%
http_to_http_acks	ingress throughput	+0.98	[+0.78, +1.18]	100.00%
datadog_agent_remap_blackhole_acks	ingress throughput	+0.94	[+0.84, +1.05]	100.00%
datadog_agent_remap_blackhole	ingress throughput	+0.89	[+0.79, +0.99]	100.00%
http_text_to_http_json	ingress throughput	+0.78	[+0.68, +0.89]	100.00%
datadog_agent_remap_datadog_logs	ingress throughput	+0.66	[+0.51, +0.80]	100.00%
fluent_elasticsearch	ingress throughput	+0.38	[+0.08, +0.69]	96.31%
syslog_splunk_hec_logs	ingress throughput	+0.33	[+0.25, +0.40]	100.00%
syslog_loki	ingress throughput	+0.08	[+0.01, +0.14]	94.16%
http_to_s3	ingress throughput	+0.07	[-0.21, +0.36]	31.56%
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	+0.01	[-0.15, +0.18]	11.97%
splunk_hec_indexer_ack_blackhole	ingress throughput	+0.00	[-0.16, +0.17]	3.57%
http_to_http_json	ingress throughput	-0.18	[-0.24, -0.11]	100.00%
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.28	[-0.43, -0.13]	99.82%
otlp_grpc_to_blackhole	ingress throughput	-1.03	[-1.15, -0.91]	100.00%
otlp_http_to_blackhole	ingress throughput	-1.07	[-1.23, -0.91]	100.00%
enterprise_http_to_http	ingress throughput	-1.09	[-1.26, -0.93]	100.00%
syslog_log2metric_humio_metrics	ingress throughput	-1.46	[-1.59, -1.32]	100.00%
syslog_log2metric_splunk_hec_metrics	ingress throughput	-1.95	[-2.10, -1.80]	100.00%
syslog_humio_logs	ingress throughput	-2.42	[-2.54, -2.30]	100.00%
syslog_regex_logs2metric_ddmetrics	ingress throughput	-3.75	[-4.02, -3.49]	100.00%

[jches]

Thanks everyone for the reviews and feedback 🎉

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to no response for status checks Oct 11, 2023

Is this normal? I didn't see any failed checks..do I need to pull changes from master into my branch to get a clean merge?

[github-actions]

Regression Detector Results

Run ID: b9ac5cca-20ce-431e-907c-8dfba3b3093c
Baseline: 67c4beb
Comparison: 4cbae11
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean %	Δ mean % CI	confidence
http_to_http_acks	ingress throughput	+1.83	[+1.63, +2.02]	100.00%
datadog_agent_remap_blackhole	ingress throughput	+1.49	[+1.37, +1.60]	100.00%
datadog_agent_remap_datadog_logs	ingress throughput	+0.75	[+0.60, +0.91]	100.00%
syslog_regex_logs2metric_ddmetrics	ingress throughput	+0.56	[+0.41, +0.72]	100.00%
syslog_loki	ingress throughput	+0.46	[+0.40, +0.53]	100.00%
datadog_agent_remap_datadog_logs_acks	ingress throughput	+0.35	[+0.18, +0.52]	99.93%
datadog_agent_remap_blackhole_acks	ingress throughput	+0.31	[+0.20, +0.41]	100.00%
file_to_blackhole	egress throughput	+0.28	[-0.97, +1.53]	28.99%
splunk_hec_indexer_ack_blackhole	ingress throughput	+0.01	[-0.16, +0.17]	5.27%
http_to_http_json	ingress throughput	+0.00	[-0.05, +0.05]	5.93%
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.00	[-0.14, +0.14]	1.32%
socket_to_socket_blackhole	ingress throughput	-0.08	[-0.16, +0.01]	86.39%
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.10	[-0.26, +0.07]	66.23%
fluent_elasticsearch	ingress throughput	-0.20	[-0.50, +0.10]	72.83%
http_to_s3	ingress throughput	-0.34	[-0.63, -0.05]	94.94%
enterprise_http_to_http	ingress throughput	-0.46	[-0.66, -0.26]	99.98%
syslog_log2metric_humio_metrics	ingress throughput	-0.51	[-0.61, -0.40]	100.00%
otlp_grpc_to_blackhole	ingress throughput	-0.59	[-0.71, -0.47]	100.00%
syslog_humio_logs	ingress throughput	-1.23	[-1.34, -1.12]	100.00%
syslog_splunk_hec_logs	ingress throughput	-1.34	[-1.42, -1.26]	100.00%
http_to_http_noack	ingress throughput	-1.49	[-1.66, -1.31]	100.00%
otlp_http_to_blackhole	ingress throughput	-1.60	[-1.77, -1.44]	100.00%
http_text_to_http_json	ingress throughput	-1.95	[-2.07, -1.82]	100.00%
syslog_log2metric_splunk_hec_metrics	ingress throughput	-2.26	[-2.42, -2.11]	100.00%
splunk_hec_route_s3	ingress throughput	-2.45	[-2.96, -1.94]	100.00%

[dsmith3197]

Is this normal? I didn't see any failed checks..do I need to pull changes from master into my branch to get a clean merge?

No, that shouldn't of happened. I added it to the merge queue again.

[jches]

Removed from the queue again - is it just a "retry until success" kind of thing? 🤪
This branch is quite a few commits behind master, but shouldn't have any conflicts.

[jszwedko]

Apologies @jches ! You are hitting an issue with the Mac CI runners that should be fixed by #18823. Once that goes in, this PR should be able to make it in.

[jches]

ah I see, thanks for the update 👍

[github-actions]

Regression Detector Results

Run ID: 5870dc0e-ed7d-4fd6-9a79-114cde2e292a
Baseline: 331c5a0
Comparison: f2efb1a
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean %	Δ mean % CI	confidence
socket_to_socket_blackhole	ingress throughput	+3.66	[+3.55, +3.76]	100.00%
http_to_http_noack	ingress throughput	+1.51	[+1.32, +1.69]	100.00%
file_to_blackhole	egress throughput	+1.36	[+0.20, +2.51]	94.72%
otlp_http_to_blackhole	ingress throughput	+0.98	[+0.81, +1.14]	100.00%
syslog_log2metric_humio_metrics	ingress throughput	+0.71	[+0.57, +0.84]	100.00%
syslog_loki	ingress throughput	+0.70	[+0.62, +0.78]	100.00%
http_to_s3	ingress throughput	+0.48	[+0.19, +0.77]	99.41%
datadog_agent_remap_blackhole	ingress throughput	+0.43	[+0.33, +0.54]	100.00%
syslog_splunk_hec_logs	ingress throughput	+0.39	[+0.32, +0.46]	100.00%
otlp_grpc_to_blackhole	ingress throughput	+0.12	[-0.01, +0.24]	87.64%
enterprise_http_to_http	ingress throughput	+0.11	[-0.07, +0.28]	66.86%
datadog_agent_remap_datadog_logs	ingress throughput	+0.07	[-0.11, +0.25]	48.10%
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	+0.00	[-0.16, +0.17]	3.30%
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	+0.00	[-0.14, +0.14]	2.46%
splunk_hec_indexer_ack_blackhole	ingress throughput	+0.00	[-0.16, +0.16]	1.25%
http_to_http_json	ingress throughput	-0.00	[-0.05, +0.05]	2.43%
datadog_agent_remap_blackhole_acks	ingress throughput	-0.26	[-0.37, -0.15]	99.98%
syslog_humio_logs	ingress throughput	-0.37	[-0.49, -0.26]	100.00%
splunk_hec_route_s3	ingress throughput	-0.46	[-1.00, +0.09]	83.01%
datadog_agent_remap_datadog_logs_acks	ingress throughput	-0.82	[-0.97, -0.66]	100.00%
syslog_log2metric_splunk_hec_metrics	ingress throughput	-0.82	[-0.97, -0.67]	100.00%
syslog_regex_logs2metric_ddmetrics	ingress throughput	-1.02	[-1.18, -0.86]	100.00%
fluent_elasticsearch	ingress throughput	-1.34	[-1.64, -1.04]	100.00%
http_text_to_http_json	ingress throughput	-1.79	[-1.91, -1.68]	100.00%
http_to_http_acks	ingress throughput	-3.14	[-3.30, -2.99]	100.00%