Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement(prometheus_exporter sink): use common test helpers in tests #14080

Conversation

prognant
Copy link
Contributor

part of #14077
part of #13995

@netlify
Copy link

netlify bot commented Aug 23, 2022

Deploy Preview for vector-project canceled.

Name Link
🔨 Latest commit 38ccbf4
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/63060cf98e3bb4000819f417

@prognant prognant added the ci-condition: integration tests enable Run integration tests on this PR label Aug 23, 2022
@github-actions github-actions bot added the domain: sinks Anything related to the Vector's sinks label Aug 23, 2022
@prognant prognant marked this pull request as ready for review August 23, 2022 16:12
@github-actions
Copy link

Soak Test Results

Baseline: fc44685
Comparison: ce456de
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_log2metric_humio_metrics 116.75KiB 0.92 100.00% 12.39MiB 400.14KiB 8.17KiB 0 0.0315422 12.5MiB 558.99KiB 11.39KiB 0 0.0436621 False False
http_pipelines_blackhole_acks 5.28KiB 0.42 93.67% 1.22MiB 114.44KiB 2.33KiB 0 0.091269 1.23MiB 79.81KiB 1.63KiB 0 0.0633795 False False
datadog_agent_remap_blackhole_acks 210.69KiB 0.34 96.26% 59.95MiB 4.18MiB 86.97KiB 0 0.0696503 60.16MiB 2.47MiB 51.72KiB 0 0.0410576 False False
http_text_to_http_json 111.73KiB 0.27 100.00% 39.98MiB 725.86KiB 14.82KiB 0 0.017726 40.09MiB 723.92KiB 14.78KiB 0 0.0176304 False False
http_pipelines_blackhole 1.61KiB 0.1 53.13% 1.65MiB 41.33KiB 864.7B 0 0.0244707 1.65MiB 100.99KiB 2.06KiB 0 0.0597418 False False
splunk_hec_to_splunk_hec_logs_noack 18.26KiB 0.07 88.07% 23.82MiB 465.99KiB 9.52KiB 0 0.0190992 23.84MiB 335.05KiB 6.84KiB 0 0.013722 False False
datadog_agent_remap_blackhole 37.08KiB 0.06 20.00% 59.82MiB 5.66MiB 117.76KiB 0 0.0945381 59.85MiB 4.16MiB 86.96KiB 0 0.0695724 False False
enterprise_http_to_http -2.47KiB -0.01 26.08% 23.85MiB 258.59KiB 5.28KiB 0 0.0105872 23.84MiB 255.37KiB 5.22KiB 0 0.0104565 False False
splunk_hec_indexer_ack_blackhole -11.87KiB -0.05 34.65% 23.75MiB 891.12KiB 18.13KiB 0 0.03663 23.74MiB 946.68KiB 19.25KiB 0 0.038933 False False
splunk_hec_to_splunk_hec_logs_acks -15.29KiB -0.06 46.90% 23.76MiB 821.49KiB 16.72KiB 0 0.0337563 23.75MiB 874.53KiB 17.78KiB 0 0.0359584 False False
file_to_blackhole -109.09KiB -0.11 63.16% 95.32MiB 3.77MiB 78.19KiB 0 0.0395909 95.22MiB 4.46MiB 92.69KiB 0 0.0468332 False False
http_to_http_json -47.18KiB -0.19 99.95% 23.85MiB 330.47KiB 6.75KiB 0 0.0135296 23.8MiB 575.07KiB 11.73KiB 0 0.0235892 False False
fluent_elasticsearch -170.57KiB -0.21 100.00% 79.47MiB 52.86KiB 1.07KiB 0 0.000649445 79.31MiB 1.48MiB 30.5KiB 0 0.0186866 False False
splunk_hec_route_s3 -42.36KiB -0.22 47.85% 19.06MiB 2.3MiB 47.96KiB 0 0.120796 19.02MiB 2.17MiB 45.46KiB 0 0.114315 False False
http_to_http_noack -78.15KiB -0.32 99.94% 23.84MiB 408.39KiB 8.35KiB 0 0.0167275 23.76MiB 1.02MiB 21.22KiB 0 0.0428032 False False
syslog_regex_logs2metric_ddmetrics -69.55KiB -0.56 100.00% 12.19MiB 515.86KiB 10.51KiB 0 0.0413301 12.12MiB 522.85KiB 10.66KiB 0 0.0421252 False False
datadog_agent_remap_datadog_logs -439.59KiB -0.67 100.00% 63.68MiB 365.32KiB 7.47KiB 0 0.00560082 63.25MiB 4.27MiB 88.98KiB 0 0.0675403 False False
syslog_splunk_hec_logs -115.55KiB -0.71 100.00% 15.98MiB 961.95KiB 19.57KiB 0 0.0587758 15.87MiB 871.25KiB 17.76KiB 0 0.0536123 False False
syslog_loki -126.64KiB -0.83 100.00% 14.89MiB 282.92KiB 5.79KiB 0 0.0185514 14.77MiB 747.55KiB 15.2KiB 0 0.0494284 False False
http_to_http_acks -189.96KiB -1.05 60.19% 17.74MiB 7.76MiB 162.19KiB 0 0.437408 17.55MiB 7.45MiB 155.61KiB 0 0.424539 True True
datadog_agent_remap_datadog_logs_acks -672.42KiB -1.05 100.00% 62.71MiB 3.11MiB 65.06KiB 0 0.0495994 62.06MiB 4.47MiB 92.99KiB 0 0.0719745 False False
socket_to_socket_blackhole -295.43KiB -1.22 100.00% 23.58MiB 191.2KiB 3.9KiB 0 0.00791794 23.29MiB 138.39KiB 2.83KiB 0 0.00580202 False False
syslog_log2metric_splunk_hec_metrics -224.62KiB -1.24 100.00% 17.75MiB 351.08KiB 7.16KiB 0 0.0193167 17.53MiB 675.79KiB 13.77KiB 0 0.0376478 False False
http_pipelines_no_grok_blackhole -143.71KiB -1.28 100.00% 10.96MiB 45.43KiB 949.69B 0 0.00404807 10.82MiB 1.05MiB 21.78KiB 0 0.0966993 False False
syslog_humio_logs -223.02KiB -1.37 100.00% 15.85MiB 303.42KiB 6.19KiB 0 0.0186881 15.63MiB 361.46KiB 7.4KiB 0 0.0225731 False False

@prognant prognant requested a review from neuronull August 24, 2022 09:43
@prognant prognant changed the title enhancement(prometheus_exporter sink): use common test helpers in integration tests enhancement(prometheus_exporter sink): use common test helpers in tests Aug 24, 2022
@github-actions
Copy link

Soak Test Results

Baseline: 92dfb84
Comparison: 38ccbf4
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole_acks 2.06MiB 3.53 100.00% 58.47MiB 4.01MiB 83.59KiB 0 0.0686403 60.53MiB 4.06MiB 84.95KiB 0 0.0670396 False False
splunk_hec_route_s3 286.24KiB 1.55 100.00% 18.07MiB 2.39MiB 49.77KiB 0 0.132235 18.35MiB 2.31MiB 48.24KiB 0 0.125686 False False
datadog_agent_remap_datadog_logs_acks 788.63KiB 1.25 100.00% 61.51MiB 2.87MiB 60.03KiB 0 0.0466449 62.28MiB 4.42MiB 91.96KiB 0 0.0709163 False False
http_text_to_http_json 448.37KiB 1.1 100.00% 39.75MiB 748.04KiB 15.27KiB 0 0.0183714 40.19MiB 775.51KiB 15.83KiB 0 0.0188387 False False
datadog_agent_remap_blackhole 617.95KiB 0.99 100.00% 60.69MiB 4.12MiB 85.95KiB 0 0.0679416 61.29MiB 2.74MiB 57.1KiB 0 0.0446259 False False
http_pipelines_blackhole_acks 4.34KiB 0.34 90.07% 1.23MiB 108.23KiB 2.2KiB 0 0.085786 1.24MiB 70.75KiB 1.44KiB 0 0.0558845 False False
splunk_hec_to_splunk_hec_logs_noack 16.12KiB 0.07 82.69% 23.82MiB 477.78KiB 9.75KiB 0 0.0195823 23.84MiB 327.96KiB 6.7KiB 0 0.0134328 False False
splunk_hec_indexer_ack_blackhole 10.05KiB 0.04 29.71% 23.74MiB 927.99KiB 18.87KiB 0 0.0381619 23.75MiB 903.21KiB 18.38KiB 0 0.0371278 False False
enterprise_http_to_http -982.11B -0 10.72% 23.85MiB 245.02KiB 5.0KiB 0 0.010032 23.85MiB 247.23KiB 5.06KiB 0 0.0101226 False False
splunk_hec_to_splunk_hec_logs_acks -3.96KiB -0.02 12.54% 23.75MiB 863.76KiB 17.57KiB 0 0.0355121 23.74MiB 881.09KiB 17.91KiB 0 0.0362306 False False
file_to_blackhole -45.54KiB -0.05 35.20% 95.36MiB 3.48MiB 72.03KiB 0 0.0364385 95.31MiB 3.32MiB 68.98KiB 0 0.0347915 False False
http_to_http_json -26.63KiB -0.11 96.98% 23.84MiB 345.06KiB 7.05KiB 0 0.0141299 23.82MiB 492.18KiB 10.06KiB 0 0.0201765 False False
fluent_elasticsearch -142.58KiB -0.18 100.00% 79.47MiB 53.94KiB 1.09KiB 0 0.000662722 79.33MiB 1.43MiB 29.43KiB 0 0.0180059 False False
http_to_http_noack -67.29KiB -0.28 99.85% 23.84MiB 400.05KiB 8.17KiB 0 0.0163851 23.77MiB 961.29KiB 19.59KiB 0 0.0394814 False False
syslog_regex_logs2metric_ddmetrics -71.34KiB -0.55 100.00% 12.58MiB 517.78KiB 10.56KiB 0 0.0401724 12.51MiB 573.03KiB 11.68KiB 0 0.0447067 False False
datadog_agent_remap_datadog_logs -383.21KiB -0.6 100.00% 62.8MiB 1.19MiB 24.99KiB 0 0.0189862 62.43MiB 4.01MiB 83.46KiB 0 0.0641444 False False
http_pipelines_blackhole -11.3KiB -0.64 100.00% 1.72MiB 10.77KiB 225.36B 0 0.00612891 1.7MiB 95.89KiB 1.96KiB 0 0.054945 False False
syslog_loki -113.18KiB -0.75 100.00% 14.78MiB 329.42KiB 6.74KiB 0 0.0217648 14.67MiB 743.94KiB 15.12KiB 0 0.0495225 False False
http_pipelines_no_grok_blackhole -103.39KiB -0.92 100.00% 11.03MiB 261.1KiB 5.33KiB 0 0.0231086 10.93MiB 1019.51KiB 20.74KiB 0 0.0910665 False False
syslog_log2metric_humio_metrics -155.02KiB -1.16 100.00% 13.01MiB 206.09KiB 4.21KiB 0 0.0154605 12.86MiB 517.81KiB 10.54KiB 0 0.0393028 False False
syslog_splunk_hec_logs -214.84KiB -1.29 100.00% 16.21MiB 725.91KiB 14.79KiB 0 0.0437325 16.0MiB 729.23KiB 14.87KiB 0 0.0445089 False False
syslog_humio_logs -247.07KiB -1.44 100.00% 16.7MiB 153.52KiB 3.14KiB 0 0.00897511 16.46MiB 221.54KiB 4.54KiB 0 0.0131413 False False
http_to_http_acks -283.96KiB -1.57 77.91% 17.65MiB 7.85MiB 164.03KiB 0 0.444522 17.37MiB 7.86MiB 163.99KiB 0 0.452249 True True
syslog_log2metric_splunk_hec_metrics -357.02KiB -1.94 100.00% 17.93MiB 669.1KiB 13.63KiB 0 0.0364358 17.58MiB 754.01KiB 15.35KiB 0 0.0418737 False False
socket_to_socket_blackhole -488.81KiB -2.02 100.00% 23.63MiB 346.85KiB 7.08KiB 0 0.0143308 23.15MiB 181.37KiB 3.7KiB 0 0.00764797 False False

@prognant prognant merged commit eb1aae2 into master Aug 24, 2022
@prognant prognant deleted the prognant/use_common_test_facilities_in_prometheus_exporter_intg_tests branch August 24, 2022 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-condition: integration tests enable Run integration tests on this PR domain: sinks Anything related to the Vector's sinks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants