2.32.0 release #733

matthewfala · 2023-09-26T04:00:08Z

2.32.0 Release

This release includes:

Fluent Bit 1.9.10
Amazon CloudWatch Logs for Fluent Bit 1.9.4
Amazon Kinesis Streams for Fluent Bit 1.10.2
Amazon Kinesis Firehose for Fluent Bit 1.7.2

Compared to 2.31.12.20230911 this release adds:

Feature - Amazon Managed Prometheus support added via sigv4 authentication on prometheus_remote_write aws-for-fluent-bit:256
Enhancement - Allow multiple cloudwatch_log output workers by removing sequence tokens from API requests aws-for-fluent-bit:526
Bug - Fix multiline input behavior when multiple streams are parsed (stderr, stdout) together. Multiline logs are no longer terminated when streams are switched between fluent-bit:7469

matthewfala · 2023-09-26T04:06:07Z

Please see the following cherry-pick diffs:
Cloudwatch Sequence Token Deprecation: matthewfala/fluent-bit#58
Prometheus: matthewfala/fluent-bit#59
Multiline flush fix: matthewfala/fluent-bit#60

matthewfala · 2023-09-26T22:19:35Z

Also updated the cherry-picking system in the docker build file to end build on a single failure.

matthewfala · 2023-09-28T01:02:15Z

Updated the master and CWL cherry-pick: https://github.com/fluent/fluent-bit/compare/6c9656d3d7e965faf7ee0fdbe657f2775bc58c24..737cb822fca7274d76c2e15c2ef6ff6faae872d0

matthewfala · 2023-09-28T01:43:01Z

Updated

PettitWesley · 2023-09-28T15:46:44Z

AWS_FLB_CHERRY_PICKS

-# TCP input: user friendly message for skipping logs https://github.com/fluent/fluent-bit/pull/6061
-https://github.com/fluent/fluent-bit.git 1.9 c5d57ccf4f85af77e2a2bbb1121df95887e2962f
-
-# ElasticSearch: fix buffer bulk over run https://github.com/fluent/fluent-bit/pull/5770
-https://github.com/fluent/fluent-bit.git 1.9 d3a0697c3c4a876250cc6c2c4ff44cd77908ea3c
-
-# windows: fix time zone offset https://github.com/fluent/fluent-bit/pull/6368
-https://github.com/fluent/fluent-bit.git 1.9 cae9a8f9cb6eab7449f14d914f9ba1d8d4e93cf0
-


@matthewfala why are these deleted?

All of these PRs were merged int 1.9 series by v1.9.10. Please see:
fluent/fluent-bit#6368
fluent/fluent-bit#5770
fluent/fluent-bit#6061

You can see all of them in the commit history here: https://github.com/fluent/fluent-bit/commits/v1.9.10?before=760956f50cdc7eef3047df20d2299202a8c68594+35&branch=v1.9.10&qualified_name=refs%2Ftags%2Fv1.9.10

matthewfala · 2023-09-28T18:34:18Z

Update wording to
Compared to 2.31.12.20230911 this release adds:

- Feature - Customers can send metrics to Amazon Managed Prometheus via added sigv4 authentication on prometheus_remote_write https://github.com/aws/aws-for-fluent-bit/issues/256
- Enhancement - Customers can route logs to CloudWatch Logs at higher throughputs by increasing number of output workers as cloudwatch output plugin removed sequence tokens from API requests https://github.com/aws/aws-for-fluent-bit/issues/526
- Fix - Fix multiline input behavior when multiple streams are parsed (stderr, stdout) together. Multiline logs are no longer terminated when streams are switched between https://github.com/fluent/fluent-bit/pull/7469

Update public doc - Request for doc change (not-blocking)
Update public doc - AWS FireLens Example (9/28)
- Add link to release notes
Update public docs once

matthewfala · 2023-09-29T01:22:20Z

Update the examples: aws-samples/amazon-ecs-firelens-examples#125
Update cloudwatch_logs guidance issue: #340

matthewfala · 2023-09-29T03:26:48Z

Updated cloudwatch_logs cherrypick https://github.com/matthewfala/fluent-bit/compare/4c92b10638b02cd6203219464a24dc7ad54c1acb..8ee560e388bbbf850069c81bbca06275f330baeb

https://github.com/aws/aws-for-fluent-bit/compare/133a592172c350e9711cf6ab1232b2de14b8c2a8..44ac0a9aa454aaa16f0e3fe796366551b1a3a67e

PettitWesley · 2023-09-29T04:47:19Z

CHANGELOG.md

+
+Compared to `2.31.12.20230911` this release adds:
+* Feature - Amazon Managed Prometheus support added via sigv4 authentication on `prometheus_remote_write` [aws-for-fluent-bit:256](https://github.com/aws/aws-for-fluent-bit/issues/256)
+* Enhancement - Allow multiple `cloudwatch_log` output workers by removing sequence tokens from API requests [aws-for-fluent-bit:526](https://github.com/aws/aws-for-fluent-bit/issues/526)


Let's link to the new firelens example somewhere in the release notes.

the final link is predictable

Sure! Link added.

PettitWesley · 2023-09-29T04:49:13Z

Makefile

@@ -28,7 +28,7 @@ release: build build-init
 debug: main-debug init-debug

 .PHONY: build
-build: linux-plugins


build should still build linux-plugins?

Build dockerfile/image does not depend on the linux-plugins image. Actually only the release and main-release depends on the "linux plugins"

It was slowing down my fluent-bit build tests.

matthewfala · 2023-09-29T19:44:09Z

CHANGELOG.md

+* Amazon Kinesis Streams for Fluent Bit 1.10.2
+* Amazon Kinesis Firehose for Fluent Bit 1.7.2
+
+Compared to `2.31.12.20230911` this release adds:


These notes are not updated. Resolve before merge

Also the release note should be Compared to 2.31.12, the date stamped re-builds for linux only don't count as a full release.

Sure! Updated.

matthewfala · 2023-09-29T20:43:51Z

CHANGELOG.md

+* Amazon Kinesis Firehose for Fluent Bit 1.7.2
+
+Compared to `2.31.12` this release adds:
+* Feature - Customers can send metrics to Amazon Managed Prometheus via added sigv4 authentication on `prometheus_remote_write`. Refer to [amazon-ecs-firelens-examples](https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/amazon-managed-service-for-prometheus/README.md) for information on how to export metrics to AMP on ECS [aws-for-fluent-bit:256](https://github.com/aws/aws-for-fluent-bit/issues/256)


Added link to prometheus example

matthewfala · 2023-09-29T20:44:17Z

CHANGELOG.md

+* Amazon Kinesis Streams for Fluent Bit 1.10.2
+* Amazon Kinesis Firehose for Fluent Bit 1.7.2
+
+Compared to `2.31.12` this release adds:


Updated version comparison to 2.31.12

matthewfala · 2023-10-02T22:49:30Z

Consider adding the following patches:

Coroutine drop fix: upstream: premature connection destruction fix fluent/fluent-bit#7728
Throttle fix: matthewfala/fluent-bit@7b05b7e

matthewfala · 2023-10-04T02:55:36Z

Throttle fix is a backport of: https://github.com/fluent/fluent-bit/pull/6500/files

matthewfala · 2023-10-04T03:34:08Z

Created 1.9 backport of the network fix: https://github.com/matthewfala/fluent-bit/tree/2.32.0-premature-connection-destruction

PettitWesley · 2023-10-16T20:25:06Z

CHANGELOG.md

+* Fix - Fix multiline input behavior when multiple streams are parsed (stderr, stdout) together. Multiline logs are no longer terminated when streams are switched between [fluent-bit:7469](https://github.com/fluent/fluent-bit/pull/7469)
+* Fix - Fix networking edgecase causing data loss and OOM related issues via net timeout event injection resolution [fluent-bit:7728](https://github.com/fluent/fluent-bit/pull/7728/files)
+* Fix - Allow reduced filter_throttle log output with `print_status` `false` configuration option fix [fluent-bit:7469](https://github.com/fluent/fluent-bit/pull/6500/files)
+


@matthewfala we should add on3e more things from here, when I tried to stage 2.32: https://github.com/aws/aws-for-fluent-bit/pull/658/files

AWS Profile support

Documenting our decision:

Will further review this feature item in the future in the 2.33.0 release. Requires further review and testing. link

Proceed with release

Signed-off-by: Matthew Fala <[email protected]>

matthewfala · 2023-11-02T19:31:13Z

For documentation: Test the network fix 2.32.0-premature-connection-destruction change via:

diff --git a/src/flb_network.c b/src/flb_network.c
index f8cc7068d..d63d7ab94 100644
--- a/src/flb_network.c
+++ b/src/flb_network.c
@@ -441,7 +441,7 @@ static int net_connect_async(int fd,
     ret = mk_event_add(u_conn->evl,
                        fd,
                        FLB_ENGINE_EV_THREAD,
-                       MK_EVENT_WRITE, &u_conn->event);
+                       MK_EVENT_READ, &u_conn->event);
     u_conn->event.priority = FLB_ENGINE_PRIORITY_CONNECT;
     if (ret == -1) {
         /*

This will cause the network events to artificially timeout, causing the injection timeout code which is modified to be triggered.

matthewfala · 2023-11-02T20:49:55Z

Add update to the error log to include api caller: https://github.com/fluent/fluent-bit/pull/8122/files

matthewfala requested a review from a team as a code owner September 26, 2023 04:00

matthewfala mentioned this pull request Sep 26, 2023

Support New CloudWatch Logs Sequence Token Deprecation and Ingestion Limit Removal #526

Closed

matthewfala force-pushed the 2.32.0-release branch 2 times, most recently from b369bd9 to e82ef87 Compare September 26, 2023 22:18

matthewfala force-pushed the 2.32.0-release branch from e82ef87 to 6c041fb Compare September 28, 2023 01:10

PettitWesley reviewed Sep 28, 2023

View reviewed changes

PettitWesley approved these changes Sep 28, 2023

View reviewed changes

matthewfala mentioned this pull request Sep 29, 2023

2.32.0 prometheus cw aws-samples/amazon-ecs-firelens-examples#125

Merged

matthewfala force-pushed the 2.32.0-release branch 2 times, most recently from 133a592 to 44ac0a9 Compare September 29, 2023 03:25

PettitWesley reviewed Sep 29, 2023

View reviewed changes

matthewfala commented Sep 29, 2023

View reviewed changes

matthewfala force-pushed the 2.32.0-release branch 2 times, most recently from 8911319 to 1d379bd Compare September 29, 2023 20:43

matthewfala commented Sep 29, 2023

View reviewed changes

matthewfala requested a review from PettitWesley September 29, 2023 20:44

matthewfala force-pushed the 2.32.0-release branch 2 times, most recently from 492bf18 to 250a64e Compare October 5, 2023 23:34

matthewfala force-pushed the 2.32.0-release branch from 250a64e to 6732a5d Compare October 5, 2023 23:53

PettitWesley reviewed Oct 16, 2023

View reviewed changes

matthewfala added 6 commits October 19, 2023 21:10

enhancement: cloudwatch_logs sequence token deprecation

b3473b9

Signed-off-by: Matthew Fala <[email protected]>

feature: prometheus sigv4 authorization

f6ed3e7

Signed-off-by: Matthew Fala <[email protected]>

multiline: add flush fix

0120dcd

Signed-off-by: Matthew Fala <[email protected]>

Bump version to 2.32.0

0a5047c

build: fix patching cherry picking issues

aba4bff

Signed-off-by: Matthew Fala <[email protected]>

cherry-picks: add network event drop fix and throttle print_status fix

1f68d57

matthewfala force-pushed the 2.32.0-release branch from 6732a5d to 1f68d57 Compare October 19, 2023 21:14

matthewfala merged commit 00496b6 into aws:mainline Oct 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2.32.0 release #733

2.32.0 release #733

matthewfala commented Sep 26, 2023 •

edited

Loading

matthewfala commented Sep 26, 2023

matthewfala commented Sep 26, 2023

matthewfala commented Sep 28, 2023

matthewfala commented Sep 28, 2023

PettitWesley Sep 28, 2023

matthewfala Sep 28, 2023

PettitWesley Sep 28, 2023

matthewfala commented Sep 28, 2023 •

edited

Loading

matthewfala commented Sep 29, 2023 •

edited

Loading

matthewfala commented Sep 29, 2023 •

edited

Loading

PettitWesley Sep 29, 2023

PettitWesley Sep 29, 2023

matthewfala Sep 29, 2023

PettitWesley Sep 29, 2023

matthewfala Sep 29, 2023

matthewfala Sep 29, 2023

PettitWesley Sep 29, 2023

matthewfala Sep 29, 2023

matthewfala Sep 29, 2023

matthewfala Sep 29, 2023

matthewfala commented Oct 2, 2023

matthewfala commented Oct 4, 2023

matthewfala commented Oct 4, 2023

PettitWesley Oct 16, 2023 •

edited

Loading

matthewfala Oct 19, 2023

matthewfala commented Nov 2, 2023

matthewfala commented Nov 2, 2023

2.32.0 release #733

2.32.0 release #733

Conversation

matthewfala commented Sep 26, 2023 • edited Loading

2.32.0 Release

matthewfala commented Sep 26, 2023

matthewfala commented Sep 26, 2023

matthewfala commented Sep 28, 2023

matthewfala commented Sep 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewfala commented Sep 28, 2023 • edited Loading

matthewfala commented Sep 29, 2023 • edited Loading

matthewfala commented Sep 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewfala commented Oct 2, 2023

matthewfala commented Oct 4, 2023

matthewfala commented Oct 4, 2023

PettitWesley Oct 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewfala commented Nov 2, 2023

matthewfala commented Nov 2, 2023

matthewfala commented Sep 26, 2023 •

edited

Loading

matthewfala commented Sep 28, 2023 •

edited

Loading

matthewfala commented Sep 29, 2023 •

edited

Loading

matthewfala commented Sep 29, 2023 •

edited

Loading

PettitWesley Oct 16, 2023 •

edited

Loading