out_s3: add store_dir_limit_size to limit S3 disk usage

[PettitWesley]

Signed-off-by: Wesley Pettit [email protected]

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Attached local packaging test output showing all targets (including any new ones) build.

Documentation

• Documentation required for this feature
  https://github.com/fluent/fluent-bit-docs/pull/956/files

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[PettitWesley]

With it we get:

[2022/08/16 21:37:27] [ warn] [output:s3:s3.0] Buffer is almost full: current_buffer_size=19483080, store_dir_limit_size=20000000 bytes
[2022/08/16 21:37:33] [error] [output:s3:s3.0] Buffer is full: current_buffer_size=19997432, new_data=41193, store_dir_limit_size=20000000 bytes

[matthewfala]

Looks good!

[PettitWesley]

Doc PR: https://github.com/fluent/fluent-bit-docs/pull/956/files

[PettitWesley]

@edsiper Ready for merge/review.

[PettitWesley]

fluent/fluent-bit-docs#971