stackdriver: add cloud_logging_base_url

[braydonk]

Add the Cloud_Logging_Base_Url configuration option to the out_stackdriver plugin. This feature is to support alternate universe_domains in the output plugin. A majority of users will never need to change this.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[INPUT]
    Name dummy

[OUTPUT]
    Name stackdriver
    Match *
    Cloud_Logging_Base_Url https://www.logging-other.com

• Debug log output from testing the change

Please refer to the Valgrind output below. In that output, I used a universe domain that doesn't exist; I don't have an effective way to test an alternate universe domain at the moment. I also tested with https://logging.googleapis.com manually set through the config to still test the config building path that constructs the domain from the base URL in the config and it worked the same.

• Attached Valgrind output that shows no leaks or memory corruption was found

Valgrind debug output with https://logging-other.com

braydonk@debian-dev-machine:~/Git/fluent-bit/build/bin$ valgrind --leak-check=full ./fluent-bit -v -c x.conf                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
==3177720== Memcheck, a memory error detector                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
==3177720== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
==3177720== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
==3177720== Command: ./fluent-bit -v -c x.conf                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
==3177720==                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
Fluent Bit v3.2.3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
* Copyright (C) 2015-2024 The Fluent Bit Authors                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
* https://fluentbit.io                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
______ _                  _    ______ _ _           _____  _____                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
|  ___| |                | |   | ___ (_) |         |____ |/ __  \                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
[2024/12/30 15:20:11] [ info] Configuration:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
[2024/12/30 15:20:11] [ info]  flush time     | 1.000000 seconds                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
[2024/12/30 15:20:11] [ info]  grace          | 5 seconds                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
[2024/12/30 15:20:11] [ info]  daemon         | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
[2024/12/30 15:20:11] [ info] ___________                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
[2024/12/30 15:20:11] [ info]  inputs:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
[2024/12/30 15:20:11] [ info]      dummy                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
[2024/12/30 15:20:11] [ info] ___________                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
[2024/12/30 15:20:11] [ info]  filters:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
[2024/12/30 15:20:11] [ info] ___________                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
[2024/12/30 15:20:11] [ info]  outputs:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
[2024/12/30 15:20:11] [ info]      stackdriver.0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
[2024/12/30 15:20:11] [ info] ___________                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
[2024/12/30 15:20:11] [ info]  collectors:
[2024/12/30 15:20:11] [ info] [fluent bit] version=3.2.3, commit=b714f6d815, pid=3177720
[2024/12/30 15:20:11] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/12/30 15:20:11] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/12/30 15:20:11] [ info] [simd    ] disabled
[2024/12/30 15:20:11] [ info] [cmetrics] version=0.9.9
[2024/12/30 15:20:11] [ info] [ctraces ] version=0.5.7
[2024/12/30 15:20:11] [ info] [input:dummy:dummy.0] initializing
[2024/12/30 15:20:11] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/12/30 15:20:11] [debug] [dummy:dummy.0] created event channels: read=25 write=26
[2024/12/30 15:20:11] [debug] [stackdriver:stackdriver.0] created event channels: read=27 write=28
[2024/12/30 15:20:11] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[2024/12/30 15:20:11] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2024/12/30 15:20:11] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2024/12/30 15:20:11] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:20:11] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:20:11] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:20:11] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:20:11] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:20:11] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:20:11] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:20:11] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:20:11] [ info] [output:stackdriver:stackdriver.0] worker #0 started
[2024/12/30 15:20:13] [debug] [task] created task=0x5dac070 id=0 OK
[2024/12/30 15:20:13] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:20:13] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:13] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:13] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:13] [debug] [out flush] cb_destroy coro_id=0
[2024/12/30 15:20:13] [debug] [retry] new retry created for task_id=0 attempts=1
[2024/12/30 15:20:13] [ warn] [engine] failed to flush chunk '3177720-1735572012.112976005.flb', retry in 11 seconds: task_id=0, input=dummy.0 > output=stackdriver.0 (out_id=0)
[2024/12/30 15:20:14] [debug] [task] created task=0x5e54fb0 id=1 OK
[2024/12/30 15:20:14] [debug] [output:stackdriver:stackdriver.0] task_id=1 assigned to thread #0
[2024/12/30 15:20:14] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:14] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:14] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:14] [debug] [retry] new retry created for task_id=1 attempts=1
[2024/12/30 15:20:14] [debug] [out flush] cb_destroy coro_id=1
[2024/12/30 15:20:14] [ warn] [engine] failed to flush chunk '3177720-1735572013.222851586.flb', retry in 8 seconds: task_id=1, input=dummy.0 > output=stackdriver.0 (out_id=0)
[2024/12/30 15:20:15] [debug] [task] created task=0x5efd950 id=2 OK
[2024/12/30 15:20:15] [debug] [output:stackdriver:stackdriver.0] task_id=2 assigned to thread #0
[2024/12/30 15:20:15] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:15] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:15] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:15] [debug] [out flush] cb_destroy coro_id=2
[2024/12/30 15:20:15] [debug] [retry] new retry created for task_id=2 attempts=1
[2024/12/30 15:20:15] [ warn] [engine] failed to flush chunk '3177720-1735572014.74535165.flb', retry in 7 seconds: task_id=2, input=dummy.0 > output=stackdriver.0 (out_id=0)
[2024/12/30 15:20:16] [debug] [task] created task=0x5fa8330 id=3 OK
[2024/12/30 15:20:16] [debug] [output:stackdriver:stackdriver.0] task_id=3 assigned to thread #0
[2024/12/30 15:20:16] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:16] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:16] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:16] [debug] [retry] new retry created for task_id=3 attempts=1
[2024/12/30 15:20:16] [debug] [out flush] cb_destroy coro_id=3
[2024/12/30 15:20:16] [ warn] [engine] failed to flush chunk '3177720-1735572015.73120248.flb', retry in 8 seconds: task_id=3, input=dummy.0 > output=stackdriver.0 (out_id=0)
[2024/12/30 15:20:17] [debug] [task] created task=0x7867d10 id=4 OK
[2024/12/30 15:20:17] [debug] [output:stackdriver:stackdriver.0] task_id=4 assigned to thread #0
[2024/12/30 15:20:17] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:17] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:17] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:17] [debug] [out flush] cb_destroy coro_id=4
[2024/12/30 15:20:17] [debug] [retry] new retry created for task_id=4 attempts=1
[2024/12/30 15:20:17] [ warn] [engine] failed to flush chunk '3177720-1735572016.87311800.flb', retry in 7 seconds: task_id=4, input=dummy.0 > output=stackdriver.0 (out_id=0)
[2024/12/30 15:20:18] [debug] [task] created task=0x7910670 id=5 OK
[2024/12/30 15:20:18] [debug] [output:stackdriver:stackdriver.0] task_id=5 assigned to thread #0
[2024/12/30 15:20:18] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:18] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:18] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:18] [debug] [retry] new retry created for task_id=5 attempts=1
[2024/12/30 15:20:18] [debug] [out flush] cb_destroy coro_id=5
[2024/12/30 15:20:18] [ warn] [engine] failed to flush chunk '3177720-1735572017.78804687.flb', retry in 8 seconds: task_id=5, input=dummy.0 > output=stackdriver.0 (out_id=0)
[2024/12/30 15:20:19] [debug] [task] created task=0x79b9030 id=6 OK
[2024/12/30 15:20:19] [debug] [output:stackdriver:stackdriver.0] task_id=6 assigned to thread #0
[2024/12/30 15:20:19] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:19] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:19] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:19] [debug] [retry] new retry created for task_id=6 attempts=1
[2024/12/30 15:20:19] [debug] [out flush] cb_destroy coro_id=6
[2024/12/30 15:20:19] [ warn] [engine] failed to flush chunk '3177720-1735572018.73267064.flb', retry in 7 seconds: task_id=6, input=dummy.0 > output=stackdriver.0 (out_id=0)
[2024/12/30 15:20:20] [debug] [task] created task=0x7a61a30 id=7 OK
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=7 assigned to thread #0
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:20] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:20] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:20] [debug] [retry] new retry created for task_id=7 attempts=1
[2024/12/30 15:20:20] [debug] [out flush] cb_destroy coro_id=7
[2024/12/30 15:20:20] [ warn] [engine] failed to flush chunk '3177720-1735572019.79273157.flb', retry in 10 seconds: task_id=7, input=dummy.0 > output=stackdriver.0 (out_id=0)
^C[2024/12/30 15:20:20] [engine] caught signal (SIGINT)
[2024/12/30 15:20:20] [debug] [task] created task=0x7b0a360 id=8 OK
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=8 assigned to thread #0
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:20] [ warn] [engine] service will shutdown in max 5 seconds
[2024/12/30 15:20:20] [debug] [engine] re-scheduled retry=0x5e54de0 for task 0
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:20] [debug] [engine] re-scheduled retry=0x5efd780 for task 1
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:20] [debug] [engine] re-scheduled retry=0x5fa6120 for task 2
[2024/12/30 15:20:20] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:20] [debug] [engine] re-scheduled retry=0x7867b40 for task 3
[2024/12/30 15:20:20] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:20] [debug] [engine] re-scheduled retry=0x79104a0 for task 4
[2024/12/30 15:20:20] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:20] [debug] [engine] re-scheduled retry=0x79b8e60 for task 5
[2024/12/30 15:20:20] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:20] [debug] [engine] re-scheduled retry=0x7a61860 for task 6
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:20] [debug] [engine] re-scheduled retry=0x7b0a140 for task 7
[2024/12/30 15:20:20] [debug] [out flush] cb_destroy coro_id=8
[2024/12/30 15:20:20] [ info] [input] pausing dummy.0
[2024/12/30 15:20:20] [debug] [out flush] cb_destroy coro_id=9
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=1 assigned to thread #0
[2024/12/30 15:20:20] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=2 assigned to thread #0
[2024/12/30 15:20:20] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=3 assigned to thread #0
[2024/12/30 15:20:20] [debug] [out flush] cb_destroy coro_id=10
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=4 assigned to thread #0
[2024/12/30 15:20:20] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=5 assigned to thread #0
[2024/12/30 15:20:20] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=6 assigned to thread #0
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:20] [debug] [output:stackdriver:stackdriver.0] task_id=7 assigned to thread #0
[2024/12/30 15:20:21] [debug] [out flush] cb_destroy coro_id=11
[2024/12/30 15:20:20] [debug] [retry] new retry created for task_id=8 attempts=1
[2024/12/30 15:20:21] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:20] [ warn] [engine] failed to flush chunk '3177720-1735572020.76303283.flb', retry in 1 seconds: task_id=8, input=dummy.0 > output=stackdriver.0 (out_id=0)
[2024/12/30 15:20:21] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:20] [debug] [task] task_id=0 reached retry-attempts limit 1/1
[2024/12/30 15:20:21] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:20] [error] [engine] chunk '3177720-1735572012.112976005.flb' cannot be retried: task_id=0, input=dummy.0 > output=stackdriver.0
[2024/12/30 15:20:21] [debug] [out flush] cb_destroy coro_id=12
[2024/12/30 15:20:20] [debug] [task] destroy task=0x5dac070 (task_id=0)
[2024/12/30 15:20:21] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:21] [debug] [task] task_id=1 reached retry-attempts limit 1/1
[2024/12/30 15:20:21] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:21] [error] [engine] chunk '3177720-1735572013.222851586.flb' cannot be retried: task_id=1, input=dummy.0 > output=stackdriver.0
[2024/12/30 15:20:21] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:21] [debug] [task] destroy task=0x5e54fb0 (task_id=1)
[2024/12/30 15:20:21] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:21] [debug] [task] task_id=2 reached retry-attempts limit 1/1
[2024/12/30 15:20:21] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:21] [error] [engine] chunk '3177720-1735572014.74535165.flb' cannot be retried: task_id=2, input=dummy.0 > output=stackdriver.0
[2024/12/30 15:20:21] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:21] [debug] [task] destroy task=0x5efd950 (task_id=2)
[2024/12/30 15:20:21] [debug] [out flush] cb_destroy coro_id=13
[2024/12/30 15:20:21] [debug] [task] task_id=3 reached retry-attempts limit 1/1
[2024/12/30 15:20:21] [debug] [out flush] cb_destroy coro_id=14
[2024/12/30 15:20:21] [error] [engine] chunk '3177720-1735572015.73120248.flb' cannot be retried: task_id=3, input=dummy.0 > output=stackdriver.0
[2024/12/30 15:20:21] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:21] [debug] [task] destroy task=0x5fa8330 (task_id=3)
[2024/12/30 15:20:21] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:21] [debug] [task] task_id=4 reached retry-attempts limit 1/1
[2024/12/30 15:20:21] [debug] [out flush] cb_destroy coro_id=15
2024/12/30 15:20:21] [error] [engine] chunk '3177720-1735572016.87311800.flb' cannot be retried: task_id=4, input=dummy.0 > output=stackdriver.0
[2024/12/30 15:20:21] [debug] [task] destroy task=0x7867d10 (task_id=4)
[2024/12/30 15:20:21] [debug] [task] task_id=5 reached retry-attempts limit 1/1
[2024/12/30 15:20:21] [error] [engine] chunk '3177720-1735572017.78804687.flb' cannot be retried: task_id=5, input=dummy.0 > output=stackdriver.0
[2024/12/30 15:20:21] [debug] [task] destroy task=0x7910670 (task_id=5)
[2024/12/30 15:20:21] [debug] [task] task_id=6 reached retry-attempts limit 1/1
[2024/12/30 15:20:21] [error] [engine] chunk '3177720-1735572018.73267064.flb' cannot be retried: task_id=6, input=dummy.0 > output=stackdriver.0
[2024/12/30 15:20:21] [debug] [task] destroy task=0x79b9030 (task_id=6)
[2024/12/30 15:20:21] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:21] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:21] [debug] [output:stackdriver:stackdriver.0] task_id=8 assigned to thread #0
[2024/12/30 15:20:21] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:20:21] [ info] [task] dummy/dummy.0 has 2 pending task(s):
[2024/12/30 15:20:21] [ info] [task]   task_id=7 still running on route(s): stackdriver/stackdriver.0 
[2024/12/30 15:20:21] [ info] [task]   task_id=8 still running on route(s): stackdriver/stackdriver.0 
[2024/12/30 15:20:21] [ info] [input] pausing dummy.0
[2024/12/30 15:20:21] [debug] [out flush] cb_destroy coro_id=16
[2024/12/30 15:20:21] [debug] [task] task_id=7 reached retry-attempts limit 1/1
[2024/12/30 15:20:21] [error] [engine] chunk '3177720-1735572019.79273157.flb' cannot be retried: task_id=7, input=dummy.0 > output=stackdriver.0
[2024/12/30 15:20:21] [debug] [task] destroy task=0x7a61a30 (task_id=7)
[2024/12/30 15:20:21] [ warn] [net] getaddrinfo(host='logging-other.com', err=4): Domain name not found
[2024/12/30 15:20:21] [debug] [upstream] connection #-1 failed to logging-other.com:443
[2024/12/30 15:20:21] [debug] [out flush] cb_destroy coro_id=17
[2024/12/30 15:20:21] [debug] [task] task_id=8 reached retry-attempts limit 1/1
[2024/12/30 15:20:21] [error] [engine] chunk '3177720-1735572020.76303283.flb' cannot be retried: task_id=8, input=dummy.0 > output=stackdriver.0
[2024/12/30 15:20:21] [debug] [task] destroy task=0x7b0a360 (task_id=8)
[2024/12/30 15:20:22] [ info] [engine] service has stopped (0 pending tasks)
[2024/12/30 15:20:22] [ info] [input] pausing dummy.0
[2024/12/30 15:20:22] [ info] [output:stackdriver:stackdriver.0] thread worker #0 stopping...
[2024/12/30 15:20:22] [ info] [output:stackdriver:stackdriver.0] thread worker #0 stopped
==3177720== 
==3177720== HEAP SUMMARY:
==3177720==     in use at exit: 0 bytes in 0 blocks
==3177720==   total heap usage: 28,554 allocs, 28,554 frees, 9,214,446 bytes allocated
==3177720== 
==3177720== All heap blocks were freed -- no leaks are possible
==3177720== 
==3177720== For lists of detected and suppressed errors, rerun with: -s
==3177720== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Valgrind debug output with https://logging.googleapis.com manually set

braydonk@debian-dev-machine:~/Git/fluent-bit/build/bin$ valgrind --leak-check=full ./fluent-bit -v -c x.conf
==3177038== Memcheck, a memory error detector
==3177038== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3177038== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==3177038== Command: ./fluent-bit -v -c x.conf
==3177038== 
Fluent Bit v3.2.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2024/12/30 15:15:33] [ info] Configuration:
[2024/12/30 15:15:33] [ info]  flush time     | 1.000000 seconds
[2024/12/30 15:15:33] [ info]  grace          | 5 seconds
[2024/12/30 15:15:33] [ info]  daemon         | 0
[2024/12/30 15:15:33] [ info] ___________
[2024/12/30 15:15:33] [ info]  inputs:
[2024/12/30 15:15:33] [ info]      dummy
[2024/12/30 15:15:33] [ info] ___________
[2024/12/30 15:15:33] [ info]  filters:
[2024/12/30 15:15:33] [ info] ___________
[2024/12/30 15:15:33] [ info]  outputs:
[2024/12/30 15:15:33] [ info]      stackdriver.0
[2024/12/30 15:15:33] [ info] ___________
[2024/12/30 15:15:33] [ info]  collectors:
[2024/12/30 15:15:33] [ info] [fluent bit] version=3.2.3, commit=b714f6d815, pid=3177038
[2024/12/30 15:15:33] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/12/30 15:15:33] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/12/30 15:15:33] [ info] [simd    ] disabled
[2024/12/30 15:15:33] [ info] [cmetrics] version=0.9.9
[2024/12/30 15:15:33] [ info] [ctraces ] version=0.5.7
[2024/12/30 15:15:34] [ info] [input:dummy:dummy.0] initializing
[2024/12/30 15:15:34] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/12/30 15:15:34] [debug] [dummy:dummy.0] created event channels: read=25 write=26
[2024/12/30 15:15:34] [debug] [stackdriver:stackdriver.0] created event channels: read=27 write=28
[2024/12/30 15:15:34] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[2024/12/30 15:15:34] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2024/12/30 15:15:34] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2024/12/30 15:15:34] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:34] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:34] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:34] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:34] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:34] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:34] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:34] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:34] [ info] [output:stackdriver:stackdriver.0] worker #0 started
[2024/12/30 15:15:36] [debug] [task] created task=0x5dac070 id=0 OK
[2024/12/30 15:15:36] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:36] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:37] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 is connected
[2024/12/30 15:15:37] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:37] [debug] [task] created task=0x5e959a0 id=1 OK
[2024/12/30 15:15:37] [debug] [output:stackdriver:stackdriver.0] task_id=1 assigned to thread #0
[2024/12/30 15:15:37] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:37] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 is connected
[2024/12/30 15:15:37] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:37] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:37] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:37] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:37] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:37] [debug] [out flush] cb_destroy coro_id=1
[2024/12/30 15:15:37] [debug] [task] destroy task=0x5e959a0 (task_id=1)
[2024/12/30 15:15:37] [debug] [out flush] cb_destroy coro_id=0
[2024/12/30 15:15:37] [debug] [task] destroy task=0x5dac070 (task_id=0)
[2024/12/30 15:15:38] [debug] [task] created task=0x5f75520 id=0 OK
[2024/12/30 15:15:38] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:38] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:38] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:38] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:38] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:38] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:38] [debug] [out flush] cb_destroy coro_id=2
[2024/12/30 15:15:38] [debug] [task] destroy task=0x5f75520 (task_id=0)
[2024/12/30 15:15:39] [debug] [task] created task=0x5fe03b0 id=0 OK
[2024/12/30 15:15:39] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:39] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:39] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:39] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:39] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:39] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:39] [debug] [out flush] cb_destroy coro_id=3
[2024/12/30 15:15:39] [debug] [task] destroy task=0x5fe03b0 (task_id=0)
[2024/12/30 15:15:40] [debug] [task] created task=0x5ffce80 id=0 OK
[2024/12/30 15:15:40] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:40] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:40] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:40] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:40] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:40] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:40] [debug] [out flush] cb_destroy coro_id=4
[2024/12/30 15:15:40] [debug] [task] destroy task=0x5ffce80 (task_id=0)
[2024/12/30 15:15:41] [debug] [task] created task=0x78c9070 id=0 OK
[2024/12/30 15:15:41] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:41] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:41] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:41] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:41] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:41] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:41] [debug] [task] destroy task=0x78c9070 (task_id=0)
[2024/12/30 15:15:41] [debug] [out flush] cb_destroy coro_id=5
[2024/12/30 15:15:42] [debug] [task] created task=0x7931ec0 id=0 OK
[2024/12/30 15:15:42] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:42] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:42] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:42] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:42] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:42] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:42] [debug] [out flush] cb_destroy coro_id=6
[2024/12/30 15:15:42] [debug] [task] destroy task=0x7931ec0 (task_id=0)
[2024/12/30 15:15:43] [debug] [task] created task=0x799ad10 id=0 OK
[2024/12/30 15:15:43] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:43] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:43] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:43] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:43] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:43] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:43] [debug] [out flush] cb_destroy coro_id=7
[2024/12/30 15:15:43] [debug] [task] destroy task=0x799ad10 (task_id=0)
[2024/12/30 15:15:44] [debug] [task] created task=0x7a05ba0 id=0 OK
[2024/12/30 15:15:44] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:44] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:44] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:44] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:44] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:44] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:44] [debug] [task] destroy task=0x7a05ba0 (task_id=0)
[2024/12/30 15:15:44] [debug] [out flush] cb_destroy coro_id=8
[2024/12/30 15:15:45] [debug] [task] created task=0x7a6e9f0 id=0 OK
[2024/12/30 15:15:45] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:45] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:45] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:45] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:45] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:45] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:45] [debug] [out flush] cb_destroy coro_id=9
[2024/12/30 15:15:45] [debug] [task] destroy task=0x7a6e9f0 (task_id=0)
[2024/12/30 15:15:46] [debug] [task] created task=0x7ad7840 id=0 OK
[2024/12/30 15:15:46] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:46] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:46] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:46] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:46] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:46] [debug] [upstream] KA connection #49 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:46] [debug] [out flush] cb_destroy coro_id=10
[2024/12/30 15:15:46] [debug] [task] destroy task=0x7ad7840 (task_id=0)
^C[2024/12/30 15:15:46] [engine] caught signal (SIGINT)
[2024/12/30 15:15:46] [debug] [task] created task=0x7b406e0 id=0 OK
[2024/12/30 15:15:46] [debug] [output:stackdriver:stackdriver.0] task_id=0 assigned to thread #0
[2024/12/30 15:15:46] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2024/12/30 15:15:46] [ warn] [engine] service will shutdown in max 5 seconds
[2024/12/30 15:15:46] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 has been assigned (recycled)
[2024/12/30 15:15:46] [ info] [input] pausing dummy.0
[2024/12/30 15:15:46] [debug] [http_client] not using http_proxy for header
[2024/12/30 15:15:46] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2024/12/30 15:15:46] [debug] [upstream] KA connection #48 to logging.googleapis.com:443 is now available
[2024/12/30 15:15:46] [debug] [task] destroy task=0x7b406e0 (task_id=0)
[2024/12/30 15:15:46] [debug] [out flush] cb_destroy coro_id=11
[2024/12/30 15:15:47] [ info] [engine] service has stopped (0 pending tasks)
[2024/12/30 15:15:47] [ info] [input] pausing dummy.0
[2024/12/30 15:15:47] [ info] [output:stackdriver:stackdriver.0] thread worker #0 stopping...
[2024/12/30 15:15:47] [ info] [output:stackdriver:stackdriver.0] thread worker #0 stopped
==3177038== 
==3177038== HEAP SUMMARY:
==3177038==     in use at exit: 0 bytes in 0 blocks
==3177038==   total heap usage: 13,042 allocs, 13,042 frees, 6,622,294 bytes allocated
==3177038== 
==3177038== All heap blocks were freed -- no leaks are possible
==3177038== 
==3177038== For lists of detected and suppressed errors, rerun with: -s
==3177038== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

fluent/fluent-bit-docs#1538

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[braydonk]

The -DFLB_SANITIZE_MEMORY=On tests on MacOS are a flake; the failing test has nothing to do with what this PR changes.

[qingling128]

LGTM