in_http: add support for form/url_encoded payloads.

[pwhelan]

Add support for Content-Type:application/x-www-form-urlencoded to the in_http plugin.

This PR addresses #5385.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[pwhelan]

The standard configuration should work since the payload is detected by the Content-Type header.

Here is a valgrind log with debugging enabled:

valgrind ./build/bin/fluent-bit -v -i http -p port=8888 -o stdout -p format=json -f 1
==105154== Memcheck, a memory error detector
==105154== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==105154== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==105154== Command: ./build/bin/fluent-bit -v -i http -p port=8888 -o stdout -p format=json -f 1
==105154== 
Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/03/06 12:35:42] [ info] Configuration:
[2023/03/06 12:35:42] [ info]  flush time     | 1.000000 seconds
[2023/03/06 12:35:42] [ info]  grace          | 5 seconds
[2023/03/06 12:35:42] [ info]  daemon         | 0
[2023/03/06 12:35:42] [ info] ___________
[2023/03/06 12:35:42] [ info]  inputs:
[2023/03/06 12:35:42] [ info]      http
[2023/03/06 12:35:42] [ info] ___________
[2023/03/06 12:35:42] [ info]  filters:
[2023/03/06 12:35:42] [ info] ___________
[2023/03/06 12:35:42] [ info]  outputs:
[2023/03/06 12:35:42] [ info]      stdout.0
[2023/03/06 12:35:42] [ info] ___________
[2023/03/06 12:35:42] [ info]  collectors:
[2023/03/06 12:35:42] [ info] [fluent bit] version=2.1.0, commit=3b7fab95fc, pid=105154
[2023/03/06 12:35:42] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/03/06 12:35:42] [ info] [output:stdout:stdout.0] worker #0 started
[2023/03/06 12:35:42] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/03/06 12:35:42] [ info] [cmetrics] version=0.5.8
[2023/03/06 12:35:42] [ info] [ctraces ] version=0.3.0
[2023/03/06 12:35:42] [ info] [input:http:http.0] initializing
[2023/03/06 12:35:42] [ info] [input:http:http.0] storage_strategy='memory' (memory only)
[2023/03/06 12:35:42] [debug] [http:http.0] created event channels: read=38 write=39
[2023/03/06 12:35:42] [debug] [downstream] listening on 0.0.0.0:8888
[2023/03/06 12:35:42] [debug] [stdout:stdout.0] created event channels: read=41 write=42
[2023/03/06 12:35:42] [ info] [sp] stream processor started
[2023/03/06 12:35:46] [debug] [input chunk] update output instances with new chunk size diff=375
[2023/03/06 12:35:47] [debug] [task] created task=0x556cf20 id=0 OK
[2023/03/06 12:35:47] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[{"date":1678116946.290122,"token":"gIkuvaNzQIHg97ATvDxqgjtO","team_id":"T0001","team_domain":"example","enterprise_id":"E0001","enterprise_name":"Globular%20Construct%20Inc","channel_id":"C2147483705","channel_name":"test","user_id":"U2147483697","user_name":"Steve","command":"/weather","text":"94070","response_url":"https://hooks.slack.com/commands/1234/5678","trigger_id":"13345224609.738474920.8088930838d88f008e0","api_app_id":"A123456"}]
[2023/03/06 12:35:47] [debug] [out flush] cb_destroy coro_id=0
[2023/03/06 12:35:47] [debug] [task] destroy task=0x556cf20 (task_id=0)
[2023/03/06 12:35:47] [debug] [input chunk] update output instances with new chunk size diff=375
[2023/03/06 12:35:48] [debug] [task] created task=0x5649130 id=0 OK
[{"date":1678116947.942072,"token":"gIkuvaNzQIHg97ATvDxqgjtO","team_id":"T0001","team_domain":"example","enterprise_id":"E0001","enterprise_name":"Globular%20Construct%20Inc","channel_id":"C2147483705","channel_name":"test","user_id":"U2147483697","user_name":"Steve","command":"/weather","text":"94070","response_url":"https://hooks.slack.com/commands/1234/5678","trigger_id":"13345224609.738474920.8088930838d88f008e0","api_app_id":"A123456"}]
[2023/03/06 12:35:48] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/03/06 12:35:48] [debug] [out flush] cb_destroy coro_id=1
[2023/03/06 12:35:48] [debug] [task] destroy task=0x5649130 (task_id=0)
[2023/03/06 12:35:48] [debug] [input chunk] update output instances with new chunk size diff=375
[{"date":1678116948.925301,"token":"gIkuvaNzQIHg97ATvDxqgjtO","team_id":"T0001","team_domain":"example","enterprise_id":"E0001","enterprise_name":"Globular%20Construct%20Inc","channel_id":"C2147483705","channel_name":"test","user_id":"U2147483697","user_name":"Steve","command":"/weather","text":"94070","response_url":"https://hooks.slack.com/commands/1234/5678","trigger_id":"13345224609.738474920.8088930838d88f008e0","api_app_id":"A123456"}]
[2023/03/06 12:35:49] [debug] [task] created task=0x5724e80 id=0 OK
[2023/03/06 12:35:49] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/03/06 12:35:49] [debug] [out flush] cb_destroy coro_id=2
[2023/03/06 12:35:49] [debug] [task] destroy task=0x5724e80 (task_id=0)
[2023/03/06 12:35:49] [debug] [input chunk] update output instances with new chunk size diff=375
[2023/03/06 12:35:50] [debug] [task] created task=0x5800bd0 id=0 OK
[2023/03/06 12:35:50] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[{"date":1678116949.676583,"token":"gIkuvaNzQIHg97ATvDxqgjtO","team_id":"T0001","team_domain":"example","enterprise_id":"E0001","enterprise_name":"Globular%20Construct%20Inc","channel_id":"C2147483705","channel_name":"test","user_id":"U2147483697","user_name":"Steve","command":"/weather","text":"94070","response_url":"https://hooks.slack.com/commands/1234/5678","trigger_id":"13345224609.738474920.8088930838d88f008e0","api_app_id":"A123456"}]
[2023/03/06 12:35:50] [debug] [out flush] cb_destroy coro_id=3
[2023/03/06 12:35:50] [debug] [task] destroy task=0x5800bd0 (task_id=0)
[2023/03/06 12:35:50] [debug] [input chunk] update output instances with new chunk size diff=375
[2023/03/06 12:35:51] [debug] [task] created task=0x581f8a0 id=0 OK
[2023/03/06 12:35:51] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[{"date":1678116950.423915,"token":"gIkuvaNzQIHg97ATvDxqgjtO","team_id":"T0001","team_domain":"example","enterprise_id":"E0001","enterprise_name":"Globular%20Construct%20Inc","channel_id":"C2147483705","channel_name":"test","user_id":"U2147483697","user_name":"Steve","command":"/weather","text":"94070","response_url":"https://hooks.slack.com/commands/1234/5678","trigger_id":"13345224609.738474920.8088930838d88f008e0","api_app_id":"A123456"}]
[2023/03/06 12:35:51] [debug] [out flush] cb_destroy coro_id=4
[2023/03/06 12:35:51] [debug] [task] destroy task=0x581f8a0 (task_id=0)
[2023/03/06 12:35:51] [debug] [input chunk] update output instances with new chunk size diff=375
[2023/03/06 12:35:51] [debug] [input chunk] update output instances with new chunk size diff=375
[2023/03/06 12:35:52] [debug] [task] created task=0x7246520 id=0 OK
[{"date":1678116951.144309,"token":"gIkuvaNzQIHg97ATvDxqgjtO","team_id":"T0001","team_domain":"example","enterprise_id":"E0001","enterprise_name":"Globular%20Construct%20Inc","channel_id":"C2147483705","channel_name":"test","user_id":"U2147483697","user_name":"Steve","command":"/weather","text":"94070","response_url":"https://hooks.slack.com/commands/1234/5678","trigger_id":"13345224609.738474920.8088930838d88f008e0","api_app_id":"A123456"},{"date":1678116951.853738,"token":"gIkuvaNzQIHg97ATvDxqgjtO","team_id":"T0001","team_domain":"example","enterprise_id":"E0001","enterprise_name":"Globular%20Construct%20Inc","channel_id":"C2147483705","channel_name":"test","user_id":"U2147483697","user_name":"Steve","command":"/weather","text":"94070","response_url":"https://hooks.slack.com/commands/1234/5678","trigger_id":"13345224609.738474920.8088930838d88f008e0","api_app_id":"A123456"}]
[2023/03/06 12:35:52] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/03/06 12:35:52] [debug] [out flush] cb_destroy coro_id=5
[2023/03/06 12:35:52] [debug] [task] destroy task=0x7246520 (task_id=0)
[2023/03/06 12:35:52] [debug] [input chunk] update output instances with new chunk size diff=375
[2023/03/06 12:35:53] [debug] [task] created task=0x7324630 id=0 OK
[{"date":1678116952.656844,"token":"gIkuvaNzQIHg97ATvDxqgjtO","team_id":"T0001","team_domain":"example","enterprise_id":"E0001","enterprise_name":"Globular%20Construct%20Inc","channel_id":"C2147483705","channel_name":"test","user_id":"U2147483697","user_name":"Steve","command":"/weather","text":"94070","response_url":"https://hooks.slack.com/commands/1234/5678","trigger_id":"13345224609.738474920.8088930838d88f008e0","api_app_id":"A123456"}]
[2023/03/06 12:35:53] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/03/06 12:35:53] [debug] [task] destroy task=0x7324630 (task_id=0)
[2023/03/06 12:35:53] [debug] [out flush] cb_destroy coro_id=6
^C[2023/03/06 12:35:54] [engine] caught signal (SIGINT)
[2023/03/06 12:35:54] [ warn] [engine] service will shutdown in max 5 seconds
[2023/03/06 12:35:55] [ info] [engine] service has stopped (0 pending tasks)
[2023/03/06 12:35:55] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/03/06 12:35:55] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==105154== 
==105154== HEAP SUMMARY:
==105154==     in use at exit: 0 bytes in 0 blocks
==105154==   total heap usage: 2,224 allocs, 2,224 frees, 7,215,199 bytes allocated
==105154== 
==105154== All heap blocks were freed -- no leaks are possible
==105154== 
==105154== For lists of detected and suppressed errors, rerun with: -s
==105154== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[cosmo0920]

Looks attractive. I'd confirmed that this patch can work with non-ASCII and URL encoded characters:

• Hiragana
• Katakana
• Chinese Characters(漢字/汉字)

% curl -X POST localhost:9880 -H "Content-Type: application/x-www-form-urlencoded" -d "param1=value1&param2=%E3%83%A4%E3%83%83%E3%83%9B%E3%83%BC"
% curl -X POST localhost:9880 -H "Content-Type: application/x-www-form-urlencoded" -d "param1=value1&param2=%E3%81%B5%E3%82%8B%E3%83%BC%E3%81%88%E3%82%93%E3%81%A8%E3%81%B3%E3%81%A3%E3%81%A8"
% curl -X POST localhost:9880 -H "Content-Type: application/x-www-form-urlencoded" -d "param1=value1&param2=Fluent+Bit%E3%82%92%E4%BD%BF%E3%81%86"             
% curl -X POST localhost:9880 -H "Content-Type: application/x-www-form-urlencoded" -d "param1=value1&param2=%E4%BB%8A%E5%A4%A9%E6%88%91%E5%8E%BB%E4%BA%86%E4%B8%A4%E6%89%80%E5%AD%A6%E6%A0%A1"

[0] http.0: [1678239371.634798000, {"param1"=>"value1", "param2"=>"ヤッホー"}]
[0] http.0: [1678239384.303760000, {"param1"=>"value1", "param2"=>"ふるーえんとびっと"}]
[0] http.0: [1678239430.617883000, {"param1"=>"value1", "param2"=>"Fluent Bitを使う"}]
[0] http.0: [1678239720.287393000, {"param1"=>"value1", "param2"=>"今天我去了两所学校"}]

[pwhelan]

What is missing is support for Unicode encoding, ie: %uXXXX. But it appears it is not in wide use:

• https://stackoverflow.com/a/1020299
• https://www.w3schools.com/tags/ref_urlencode.ASP - not even mentioned
• https://en.wikipedia.org/wiki/URL_encoding#Non-standard_implementations

[cosmo0920]

%uXXXX encoding could be used for emoji characters (with surrogate pairs UTF-8 characters). However, emojis are also handled by URL encoding. Currently, this PR seems to be enough to handle URL encoding.
If someone interested in handling %uXXXX encoding, we should implement to handle them.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.