From 26a5a0f6fb2a6ce5a4cc4be183f1aa3ee4eb28d9 Mon Sep 17 00:00:00 2001 From: composer <2789706336@qq.com> Date: Wed, 23 Oct 2024 15:42:09 +0800 Subject: [PATCH 1/5] feat: add doris output plugin Signed-off-by: composer <2789706336@qq.com> --- .gitbook.yaml | 1 + SUMMARY.md | 1 + .../configuring-fluent-bit/README.md | 1 + administration/transport-security.md | 1 + installation/sources/build-and-install.md | 1 + pipeline/outputs/doris.md | 30 +++++++++++++++++++ vale-styles/FluentBit/Spelling-exceptions.txt | 1 + 7 files changed, 36 insertions(+) create mode 100644 pipeline/outputs/doris.md diff --git a/.gitbook.yaml b/.gitbook.yaml index f763fd123..6d105b687 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -56,6 +56,7 @@ redirects: output/chronicle: ./pipeline/outputs/chronicle.md output/cloudwatch: ./pipeline/outputs/cloudwatch.md output/datadog: ./pipeline/outputs/datadog.md + output/doris: ./pipeline/outputs/doris.md output/es: ./pipeline/outputs/elasticsearch.md output/fabric: ./pipeline/outputs/azure_kusto.md output/file: ./pipeline/outputs/file.md diff --git a/SUMMARY.md b/SUMMARY.md index 432b8fff3..b4d9fb6fb 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -170,6 +170,7 @@ * [Azure Logs Ingestion API](pipeline/outputs/azure_logs_ingestion.md) * [Counter](pipeline/outputs/counter.md) * [Datadog](pipeline/outputs/datadog.md) + * [Doris](pipeline/outputs/doris.md) * [Dynatrace](pipeline/outputs/dynatrace.md) * [Elasticsearch](pipeline/outputs/elasticsearch.md) * [File](pipeline/outputs/file.md) diff --git a/administration/configuring-fluent-bit/README.md b/administration/configuring-fluent-bit/README.md index 794d41950..38ad77146 100644 --- a/administration/configuring-fluent-bit/README.md +++ b/administration/configuring-fluent-bit/README.md @@ -100,6 +100,7 @@ Outputs bigquery Send events to BigQuery via streaming insert counter Records counter datadog Send events to DataDog HTTP Event Collector + doris Apache Doris es Elasticsearch exit Exit after a number of flushes (test purposes) file Generate log file diff --git a/administration/transport-security.md b/administration/transport-security.md index 4443cfd70..dca847495 100644 --- a/administration/transport-security.md +++ b/administration/transport-security.md @@ -32,6 +32,7 @@ The following **output** plugins can take advantage of the TLS feature: * [Azure Logs Ingestion API](../pipeline/outputs/azure_logs_ingestion.md) * [BigQuery](../pipeline/outputs/bigquery.md) * [Datadog](../pipeline/outputs/datadog.md) +* [Doris](../pipeline/outputs/doris.md) * [Elasticsearch](../pipeline/outputs/elasticsearch.md) * [Forward](../pipeline/outputs/forward.md) * [GELF](../pipeline/outputs/gelf.md) diff --git a/installation/sources/build-and-install.md b/installation/sources/build-and-install.md index 99674abf8..16c5bc1c0 100644 --- a/installation/sources/build-and-install.md +++ b/installation/sources/build-and-install.md @@ -188,6 +188,7 @@ The _output plugins_ gives the capacity to flush the information to some externa | [FLB\_OUT\_COUNTER](../../pipeline/outputs/counter.md) | Enable Counter output plugin | On | | [FLB\_OUT\_CLOUDWATCH\_LOGS](../../pipeline/outputs/cloudwatch.md) | Enable Amazon CloudWatch output plugin | On | | [FLB\_OUT\_DATADOG](../../pipeline/outputs/datadog.md) | Enable Datadog output plugin | On | +| [FLB\_OUT|_DORIS](../../pipeline/outputs/doris.md) | Enable Apache Doris output plugin | On | | [FLB\_OUT\_ES](../../pipeline/outputs/elasticsearch.md) | Enable [Elastic Search](http://www.elastic.co) output plugin | On | | [FLB\_OUT\_FILE](../../pipeline/outputs/file.md) | Enable File output plugin | On | | [FLB\_OUT\_KINESIS\_FIREHOSE](../../pipeline/outputs/firehose.md) | Enable Amazon Kinesis Data Firehose output plugin | On | diff --git a/pipeline/outputs/doris.md b/pipeline/outputs/doris.md new file mode 100644 index 000000000..f696c0534 --- /dev/null +++ b/pipeline/outputs/doris.md @@ -0,0 +1,30 @@ +--- +description: Send logs to Apache Doris +--- + +# Apache Doris + +The **doris** output plugin lets you ingest your records into an +[Apache Doris](https://doris.apache.org) database. To use this plugin, you must have an +operational Doris service running in your environment. + +## Configuration Parameters + +| Key | Description | Default | +| :--- | :--- | :--- | +| `host` | HTTP address of the target Doris fe or be | `127.0.0.1` | +| `port` | HTTP port of the target Doris fe or be | `8300` | +| `user` | Username for Doris access | _none_ | +| `password` | Password for Doris access | _none_ | +| `database` | The target Doris database | _none_ | +| `table` | The target Doris table | _none_ | +| `time_key` | The name of the time key in the output record | `date` | +| `columns` | The column mappings, details in [Doris stream load](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual) | `date, log` | +| `timeout_second` | Timeout seconds for Doris stream load | `60` | +| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | + +### TLS / SSL + +Doris output plugin supports TLS/SSL. For more details about the properties +available and general configuration, refer to[TLS/SSL](../../administration/transport-security.md). + diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index 18b032117..a49cb70e8 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -36,6 +36,7 @@ deprovisioning deprovisions Devo DogStatsD +Doris downsample downsampled downsamples From 78e6985ae7c7e2f25ba8647806158bf38c7647a7 Mon Sep 17 00:00:00 2001 From: composer <2789706336@qq.com> Date: Wed, 23 Oct 2024 17:51:03 +0800 Subject: [PATCH 2/5] feat: add example Signed-off-by: composer <2789706336@qq.com> --- pipeline/outputs/doris.md | 45 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/pipeline/outputs/doris.md b/pipeline/outputs/doris.md index f696c0534..c4d7c8a8c 100644 --- a/pipeline/outputs/doris.md +++ b/pipeline/outputs/doris.md @@ -13,7 +13,7 @@ operational Doris service running in your environment. | Key | Description | Default | | :--- | :--- | :--- | | `host` | HTTP address of the target Doris fe or be | `127.0.0.1` | -| `port` | HTTP port of the target Doris fe or be | `8300` | +| `port` | HTTP port of the target Doris fe or be | `8030` | | `user` | Username for Doris access | _none_ | | `password` | Password for Doris access | _none_ | | `database` | The target Doris database | _none_ | @@ -28,3 +28,46 @@ operational Doris service running in your environment. Doris output plugin supports TLS/SSL. For more details about the properties available and general configuration, refer to[TLS/SSL](../../administration/transport-security.md). +## Get started + +To insert records into a Doris database, you run the plugin from the +command line or through the configuration file: + +### Command Line + +The **doris** plugin can read the parameters from the command through the `-p` argument (property). + +An example: + +```shell copy +fluent-bit -i cpu -t cpu -o doris \ + -m '*' \ + -p host=127.0.0.1 \ + -p port=8030 \ + -p user=admin \ + -p password=admin \ + -p database=d_fb \ + -p table=t_fb \ + -p columns='date, log=cast(cpu_p as string)' +``` + +### Configuration File + +In your main configuration file append the following `Input` and `Output` sections. + +```python +[INPUT] + Name cpu + Tag cpu + +[OUTPUT] + name doris + match * + host 127.0.0.1 + port 8030 + user admin + password admin + database d_fb + table t_fb + columns date, log=cast(cpu_p as string) +``` \ No newline at end of file From 3cf26f8492487ee7ead3fa5684a369b93ea46ede Mon Sep 17 00:00:00 2001 From: composer <2789706336@qq.com> Date: Thu, 31 Oct 2024 05:39:50 +0800 Subject: [PATCH 3/5] feat: add new config Signed-off-by: composer <2789706336@qq.com> --- pipeline/outputs/doris.md | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/pipeline/outputs/doris.md b/pipeline/outputs/doris.md index c4d7c8a8c..3adba61cb 100644 --- a/pipeline/outputs/doris.md +++ b/pipeline/outputs/doris.md @@ -12,32 +12,33 @@ operational Doris service running in your environment. | Key | Description | Default | | :--- | :--- | :--- | -| `host` | HTTP address of the target Doris fe or be | `127.0.0.1` | -| `port` | HTTP port of the target Doris fe or be | `8030` | -| `user` | Username for Doris access | _none_ | -| `password` | Password for Doris access | _none_ | -| `database` | The target Doris database | _none_ | -| `table` | The target Doris table | _none_ | -| `time_key` | The name of the time key in the output record | `date` | -| `columns` | The column mappings, details in [Doris stream load](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual) | `date, log` | -| `timeout_second` | Timeout seconds for Doris stream load | `60` | +| `host` | HTTP address of the target Doris frontend (fe) or frontend (be). | `127.0.0.1` | +| `port` | HTTP port of the target Doris frontend (fe) or frontend (be). | `8030` | +| `user` | Username for Doris access. | _none_ | +| `password` | Password for Doris access. | _none_ | +| `database` | The target Doris database. | _none_ | +| `table` | The target Doris table. | _none_ | +| `label_prefix` | Doris stream load label prefix, the final generated Label is {label_prefix}\_{timestamp}\_{uuid}. | `fluentbit` | +| `time_key` | The name of the time key in the output record. | `date` | +| `header` | Doris stream load headers. Multiple headers can be set. See [Doris stream load](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual) for details. | _none_ | +| `log_request` | Whether to output Doris Stream Load request and response metadata in logs for troubleshooting. | `true` | +| `log_progress_interval` | The time interval in seconds to calculate and output the speed in the log. Set to 0 to disable this type of logging. | `10` | | `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | ### TLS / SSL -Doris output plugin supports TLS/SSL. For more details about the properties -available and general configuration, refer to[TLS/SSL](../../administration/transport-security.md). +Doris output plugin supports TLS/SSL. See [TLS/SSL](../../administration/transport-security.md) +for more details about the supported properties and general configuration. ## Get started -To insert records into a Doris database, you run the plugin from the -command line or through the configuration file: +To insert records into a Doris database, run the plugin from the command line or define a configuration file: ### Command Line -The **doris** plugin can read the parameters from the command through the `-p` argument (property). +The Doris plugin can read the parameters from the command through the `-p` argument, -An example: +as shown in the following example: ```shell copy fluent-bit -i cpu -t cpu -o doris \ @@ -48,14 +49,14 @@ fluent-bit -i cpu -t cpu -o doris \ -p password=admin \ -p database=d_fb \ -p table=t_fb \ - -p columns='date, log=cast(cpu_p as string)' + -p header='columns date, cpu_p, log=cast(cpu_p as string)' ``` ### Configuration File -In your main configuration file append the following `Input` and `Output` sections. +In your main configuration file, append the following `Input` and `Output` sections. -```python +```python copy [INPUT] Name cpu Tag cpu @@ -69,5 +70,5 @@ In your main configuration file append the following `Input` and `Output` sectio password admin database d_fb table t_fb - columns date, log=cast(cpu_p as string) + columns date, cpu_p, log=cast(cpu_p as string) ``` \ No newline at end of file From 7a29de0fa7c0e4660ca3962b1becd36991743769 Mon Sep 17 00:00:00 2001 From: composer <2789706336@qq.com> Date: Thu, 31 Oct 2024 05:44:44 +0800 Subject: [PATCH 4/5] chore Signed-off-by: composer <2789706336@qq.com> --- pipeline/outputs/doris.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pipeline/outputs/doris.md b/pipeline/outputs/doris.md index 3adba61cb..af81acb2f 100644 --- a/pipeline/outputs/doris.md +++ b/pipeline/outputs/doris.md @@ -18,16 +18,16 @@ operational Doris service running in your environment. | `password` | Password for Doris access. | _none_ | | `database` | The target Doris database. | _none_ | | `table` | The target Doris table. | _none_ | -| `label_prefix` | Doris stream load label prefix, the final generated Label is {label_prefix}\_{timestamp}\_{uuid}. | `fluentbit` | +| `label_prefix` | Label prefix of Doris stream load, the final generated Label is {label_prefix}\_{timestamp}\_{uuid}. | `fluentbit` | | `time_key` | The name of the time key in the output record. | `date` | -| `header` | Doris stream load headers. Multiple headers can be set. See [Doris stream load](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual) for details. | _none_ | +| `header` | Headers of Doris stream load. Multiple headers can be set. See [Doris stream load](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual) for details. | _none_ | | `log_request` | Whether to output Doris Stream Load request and response metadata in logs for troubleshooting. | `true` | | `log_progress_interval` | The time interval in seconds to calculate and output the speed in the log. Set to 0 to disable this type of logging. | `10` | | `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | ### TLS / SSL -Doris output plugin supports TLS/SSL. See [TLS/SSL](../../administration/transport-security.md) +The Doris output plugin supports TLS/SSL. See [TLS/SSL](../../administration/transport-security.md) for more details about the supported properties and general configuration. ## Get started From 5d7e34b1794739cc1ebb18a4de61ddfbfdc14b56 Mon Sep 17 00:00:00 2001 From: composer <2789706336@qq.com> Date: Fri, 1 Nov 2024 01:21:56 +0800 Subject: [PATCH 5/5] feat: add yaml config Signed-off-by: composer <2789706336@qq.com> --- pipeline/outputs/doris.md | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/pipeline/outputs/doris.md b/pipeline/outputs/doris.md index af81acb2f..e8d19e675 100644 --- a/pipeline/outputs/doris.md +++ b/pipeline/outputs/doris.md @@ -56,6 +56,8 @@ fluent-bit -i cpu -t cpu -o doris \ In your main configuration file, append the following `Input` and `Output` sections. +{% tabs %} +{% tab title="fluent-bit.conf" %} ```python copy [INPUT] Name cpu @@ -70,5 +72,27 @@ In your main configuration file, append the following `Input` and `Output` secti password admin database d_fb table t_fb - columns date, cpu_p, log=cast(cpu_p as string) -``` \ No newline at end of file + header columns date, cpu_p, log=cast(cpu_p as string) +``` +{% endtab %} + +{% tab title="fluent-bit.yaml" %} +```yaml copy +pipeline: + inputs: + - name: cpu + tag: cpu + outputs: + - name: doris + match: '*' + host: 127.0.0.1 + port: 8030 + user: admin + password: admin + database: d_fb + table: t_fb + header: + - columns date, cpu_p, log=cast(cpu_p as string) +``` +{% endtab %} +{% endtabs %}