Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_doris: doc for new doris out plugin #1483

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitbook.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ redirects:
output/chronicle: ./pipeline/outputs/chronicle.md
output/cloudwatch: ./pipeline/outputs/cloudwatch.md
output/datadog: ./pipeline/outputs/datadog.md
output/doris: ./pipeline/outputs/doris.md
output/es: ./pipeline/outputs/elasticsearch.md
output/fabric: ./pipeline/outputs/azure_kusto.md
output/file: ./pipeline/outputs/file.md
Expand Down
1 change: 1 addition & 0 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@
* [Azure Logs Ingestion API](pipeline/outputs/azure_logs_ingestion.md)
* [Counter](pipeline/outputs/counter.md)
* [Datadog](pipeline/outputs/datadog.md)
* [Doris](pipeline/outputs/doris.md)
* [Dynatrace](pipeline/outputs/dynatrace.md)
* [Elasticsearch](pipeline/outputs/elasticsearch.md)
* [File](pipeline/outputs/file.md)
Expand Down
1 change: 1 addition & 0 deletions administration/configuring-fluent-bit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ Outputs
bigquery Send events to BigQuery via streaming insert
counter Records counter
datadog Send events to DataDog HTTP Event Collector
doris Apache Doris
es Elasticsearch
exit Exit after a number of flushes (test purposes)
file Generate log file
Expand Down
1 change: 1 addition & 0 deletions administration/transport-security.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ The following **output** plugins can take advantage of the TLS feature:
* [Azure Logs Ingestion API](../pipeline/outputs/azure_logs_ingestion.md)
* [BigQuery](../pipeline/outputs/bigquery.md)
* [Datadog](../pipeline/outputs/datadog.md)
* [Doris](../pipeline/outputs/doris.md)
* [Elasticsearch](../pipeline/outputs/elasticsearch.md)
* [Forward](../pipeline/outputs/forward.md)
* [GELF](../pipeline/outputs/gelf.md)
Expand Down
1 change: 1 addition & 0 deletions installation/sources/build-and-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,7 @@ The _output plugins_ gives the capacity to flush the information to some externa
| [FLB\_OUT\_COUNTER](../../pipeline/outputs/counter.md) | Enable Counter output plugin | On |
| [FLB\_OUT\_CLOUDWATCH\_LOGS](../../pipeline/outputs/cloudwatch.md) | Enable Amazon CloudWatch output plugin | On |
| [FLB\_OUT\_DATADOG](../../pipeline/outputs/datadog.md) | Enable Datadog output plugin | On |
| [FLB\_OUT|_DORIS](../../pipeline/outputs/doris.md) | Enable Apache Doris output plugin | On |
| [FLB\_OUT\_ES](../../pipeline/outputs/elasticsearch.md) | Enable [Elastic Search](http://www.elastic.co) output plugin | On |
| [FLB\_OUT\_FILE](../../pipeline/outputs/file.md) | Enable File output plugin | On |
| [FLB\_OUT\_KINESIS\_FIREHOSE](../../pipeline/outputs/firehose.md) | Enable Amazon Kinesis Data Firehose output plugin | On |
Expand Down
98 changes: 98 additions & 0 deletions pipeline/outputs/doris.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
---
description: Send logs to Apache Doris
---

# Apache Doris

The **doris** output plugin lets you ingest your records into an
[Apache Doris](https://doris.apache.org) database. To use this plugin, you must have an
operational Doris service running in your environment.

## Configuration Parameters

| Key | Description | Default |
| :--- | :--- | :--- |
| `host` | HTTP address of the target Doris frontend (fe) or frontend (be). | `127.0.0.1` |
| `port` | HTTP port of the target Doris frontend (fe) or frontend (be). | `8030` |
| `user` | Username for Doris access. | _none_ |
| `password` | Password for Doris access. | _none_ |
| `database` | The target Doris database. | _none_ |
| `table` | The target Doris table. | _none_ |
| `label_prefix` | Label prefix of Doris stream load, the final generated Label is {label_prefix}\_{timestamp}\_{uuid}. | `fluentbit` |
| `time_key` | The name of the time key in the output record. | `date` |
| `header` | Headers of Doris stream load. Multiple headers can be set. See [Doris stream load](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual) for details. | _none_ |
| `log_request` | Whether to output Doris Stream Load request and response metadata in logs for troubleshooting. | `true` |
| `log_progress_interval` | The time interval in seconds to calculate and output the speed in the log. Set to 0 to disable this type of logging. | `10` |
| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` |

### TLS / SSL

The Doris output plugin supports TLS/SSL. See [TLS/SSL](../../administration/transport-security.md)
for more details about the supported properties and general configuration.

## Get started

To insert records into a Doris database, run the plugin from the command line or define a configuration file:

### Command Line

The Doris plugin can read the parameters from the command through the `-p` argument,

as shown in the following example:

```shell copy
fluent-bit -i cpu -t cpu -o doris \
-m '*' \
-p host=127.0.0.1 \
-p port=8030 \
-p user=admin \
-p password=admin \
-p database=d_fb \
-p table=t_fb \
-p header='columns date, cpu_p, log=cast(cpu_p as string)'
```

### Configuration File

In your main configuration file, append the following `Input` and `Output` sections.

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python copy
joker-star-l marked this conversation as resolved.
Show resolved Hide resolved
[INPUT]
Name cpu
Tag cpu

[OUTPUT]
name doris
match *
host 127.0.0.1
port 8030
user admin
password admin
database d_fb
table t_fb
header columns date, cpu_p, log=cast(cpu_p as string)
```
{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml copy
pipeline:
inputs:
- name: cpu
tag: cpu
outputs:
- name: doris
match: '*'
host: 127.0.0.1
port: 8030
user: admin
password: admin
database: d_fb
table: t_fb
header:
- columns date, cpu_p, log=cast(cpu_p as string)
```
{% endtab %}
{% endtabs %}
1 change: 1 addition & 0 deletions vale-styles/FluentBit/Spelling-exceptions.txt
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ deprovisioning
deprovisions
Devo
DogStatsD
Doris
downsample
downsampled
downsamples
Expand Down