-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(new source): New fluent source #7548
Merged
Merged
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
7094c85
wip
jszwedko f5cab71
wip
jszwedko 0491304
wip
jszwedko 28bc8de
wip
jszwedko 2c1f4b1
wip
jszwedko 0637ad5
wip
jszwedko b0f4063
wip
jszwedko 2286b58
wip
jszwedko 5d2540d
Add docs and rename fluent_tag to tag
jszwedko 66c23a8
Run fluent integration tests in CI
jszwedko 51b4994
Support heartbeat messages and small edits
jszwedko d1a8cc0
Put TcpSource build_event back
jszwedko d5e8aab
Add decoder tests
jszwedko 2e5deeb
PR feedback
jszwedko 5c1f6c6
Add additional extra host for gateway
jszwedko dfe0463
Meant to readd these
jszwedko 39c4678
Fix port binding for fluent tests
jszwedko 7086abb
Merge remote-tracking branch 'origin/master' into add-fluent-source
jszwedko File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -134,6 +134,25 @@ jobs: | |
| - run: make slim-builds | ||
| - run: make test-integration-elasticsearch | ||
|
|
||
| test-integration-fluent: | ||
| name: Integration - Linux, Fluent | ||
| runs-on: ubuntu-20.04 | ||
| steps: | ||
| - uses: actions/[email protected] | ||
| - run: make ci-sweep | ||
| - uses: actions/[email protected] | ||
| with: | ||
| path: | | ||
| ~/.cargo/registry | ||
| ~/.cargo/git | ||
| key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }} | ||
| - run: sudo bash scripts/environment/bootstrap-ubuntu-20.04.sh | ||
| - run: bash scripts/environment/prepare.sh | ||
| - run: echo "::add-matcher::.github/matchers/rust.json" | ||
| - run: make slim-builds | ||
| - run: make test-integration-fluent | ||
|
|
||
|
|
||
| test-integration-gcp: | ||
| name: Integration - Linux, GCP | ||
| runs-on: ubuntu-20.04 | ||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,223 @@ | ||
| package metadata | ||
|
|
||
| components: sources: fluent: { | ||
| _port: 24224 | ||
|
|
||
| title: "Fluent" | ||
|
|
||
| classes: { | ||
| commonly_used: true | ||
| delivery: "best_effort" | ||
| deployment_roles: ["sidecar", "aggregator"] | ||
| development: "beta" | ||
| egress_method: "stream" | ||
| stateful: false | ||
| } | ||
|
|
||
| features: { | ||
| receive: { | ||
| from: { | ||
| service: services.fluent | ||
|
|
||
| interface: socket: { | ||
| api: { | ||
| title: "Fluent" | ||
| url: urls.fluent | ||
| } | ||
| direction: "incoming" | ||
| port: _port | ||
| protocols: ["tcp"] | ||
| ssl: "optional" | ||
| } | ||
| } | ||
| receive_buffer_bytes: { | ||
| enabled: true | ||
| } | ||
| keepalive: enabled: true | ||
| tls: sources.socket.features.receive.tls | ||
| } | ||
| multiline: enabled: false | ||
| } | ||
|
|
||
| support: { | ||
| targets: { | ||
| "aarch64-unknown-linux-gnu": true | ||
| "aarch64-unknown-linux-musl": true | ||
| "armv7-unknown-linux-gnueabihf": true | ||
| "armv7-unknown-linux-musleabihf": true | ||
| "x86_64-apple-darwin": true | ||
| "x86_64-pc-windows-msv": true | ||
| "x86_64-unknown-linux-gnu": true | ||
| "x86_64-unknown-linux-musl": true | ||
| } | ||
| requirements: [] | ||
| warnings: [] | ||
| notices: [] | ||
| } | ||
|
|
||
| installation: { | ||
| platform_name: null | ||
| } | ||
|
|
||
| configuration: { | ||
| address: { | ||
| description: "The address to listen for TCP connections on." | ||
| required: true | ||
| warnings: [] | ||
| type: string: { | ||
| examples: ["0.0.0.0:\(_port)"] | ||
| syntax: "literal" | ||
| } | ||
| } | ||
| } | ||
|
|
||
| output: logs: line: { | ||
| description: "A Fluent message" | ||
| fields: { | ||
| host: { | ||
| description: "The IP address the fluent message was sent from." | ||
| required: true | ||
| type: string: { | ||
| examples: ["127.0.0.1"] | ||
| syntax: "literal" | ||
| } | ||
| } | ||
| timestamp: { | ||
| description: "The timestamp extracted from the fluent message." | ||
| required: true | ||
| type: timestamp: {} | ||
| } | ||
| tag: { | ||
| description: "The tag from the fluent message." | ||
| required: true | ||
| type: string: { | ||
| examples: ["dummy.0"] | ||
| syntax: "literal" | ||
| } | ||
| } | ||
| "*": { | ||
| description: "In addition to the defined fields, all fields from the fluent message are inserted as root level fields." | ||
| required: true | ||
| type: string: { | ||
| examples: ["hello world"] | ||
| syntax: "literal" | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| examples: [ | ||
| { | ||
| title: "Dummy message from fluentd" | ||
| configuration: {} | ||
| input: """ | ||
| ```text | ||
| 2021-05-20 16:23:03.021497000 -0400 dummy: {"message":"dummy"} | ||
| ``` | ||
|
|
||
| (this is the fluentd stdout encoding of the dummy message) | ||
| """ | ||
| output: log: { | ||
| host: _values.remote_host | ||
| timestamp: "2021-05-20T20:23:03.021497Z" | ||
| tag: "dummy" | ||
| message: "dummy" | ||
| } | ||
| }, | ||
| { | ||
| title: "Dummy message from fluent-bit" | ||
| configuration: {} | ||
| input: """ | ||
| ```text | ||
| dummy.0: [1621541848.161827000, {"message"=>"dummy"}] | ||
| ``` | ||
|
|
||
| (this is the fluent-bit stdout encoding of the dummy message) | ||
| """ | ||
| output: log: { | ||
| host: _values.remote_host | ||
| timestamp: "2020-05-20T20:17:28.161827Z" | ||
| tag: "dummy.0" | ||
| message: "dummy" | ||
| } | ||
| }, | ||
| ] | ||
|
|
||
| how_it_works: { | ||
| aggregator: { | ||
| title: "Sending data from fluent agents to Vector aggregators" | ||
| body: """ | ||
| If you are already running fluent agents (Fluentd or Fluent Bit) in your infrastructure, this source can | ||
| make it easy to start getting that data into Vector. | ||
| """ | ||
| } | ||
|
|
||
| fluentd_configuartion: { | ||
| title: "Fluentd configuration" | ||
| body: """ | ||
| To configure Fluentd to forward to a Vector instance, you can use the following output configuration: | ||
|
|
||
| ```text | ||
| <match *> | ||
| @type forward | ||
| <server> | ||
| # update these to point to your vector instance | ||
| name local | ||
| host 127.0.0.1 | ||
| port 24224 | ||
| </server> | ||
| compress gzip | ||
| </match> | ||
| ``` | ||
| """ | ||
| } | ||
|
|
||
| fluentbit_configuration: { | ||
| title: "Fluent Bit configuration" | ||
| body: """ | ||
| To configure Fluent Bit to forward to a Vector instance, you can use the following output configuration: | ||
|
|
||
| ```text | ||
| [OUTPUT] | ||
| Name forward | ||
| Match * | ||
| # update these to point to your vector instance | ||
| Host 127.0.0.1 | ||
| Port 24224 | ||
| ``` | ||
| """ | ||
| } | ||
|
|
||
| secure_mode: { | ||
| title: "Secure forward mode support" | ||
| body: """ | ||
| The `fluent` source currently supports using TLS, but does not support the authentication part of the | ||
| Fluent protocol including: | ||
|
|
||
| - Shared key | ||
| - Username and password | ||
|
|
||
| And so these options of the secure forward output plugins for Fluent and Fluent Bit cannot be used. | ||
|
|
||
| If you would find this useful, [please let us know](\(urls.vector_repo)/issues/7532). | ||
| """ | ||
| } | ||
|
|
||
| acking: { | ||
| title: "Acknowledgement support" | ||
| body: """ | ||
| The `fluent` source currently does not support the acknowledgement parts of the Fluent protocol and so | ||
| the `require_ack_response` option forward output plugins for Fluent and Fluent Bit cannot be used. | ||
|
|
||
| If you would find this useful, [please let us know](\(urls.vector_repo)/issues/7533). | ||
| """ | ||
| } | ||
| } | ||
|
|
||
| telemetry: metrics: { | ||
| events_in_total: components.sources.internal_metrics.output.metrics.events_in_total | ||
| decode_errors_total: components.sources.internal_metrics.output.metrics.decode_errors_total | ||
| processed_bytes_total: components.sources.internal_metrics.output.metrics.processed_bytes_total | ||
| processed_events_total: components.sources.internal_metrics.output.metrics.processed_events_total | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
New feature to extract some of the bits from the
docker_logssource to share with thefluentintegration tests.