feat(new source): New fluent source #7548

jszwedko · 2021-05-21T19:46:13Z

New source for consuming data from fluentd and fluent-bit via their native forward protocol.

This implementation supports the various formats that the specification allows for but does not implement the authentication protocol (#7532) or the acknowledgment (#7533) parts of the protocol. I left these for follow-up based on user demand given it is a significant divergence from the existing TCP source which just consumes data rather than request/response.

Aside from the source implementation, the other significant change here is extraction of some bits of the docker_logs source into a shared docker module to use it in the fluent source integration tests. The integration tests are more similar to the docker_logs integration tests than the other component integration tests in that they each run a separate docker container rather than just standing up one long running one before running the tests as these tests need to run fluentd and fluent-bit with different configurations.

Apologies, I didn't break up the early commits into logical changes, but hopefully the PR itself is reviewable.

Closes #419

Signed-off-by: Jesse Szwedko <[email protected]>

jszwedko · 2021-05-21T19:49:05Z

Cargo.toml

@@ -343,6 +349,8 @@ wasm = ["lucet-runtime", "lucet-wasi", "lucetc", "vector-wasm"]
 # transforms and sinks should depend on this feature.
 kubernetes = ["k8s-openapi", "evmap"]

+docker = ["bollard", "dirs-next"]


New feature to extract some of the bits from the docker_logs source to share with the fluent integration tests.

jszwedko · 2021-05-21T19:49:37Z

src/docker.rs

@@ -0,0 +1,105 @@
+use bollard::{errors::Error as DockerError, Docker, API_DEFAULT_VERSION};


This file contains bits extracted from docker_logs.

jszwedko · 2021-05-21T19:50:13Z

src/sources/docker_logs.rs

@@ -1,6 +1,7 @@
 use super::util::MultilineConfig;
 use crate::{
    config::{log_schema, DataType, SourceConfig, SourceContext, SourceDescription},
+    docker::{docker, DockerTlsConfig},


Changes to this file are just removals from moving bits into src/docker.rs.

jszwedko · 2021-05-21T19:51:01Z

src/sources/util/tcp.rs

@@ -53,17 +53,17 @@ async fn make_listener(
    }
 }
 pub trait IsErrorFatal {
-    fn is_error_fatal() -> bool;
+    fn is_error_fatal(&self) -> bool;


Started injecting self here as the fluent source decides which errors are fatal based on whether they are i/o or decode errors (only i/o errors are fatal).

jszwedko · 2021-05-21T19:51:40Z

tests/config.rs

@@ -760,11 +760,7 @@ async fn parses_sink_full_es_basic_auth() {
    .unwrap();
 }

-#[cfg(all(
-    feature = "docker",


Not sure why this feature constraint was here as the feature didn't exist, but it flagged this test as failing to compile when I reintroduced it.

spencergilbert

Walked through this with @jszwedko and seems good, but definitely should have someone more rust savvy give it a once-over :)

bruceg

Nothing obviously bad jumped out at me, just a couple of minor tweaks. The real test is integrating with fluentd and fluent-bit, which you are testing.

Can fluentd run in podman as well as docker? If so, is there any option to allow for testing under podman?

bruceg · 2021-05-27T23:05:51Z

src/sources/fluent.rs

+        if log.get(log_schema().host_key()).is_none() {
+            log.insert(log_schema().host_key(), host);
+        }


Is this better with the entry interface?

Suggested change

if log.get(log_schema().host_key()).is_none() {

log.insert(log_schema().host_key(), host);

}

log.entry(log_schema().host_key()).or_insert(host);

👍 I always forget about that interface.

Actually, entry is a private function that takes a Lookup and returns a Result<Entry, _> which seemed harder to work with then what I have here. I ended up just swapping the get + is_none with contains().

bruceg · 2021-05-27T23:18:41Z

src/sources/fluent.rs

+        if !self.unread_frames.is_empty() {
+            return Ok(self.unread_frames.pop_front());
+        }


Don't need to check is_empty first:

Suggested change

if !self.unread_frames.is_empty() {

return Ok(self.unread_frames.pop_front());

}

if let Some(frame) = self.unread_frames.pop_front() {

return Ok(frame);

}

Signed-off-by: Jesse Szwedko <[email protected]>

jszwedko · 2021-05-28T17:59:06Z

Can fluentd run in podman as well as docker? If so, is there any option to allow for testing under podman?

I took a brief look at this and it seems like it could possibly work but I wasn't able to get podman running on my Linux system (I ran into this issue: https://stackoverflow.com/questions/65775075/problems-running-podman-in-ubuntu-20-04 but wasn't able to resolve it).

I was able to run podman system service -t 0 unix:///tmp/podman.sock to run a docker compatible API and then export DOCKER_HOST to point to this though and things appeared to be working (minus the issue I ran into with the uid/gids). I do think we'll need this unreleased change in podman: containers/podman#9972 to configure host.docker.internal to point to the host system (needed since we can't use 127.0.0.1 to indicate vector on OSX given that docker is running in a VM).

Would you be able to try it out?

jszwedko · 2021-05-28T18:00:11Z

Kicked off integration test to make sure it passes in CI: https://github.com/timberio/vector/actions/runs/886440980

Signed-off-by: Jesse Szwedko <[email protected]>

They need to bind to 0.0.0.0 so that they also bind to the docker bridge so that fluentd / fluentbit, running in docker, can send traffic to it. Signed-off-by: Jesse Szwedko <[email protected]>

Missed this before I merged the #7548 Signed-off-by: Jesse Szwedko <[email protected]>

jszwedko requested review from bruceg, pablosichert, spencergilbert and a team May 21, 2021 19:46

jszwedko added 13 commits May 21, 2021 15:47

wip

7094c85

Signed-off-by: Jesse Szwedko <[email protected]>

wip

f5cab71

Signed-off-by: Jesse Szwedko <[email protected]>

wip

0491304

Signed-off-by: Jesse Szwedko <[email protected]>

wip

28bc8de

Signed-off-by: Jesse Szwedko <[email protected]>

wip

2c1f4b1

Signed-off-by: Jesse Szwedko <[email protected]>

wip

0637ad5

Signed-off-by: Jesse Szwedko <[email protected]>

wip

b0f4063

Signed-off-by: Jesse Szwedko <[email protected]>

wip

2286b58

Signed-off-by: Jesse Szwedko <[email protected]>

Add docs and rename fluent_tag to tag

5d2540d

Signed-off-by: Jesse Szwedko <[email protected]>

Run fluent integration tests in CI

66c23a8

Signed-off-by: Jesse Szwedko <[email protected]>

Support heartbeat messages and small edits

51b4994

Signed-off-by: Jesse Szwedko <[email protected]>

Put TcpSource build_event back

d1a8cc0

Signed-off-by: Jesse Szwedko <[email protected]>

Add decoder tests

d5e8aab

Signed-off-by: Jesse Szwedko <[email protected]>

jszwedko force-pushed the add-fluent-source branch from 6ce5e82 to d5e8aab Compare May 21, 2021 19:47

jszwedko removed request for a team May 21, 2021 19:47

jszwedko commented May 21, 2021

View reviewed changes

spencergilbert approved these changes May 25, 2021

View reviewed changes

bruceg approved these changes May 27, 2021

View reviewed changes

jszwedko added 2 commits May 28, 2021 13:16

PR feedback

2e5deeb

Signed-off-by: Jesse Szwedko <[email protected]>

Add additional extra host for gateway

5c1f6c6

Signed-off-by: Jesse Szwedko <[email protected]>

jszwedko added 3 commits May 28, 2021 18:23

Meant to readd these

dfe0463

Signed-off-by: Jesse Szwedko <[email protected]>

Fix port binding for fluent tests

39c4678

They need to bind to 0.0.0.0 so that they also bind to the docker bridge so that fluentd / fluentbit, running in docker, can send traffic to it. Signed-off-by: Jesse Szwedko <[email protected]>

Merge remote-tracking branch 'origin/master' into add-fluent-source

7086abb

jszwedko merged commit 1408491 into master Jun 7, 2021

jszwedko deleted the add-fluent-source branch June 7, 2021 22:35

jszwedko added a commit that referenced this pull request Jun 8, 2021

chore: clippy fluent source

c19b52c

Missed this before I merged the #7548 Signed-off-by: Jesse Szwedko <[email protected]>

jszwedko mentioned this pull request Jun 8, 2021

chore: clippy fluent source #7786

Closed

jszwedko mentioned this pull request Jun 17, 2021

New fluent sink #534

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(new source): New fluent source #7548

feat(new source): New fluent source #7548

jszwedko commented May 21, 2021

jszwedko May 21, 2021

jszwedko May 21, 2021

jszwedko May 21, 2021

jszwedko May 21, 2021

jszwedko May 21, 2021

spencergilbert left a comment

bruceg left a comment

bruceg May 27, 2021

jszwedko May 28, 2021

jszwedko May 28, 2021 •

edited

Loading

bruceg May 27, 2021

jszwedko commented May 28, 2021 •

edited

Loading

jszwedko commented May 28, 2021

		@@ -0,0 +1,105 @@
		use bollard::{errors::Error as DockerError, Docker, API_DEFAULT_VERSION};

feat(new source): New fluent source #7548

feat(new source): New fluent source #7548

Conversation

jszwedko commented May 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spencergilbert left a comment

Choose a reason for hiding this comment

bruceg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jszwedko May 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jszwedko commented May 28, 2021 • edited Loading

jszwedko commented May 28, 2021

jszwedko May 28, 2021 •

edited

Loading

jszwedko commented May 28, 2021 •

edited

Loading