Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sources): multicast udp socket support #22099

Open
wants to merge 22 commits into
base: master
Choose a base branch
from

Conversation

jorgehermo9
Copy link
Contributor

@jorgehermo9 jorgehermo9 commented Dec 31, 2024

Closes #5732

This PR is still in draft. I have a few pending TODOs and also missing tests to propertly check this. Although, the happy path is working.

In order to test this, use this vector config:

[sources.multicast_udp]
type = "socket"
mode = "udp"
address = "0.0.0.0:4242"
multicast_groups = ["224.0.0.2"]


[sinks.console]
type = "console"
inputs = ["multicast_udp"]
encoding.codec = "json"

and with this command

echo "hello" | nc 224.0.0.2 4242 -u

you can see logs in vector.
image

I would like to receive some feedback about how the configuration of this setting should look like.

Also, note that IPv6 is not supported. We can work on that, but maybe it is not worth it if no one request that.

@nomalord take a look into this please, It would be great if you can build the binary from this branch and test if it works in your systems.

@dalesample as you were the first requester of this I also ping you, just in case (although this issue was created 4 years ago)

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
@github-actions github-actions bot added the domain: sources Anything related to the Vector's sources label Dec 31, 2024
@nomalord
Copy link

This looks awesome🤯
As it's new years I won't be able to check our usecase until Thursday...
I will say though, that this looks like it answers our usecase exactly🙏🙏🙏🙏🙏🙏

@jorgehermo9 jorgehermo9 changed the title feat: multicast udp socket support feat(sources): multicast udp socket support Jan 2, 2025
@jorgehermo9
Copy link
Contributor Author

Hi @nomalord perhaps you could test this?

@nomalord
Copy link

nomalord commented Jan 9, 2025

I don't really have experience with rust, but I'll try to compile the feature branch and update you🥹

@pront
Copy link
Member

pront commented Jan 9, 2025

I don't really have experience with rust, but I'll try to compile the feature branch and update you🥹

Maybe this is helpful to you:

# cd /path/to/vector/repo
# brew install gh 
gh pr checkout 22099
cargo run -- --config /path/to/your/config.yaml

@jorgehermo9
Copy link
Contributor Author

Thank you for popping in @pront 😃. I have left to include some tests to validate this PR and also add documentation & changelog. I will update this as soon as I can and mark the pr as non-draft then

@github-actions github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Jan 19, 2025
@jorgehermo9 jorgehermo9 marked this pull request as ready for review January 19, 2025 23:01
@jorgehermo9 jorgehermo9 requested review from a team as code owners January 19, 2025 23:02
@jorgehermo9
Copy link
Contributor Author

Hi @pront, this PR is now ready to review, just notifying, no hurries!

@pront
Copy link
Member

pront commented Jan 22, 2025

Hi @pront, this PR is now ready to review, just notifying, no hurries!

Thank you 🚀 I will get to this soon.

Copy link
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @jorgehermo9, great work. Nothing major came out, mostly nits.

lib/vector-config/src/stdlib.rs Show resolved Hide resolved
src/sources/socket/mod.rs Outdated Show resolved Hide resolved
src/sources/socket/mod.rs Outdated Show resolved Hide resolved
src/sources/socket/mod.rs Outdated Show resolved Hide resolved
src/sources/socket/udp.rs Outdated Show resolved Hide resolved
src/sources/socket/udp.rs Show resolved Hide resolved
src/test_util/mod.rs Show resolved Hide resolved
src/sources/socket/udp.rs Show resolved Hide resolved
src/sources/socket/udp.rs Outdated Show resolved Hide resolved
@jorgehermo9
Copy link
Contributor Author

Thanks for the review @pront! I will happily address the comments as I prefer to fix those nits before merging!

socket
.join_multicast_v4(group_addr, interface)
.map_err(|error| {
emit!(SocketMulticastGroupJoinError {
Copy link
Contributor Author

@jorgehermo9 jorgehermo9 Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As suggested https://github.com/vectordotdev/vector/pull/22099/files#r1927304213

This is logged as
image

does it look ok?


#[tokio::test]
async fn udp_invalid_multicast_group() {
assert_source_error(&COMPONENT_ERROR_TAGS, async {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refactored this a little in dc09eb5. Didn't know that assert_source_error existed. Is it ok to check that only the tag error_type is updated? (COMPONENT_ERROR_TAGS only contains that tag)

we are using a bit more tags in that error
image

pub interface: Ipv4Addr,
}

impl<E: std::fmt::Display> InternalEvent for SocketMulticastGroupJoinError<E> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inspired from the SocketBindError implementation

@@ -135,6 +137,37 @@ impl<E: std::fmt::Display> InternalEvent for SocketBindError<E> {
}
}

#[derive(Debug)]
pub struct SocketMulticastGroupJoinError<E> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have doubts about this naming. SocketMulticastJoinError, UdpSocketMulticastJoinError, UdpSocketMulticastGroupJoinError....?

This error will be created only from UDP mode, as the multicast concept only exists UDP.

Comment on lines +171 to +172
"group_addr" => group_addr,
"interface" => interface,
Copy link
Contributor Author

@jorgehermo9 jorgehermo9 Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it ok to add those two variables (group_addr and interface) to counter tags? Or should only be included in the error! log?

#[derive(Debug)]
pub struct SocketMulticastGroupJoinError<E> {
pub error: E,
pub group_addr: Ipv4Addr,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the group_addr and interface because they're the arguments required for join_multicast_v4

.join_multicast_v4(group_addr, interface)

error = %self.error,
error_code = "socket_multicast_group_join",
error_type = error_type::IO_FAILED,
stage = error_stage::RECEIVING,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error_stage receiving? I found that stage weird in this case. I think INITIALIZING or so would fit more in this case... but I see that error_stage::RECEIVING is used also for socket binding error and etc.. Just noticed this and wanted to raise it, not for addressing in this PR though

@jorgehermo9
Copy link
Contributor Author

Hi @pront. Sorry for the delay, had no time these past weeks. Addressed all your comments and left a few doubts about the new changes. Feel free to resolve any conversation, just wanted to raise some things so you are aware of it.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: external docs Anything related to Vector's external, public documentation domain: sources Anything related to the Vector's sources
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support multicast for UDP
4 participants