Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(libp2p): add SwarmBuilder #4120

Merged
merged 186 commits into from
Oct 10, 2023
Merged

Conversation

mxinden
Copy link
Member

@mxinden mxinden commented Jun 26, 2023

Description

Introduce the new libp2p::SwarmBuilder. Users should use the new libp2p::SwarmBuilder instead of the now deprecated libp2p::swarm::SwarmBuilder. See libp2p::SwarmBuilder docs on how to use the new builder.

Fixes #3657.
Fixes #3563.
Fixes #3179.

Notes & open questions

Change checklist

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • A changelog entry has been made in the appropriate crates

@mxinden
Copy link
Member Author

mxinden commented Jun 30, 2023

//CC @DougAnderson444 since this is related to https://github.com/DougAnderson444/libp2peasy.

libp2p/src/builder.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@thomaseizinger thomaseizinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work Max!

Before you continue supporting more transports, I'd appreciate some exploration on how this can be turned into a SwarmBuilder.

Specifically, we should make it possible to pass the identity in there but also generate a new one and then pass it to the Swarm as well.

libp2p/src/builder.rs Outdated Show resolved Hide resolved
libp2p/src/builder.rs Outdated Show resolved Hide resolved
libp2p/src/builder.rs Outdated Show resolved Hide resolved
- Allow providing or generating identity
- Use V1Lazy
- Make NetworkBehaviour constructor take relay Behaviour in case with_relay was called
- Rename to AuthenticatedMultiplexedTransport
@mxinden
Copy link
Member Author

mxinden commented Jul 1, 2023

Thank you for the input @thomaseizinger. What do you think of the most recent commit?

Copy link
Contributor

@thomaseizinger thomaseizinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking very nice already. I'd say that is worth a blog-post once we ship it :)

Some good content for TWIR!

libp2p/src/builder.rs Outdated Show resolved Hide resolved
libp2p/src/builder.rs Outdated Show resolved Hide resolved
libp2p/src/builder.rs Outdated Show resolved Hide resolved
libp2p/src/builder.rs Outdated Show resolved Hide resolved
libp2p/src/builder.rs Outdated Show resolved Hide resolved
libp2p/src/builder.rs Outdated Show resolved Hide resolved
libp2p/src/builder.rs Outdated Show resolved Hide resolved
mxinden added 12 commits July 8, 2023 12:06
I suggest shipping a minimal Builder at first. Users that need advanced features can use the
`with_other_transport` method or build a `Transport` without the builder.
- Make tls and noise configurable.
- Prevent accidental behavior change through feature change. E.g. enabling "noise" feature should
  not make ones transport automatically offer noise. Instead one needs to explicitly call
  `with_noise` and `without_noise`.
core/src/upgrade/map.rs Outdated Show resolved Hide resolved
@mxinden
Copy link
Member Author

mxinden commented Aug 3, 2023

In regards to the recent comments, happy to add shortcut methods to the various *Builder structs, e.g. with_dns on the OtherTransportBuilder.

@mxinden mxinden added the send-it label Oct 9, 2023
@mergify

This comment was marked as resolved.

@mergify

This comment was marked as resolved.

@mergify mergify bot merged commit d605255 into libp2p:master Oct 10, 2023
73 checks passed
mxinden added a commit to mxinden/rust-libp2p that referenced this pull request Oct 10, 2023
Support for QUIC draft 29 was removed with libp2p#4467.

libp2p#4120 reintroduced it as a faulty merge.

This commit removes it again.
.with_websocket(tls::Config::new, mplex::MplexConfig::default)
.with_websocket(noise::Config::new, mplex::MplexConfig::default)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once more, great catch @thomaseizinger.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interop tests made this one easy :)

@mxinden
Copy link
Member Author

mxinden commented Oct 10, 2023

Thank you @thomaseizinger for the last commits and the reviews in general. As always, very helpful.

mergify bot pushed a commit that referenced this pull request Oct 10, 2023
Support for QUIC draft 29 was removed with #4467.

#4120 reintroduced it as a faulty merge.

This commit removes it again.

Pull-Request: #4622.
mxinden added a commit to mxinden/rust-libp2p that referenced this pull request Oct 18, 2023
Version bump missed in libp2p#4120.
@mxinden mxinden mentioned this pull request Oct 18, 2023
4 tasks
mergify bot pushed a commit that referenced this pull request Oct 18, 2023
Version bump missed in #4120.

Pull-Request: #4683.
mergify bot pushed a commit that referenced this pull request Oct 25, 2023
This was an oversight in #4120, where we changed the creation of `Swarm`. Thus, all usages of the new `SwarmBuilder` did not actually hit this log line.

Pull-Request: #4671.
github-merge-queue bot pushed a commit to paritytech/polkadot-sdk that referenced this pull request Dec 12, 2024
# Description

Fixes #5996

https://github.com/libp2p/rust-libp2p/releases/tag/libp2p-v0.53.0
https://github.com/libp2p/rust-libp2p/blob/master/CHANGELOG.md

## Integration

Nothing special is needed, just note that `yamux_window_size` is no
longer applicable to libp2p (litep2p seems to still have it though).

## Review Notes

There are a few simplifications and improvements done in libp2p 0.53
regarding swarm interface, I'll list a few key/applicable here.

libp2p/rust-libp2p#4788 removed
`write_length_prefixed` function, so I inlined its code instead.

libp2p/rust-libp2p#4120 introduced new
`libp2p::SwarmBuilder` instead of now deprecated
`libp2p::swarm::SwarmBuilder`, the transition is straightforward and
quite ergonomic (can be seen in tests).

libp2p/rust-libp2p#4581 is the most annoying
change I have seen that basically makes many enums `#[non_exhaustive]`.
I mapped some, but those that couldn't be mapped I dealt with by
printing log messages once they are hit (the best solution I could come
up with, at least with stable Rust).

libp2p/rust-libp2p#4306 makes connection close
as soon as there are no handler using it, so I had to replace
`KeepAlive::Until` with an explicit future that flips internal boolean
after timeout, achieving the old behavior, though it should ideally be
removed completely at some point.

`yamux_window_size` is no longer used by libp2p thanks to
libp2p/rust-libp2p#4970 and generally Yamux
should have a higher performance now.

I have resolved and cleaned up all deprecations related to libp2p except
`BandwidthSinks`. Libp2p deprecated it (though it is still present in
0.54.1, which is why I didn't handle it just yet). Ideally Substrate
would finally [switch to the official Prometheus
client](paritytech/substrate#12699), in which
case we'd get metrics for free. Otherwise a bit of code will need to be
copy-pasted to maintain current behavior with `BandwidthSinks` gone,
which I left a TODO about.

The biggest change in 0.54.0 is
libp2p/rust-libp2p#4568 that changed transport
APIs and enabled unconditional potential port reuse, which can lead to
very confusing errors if running two Substrate nodes on the same machine
without changing listening port explicitly.

Overall nothing scary here, but testing is always appreciated.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.

---

Polkadot Address: 1vSxzbyz2cJREAuVWjhXUT1ds8vBzoxn2w4asNpusQKwjJd

---------

Co-authored-by: Dmitry Markin <[email protected]>
github-merge-queue bot pushed a commit to paritytech/polkadot-sdk that referenced this pull request Dec 12, 2024
# Description

Fixes #5996

https://github.com/libp2p/rust-libp2p/releases/tag/libp2p-v0.53.0
https://github.com/libp2p/rust-libp2p/blob/master/CHANGELOG.md

## Integration

Nothing special is needed, just note that `yamux_window_size` is no
longer applicable to libp2p (litep2p seems to still have it though).

## Review Notes

There are a few simplifications and improvements done in libp2p 0.53
regarding swarm interface, I'll list a few key/applicable here.

libp2p/rust-libp2p#4788 removed
`write_length_prefixed` function, so I inlined its code instead.

libp2p/rust-libp2p#4120 introduced new
`libp2p::SwarmBuilder` instead of now deprecated
`libp2p::swarm::SwarmBuilder`, the transition is straightforward and
quite ergonomic (can be seen in tests).

libp2p/rust-libp2p#4581 is the most annoying
change I have seen that basically makes many enums `#[non_exhaustive]`.
I mapped some, but those that couldn't be mapped I dealt with by
printing log messages once they are hit (the best solution I could come
up with, at least with stable Rust).

libp2p/rust-libp2p#4306 makes connection close
as soon as there are no handler using it, so I had to replace
`KeepAlive::Until` with an explicit future that flips internal boolean
after timeout, achieving the old behavior, though it should ideally be
removed completely at some point.

`yamux_window_size` is no longer used by libp2p thanks to
libp2p/rust-libp2p#4970 and generally Yamux
should have a higher performance now.

I have resolved and cleaned up all deprecations related to libp2p except
`BandwidthSinks`. Libp2p deprecated it (though it is still present in
0.54.1, which is why I didn't handle it just yet). Ideally Substrate
would finally [switch to the official Prometheus
client](paritytech/substrate#12699), in which
case we'd get metrics for free. Otherwise a bit of code will need to be
copy-pasted to maintain current behavior with `BandwidthSinks` gone,
which I left a TODO about.

The biggest change in 0.54.0 is
libp2p/rust-libp2p#4568 that changed transport
APIs and enabled unconditional potential port reuse, which can lead to
very confusing errors if running two Substrate nodes on the same machine
without changing listening port explicitly.

Overall nothing scary here, but testing is always appreciated.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.

---

Polkadot Address: 1vSxzbyz2cJREAuVWjhXUT1ds8vBzoxn2w4asNpusQKwjJd

---------

Co-authored-by: Dmitry Markin <[email protected]>
github-merge-queue bot pushed a commit to paritytech/polkadot-sdk that referenced this pull request Dec 12, 2024
# Description

Fixes #5996

https://github.com/libp2p/rust-libp2p/releases/tag/libp2p-v0.53.0
https://github.com/libp2p/rust-libp2p/blob/master/CHANGELOG.md

## Integration

Nothing special is needed, just note that `yamux_window_size` is no
longer applicable to libp2p (litep2p seems to still have it though).

## Review Notes

There are a few simplifications and improvements done in libp2p 0.53
regarding swarm interface, I'll list a few key/applicable here.

libp2p/rust-libp2p#4788 removed
`write_length_prefixed` function, so I inlined its code instead.

libp2p/rust-libp2p#4120 introduced new
`libp2p::SwarmBuilder` instead of now deprecated
`libp2p::swarm::SwarmBuilder`, the transition is straightforward and
quite ergonomic (can be seen in tests).

libp2p/rust-libp2p#4581 is the most annoying
change I have seen that basically makes many enums `#[non_exhaustive]`.
I mapped some, but those that couldn't be mapped I dealt with by
printing log messages once they are hit (the best solution I could come
up with, at least with stable Rust).

libp2p/rust-libp2p#4306 makes connection close
as soon as there are no handler using it, so I had to replace
`KeepAlive::Until` with an explicit future that flips internal boolean
after timeout, achieving the old behavior, though it should ideally be
removed completely at some point.

`yamux_window_size` is no longer used by libp2p thanks to
libp2p/rust-libp2p#4970 and generally Yamux
should have a higher performance now.

I have resolved and cleaned up all deprecations related to libp2p except
`BandwidthSinks`. Libp2p deprecated it (though it is still present in
0.54.1, which is why I didn't handle it just yet). Ideally Substrate
would finally [switch to the official Prometheus
client](paritytech/substrate#12699), in which
case we'd get metrics for free. Otherwise a bit of code will need to be
copy-pasted to maintain current behavior with `BandwidthSinks` gone,
which I left a TODO about.

The biggest change in 0.54.0 is
libp2p/rust-libp2p#4568 that changed transport
APIs and enabled unconditional potential port reuse, which can lead to
very confusing errors if running two Substrate nodes on the same machine
without changing listening port explicitly.

Overall nothing scary here, but testing is always appreciated.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.

---

Polkadot Address: 1vSxzbyz2cJREAuVWjhXUT1ds8vBzoxn2w4asNpusQKwjJd

---------

Co-authored-by: Dmitry Markin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants