From 1ad325cf949e2fe22e61da1ef172273ca9f6b26c Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Tue, 27 Sep 2022 06:32:43 -0400 Subject: [PATCH 01/11] bring protocol page into a better state --- content/concepts/protocols.md | 269 +++++++++++++--------------------- 1 file changed, 102 insertions(+), 167 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index 7cc924b6..fa708fb6 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -1,43 +1,38 @@ --- -title: Protocols -weight: 3 +title: Introduction to libp2p Protocols +menuTitle: Protocols +weight: 1 --- -There are protocols everywhere you look when you're writing network applications, and libp2p is -especially thick with them. - -The kind of protocols this article is concerned with are the ones built with libp2p itself, -using the core libp2p abstractions like [transport](/concepts/transport), [peer identity](/concepts/peers#peer-id/), [addressing](/concepts/addressing/), and so on. - -Throughout this article, we'll call this kind of protocol that is built with libp2p -a **libp2p protocol**, but you may also see them referred to as "wire protocols" or "application protocols". - -These are the protocols that define your application and provide its core functionality. - -This article will walk through some of the key [defining features of a libp2p protocol](#what-is-a-libp2p-protocol), give an overview of the [protocol negotiation process](#protocol-negotiation), and outline some of the [core libp2p protocols](#core-libp2p-protocols) that are included with libp2p and provide key functionality. - +This article is concerned with protocols that compose libp2p +using core libp2p abstractions like the [transport](/concepts/transport), +[peer identity](/concepts/peers#peer-id/), [addressing](/concepts/addressing/) +abstractions to name a few. ## What is a libp2p protocol? -A libp2p protocol has these key features: +A libp2p protocol is a network protocol that includes +the key features mentioned below. -#### Protocol Ids +#### Protocol IDs -libp2p protocols have unique string identifiers, which are used in the [protocol negotiation](#protocol-negotiation) process when connections are first opened. +libp2p protocols have unique string identifiers, which are used in the +[protocol negotiation](#protocol-negotiation) process when connections are first opened. -By convention, protocol ids have a path-like structure, with a version number as the final component: +By convention, protocol IDs have a path-like structure, with a version number as the +final component: ``` /my-app/amazing-protocol/1.0.1 ``` -Breaking changes to your protocol's wire format or semantics should result in a new version -number. See the [protocol negotiation section](#protocol-neotiation) for more information about -how version selection works during the dialing and listening process. +Breaking changes to a protocol's wire format or semantics should result in a new +version number. See the [protocol negotiation section](#protocol-neotiation) for more +information about how version selection works during the dialing and listening. {{% notice "info" %}} -While libp2p will technically accept any string as a valid protocol id, +While libp2p can accept any string as a valid protocol ID, using the recommended path structure with a version component is both developer-friendly and enables [easier matching by version](#match-using-semver). @@ -45,183 +40,123 @@ developer-friendly and enables [easier matching by version](#match-using-semver) #### Handler functions -To accept connections, a libp2p application will register handler functions for protocols using their protocol id with the -[switch][definition_switch] (aka "swarm"), or a higher level interface such as [go's Host interface](https://github.com/libp2p/go-libp2p-core/blob/master/host/host.go). - -The handler function will be invoked when an incoming stream is tagged with the registered protocol id. -If you register your handler with a [match function](#using-a-match-function), you can choose whether -to accept non-exact string matches for protocol ids, for example, to match on [semantic major versions](#match-using-semver). +A libp2p application will register handler functions +for protocols using their protocol ID with the +[switch][definition_switch] (aka "swarm"), or a higher level interface such as +[go's Host interface](https://github.com/libp2p/go-libp2p-core/blob/master/host/host.go) +to accept connections. +The handler function invokes when an incoming stream is tagged with +the registered protocol ID. If you register a handler with a +[match function](#matching-protocol-ids-and-versions), you can choose whether +to accept non-exact string matches for protocol ids, for example, to match +on [semantic major versions](#match-using-semver). #### Binary streams -The "medium" over which a libp2p protocol transpires is a bi-directional binary stream with the following -properties: +The "medium" over which a libp2p protocol transpires is a bi-directional binary stream +with the following properties: - Bidirectional, reliable delivery of binary data - Each side can read and write from the stream at any time - Data is read in the same order as it was written - - Can be "half-closed", that is, closed for writing and open for reading, or closed for reading and open for writing + - Can be "half-closed", that is, closed for writing and open for reading, or closed + for reading and open for writing - Supports backpressure - - Readers can't be flooded by eager writers + - Eager writers can't flood readers -Behind the scenes, libp2p will also ensure that the stream is [secure](/concepts/secure-comms/) and efficiently -[multiplexed](/concepts/stream-multiplexing/). This is transparent to the protocol handler, which reads and writes -unencrypted binary data over the stream. - -The format of the binary data and the mechanics of what to send when and by whom are all up to the protocol to determine. For inspiration, some [common patterns](#common-patterns) that are used in libp2p's internal protocols are outlined below. +Behind the scenes, libp2p will also ensure that the stream is +[secure](/concepts/secure-comms/) and efficiently +[multiplexed](/concepts/stream-multiplexing/). This is transparent to the protocol +handler, which reads and writes unencrypted binary data over the stream. +The protocol determines the binary data format and the transport mechanics. +For inspiration, some [common patterns](#common-patterns) that are used in libp2p's +internal protocols are outlined below. ## Protocol Negotiation -When dialing out to initiate a new stream, libp2p will send the protocol id of the protocol you want to use. -The listening peer on the other end will check the incoming protocol id against the registered protocol handlers. - -If the listening peer does not support the requested protocol, it will end the stream, and the dialing peer can -try again with a different protocol, or possibly a fallback version of the initially requested protocol. - -If the protocol is supported, the listening peer will echo back the protocol id as a signal that future data -sent over the stream will -use the agreed protocol semantics. - -This process of reaching agreement about what protocol to use for a given stream or connection is called -**protocol negotiation**. - +When dialing out to initiate a new stream, libp2p will send the protocol ID of the +protocol you want to use. The listening peer on the other end will check the incoming +protocol ID against the registered protocol handlers. -### Matching protocol ids and versions +If the listening peer does not support the requested protocol, it will end the stream, +and the dialing peer can try again with a different protocol or possibly a fallback +version of the initially requested protocol. -When you register a protocol handler, there are two methods you can use. +If the protocol is supported, the listening peer will echo back the protocol ID as +a signal that future data sent over the stream will use the agreed protocol semantics. -The first takes two arguments: a protocol id, and a handler function. If an incoming stream request sends an exact -match for the protocol id, the handler function will be invoked with the new stream as an argument. +This process of reaching an agreement about what protocol to use for a given stream +or connection is called **protocol negotiation**. -#### Using a match function +### Matching protocol IDs and versions -The second kind of protocol registration takes three arguments: the protocol id, a protocol match function, and the handler function. +There are two methods available to register a protocol handler. -When a stream request comes in whose protocol id doesn't have any exact matches, the protocol id will be passed through -all of the registered match functions. If any returns `true`, the associated handler function will be invoked. +1. **The first takes two arguments: a protocol ID and a handler function. If an incoming +stream request sends an exact match for the protocol ID; the handler function invokes +the new stream as an argument. -This gives you a lot of flexibility to do your own "fuzzy matching" and define whatever rules for protocol matching -make sense for your application. +2. **Using a match function**: The second kind of protocol registration takes three arguments: + the protocol ID, a protocol match function, and the handler function. + - When a stream request comes in whose protocol ID doesn't have any exact matches, + the protocol ID will pass through all the registered match functions if any returns + `true`, the associated handler function invokes. + - This offers a lot of flexibility to do your own "fuzzy matching" and define + whatever rules for protocol matching make sense of your application. ### Dialing a specific protocol -When dialing a remote peer to open a new stream, the initiating peer sends the protocol id that they'd like to use. The remote peer will use -the matching logic described above to accept or reject the protocol. If the protocol is rejected, the dialing peer can try again. +When dialing a remote peer to open a new stream, the initiating peer sends the protocol +ID they'd like to use. The remote peer will use the matching logic described above to +accept or reject the protocol. If the protocol is rejected, the dialing peer can try again. -When dialing, you can optionally provide a list of protocol ids instead of a single id. When you provide multiple protocol ids, they will -each be tried in succession, and the first successful match will be used if at least one of the protocols is supported by the remote peer. -This can be useful if you support a range of protocol versions, since you can propose the most recent version and fallback to older versions -if the remote hasn't adopted the latest version yet. +When dialing, you can optionally provide a list of protocol IDs instead of a single ID. +When you provide multiple protocol IDs, they will each is tried in succession, and the +first successful match will be used if at least one of the protocols is supported by the +remote peer. This can be useful if you support a range of protocol versions since you can +propose the most recent version and fallback to older versions if the remote hasn't adopted +the latest version yet. ## Core libp2p protocols -In addition to the protocols that you write when developing a libp2p application, libp2p itself defines several foundational protocols that are used for core features. +In addition to the protocols written when developing a libp2p application, libp2p defines +several foundational protocols used for core features. ### Common patterns -The protocols described below all use [protocol buffers](https://developers.google.com/protocol-buffers/) (aka protobuf) to define message schemas. - -Messages are exchanged over the wire using a very simple convention which prefixes binary -message payloads with an integer that represents the length of the payload in bytes. The -length is encoded as a [protobuf varint](https://developers.google.com/protocol-buffers/docs/encoding#varints) (variable-length integer). - - - -### Ping - -| **Protocol id** | spec | | | implementations | -|--------------------|------|---------------|---------------|-------------------| -| `/ipfs/ping/1.0.0` | N/A | [go][ping_go] | [js][ping_js] | [rust][ping_rust] | - - -[ping_go]: https://github.com/libp2p/go-libp2p/tree/master/p2p/protocol/ping -[ping_js]: https://github.com/libp2p/js-libp2p-ping -[ping_rust]: https://github.com/libp2p/rust-libp2p/blob/master/protocols/ping/src/lib.rs - - -The ping protocol is a simple liveness check that peers can use to quickly see if another peer is online. - -After the initial protocol negotiation, the dialing peer sends 32 bytes of random binary data. The listening -peer echoes the data back, and the dialing peer will verify the response and measure -the latency between request and response. - -### Identify +The protocols described below all use +[protocol buffers](https://developers.google.com/protocol-buffers/) +(aka protobuf) to define message schemas. -| **Protocol id** | spec | | | implementations | -|------------------|--------------------------------|-------------------|-------------------|-----------------------| -| `/ipfs/id/1.0.0` | [identify spec][spec_identify] | [go][identify_go] | [js][identify_js] | [rust][identify_rust] | +Messages are exchanged over the wire using a straightforward convention that prefixes +binary message payloads with an integer representing the payload's length in bytes. The +length is encoded as a +[protobuf varint](https://developers.google.com/protocol-buffers/docs/encoding#varints) +(variable-length integer). - -[spec_identify]: https://github.com/libp2p/specs/pull/97/files -[identify_go]: https://github.com/libp2p/go-libp2p/tree/master/p2p/protocol/identify -[identify_js]: https://github.com/libp2p/js-libp2p-identify -[identify_rust]: https://github.com/libp2p/rust-libp2p/tree/master/protocols/identify/src +### Protocol list -The `identify` protocol allows peers to exchange information about each other, most notably their public keys -and known network addresses. - - -The basic identify protocol works by establishing a new stream to a peer using the identify protocol id -shown in the table above. - -When the remote peer opens the new stream, they will fill out an [`Identify` protobuf message][identify_proto] containing -information about themselves, such as their public key, which is used to derive their [`PeerId`](/concepts/peers/). - -Importantly, the `Identify` message includes an `observedAddr` field that contains the [multiaddr][definition_multiaddr] that -the peer observed the request coming in on. This helps peers determine their NAT status, since it allows them to -see what other peers observe as their public address and compare it to their own view of the network. - -[identify_proto]: https://github.com/libp2p/go-libp2p/blob/master/p2p/protocol/identify/pb/identify.proto - -#### identify/push - -| **Protocol id** | spec & implementations | -|-----------------------|-------------------------------------| -| `/ipfs/id/push/1.0.0` | same as [identify above](#identify) | - -A slight variation on `identify`, the `identify/push` protocol sends the same `Identify` message, but it does so proactively -instead of in response to a request. - -This is useful if a peer starts listening on a new address, establishes a new [relay circuit](/concepts/circuit-relay/), or -learns of its public address from other peers using the standard `identify` protocol. Upon creating or learning of a new address, -the peer can push the new address to all peers it's currently aware of. This keeps everyone's routing tables up to date and -makes it more likely that other peers will discover the new address. - -### kad-dht - -`kad-dht` is a [Distributed Hash Table][wiki_dht] based on the [Kademlia][wiki_kad] routing algorithm, with some modifications. - -libp2p uses the DHT as the foundation of its [peer routing](/concepts/peer-routing/) and [content routing](/concepts/content-routing/) functionality. To learn more about DHT and the Kademlia algorithm, -check out the [Distributed Hash Tables guide][dht] on the IPFS documentation site. In addition, check out the [libp2p implementations page](https://libp2p.io/implementations/) for updates on all the kad-libp2p implementations. - - - -[spec_kad]: https://github.com/libp2p/specs/pull/108 - -[wiki_dht]: https://en.wikipedia.org/wiki/Distributed_hash_table -[wiki_kad]: https://en.wikipedia.org/wiki/Kademlia -[dht]: https://docs.ipfs.tech/concepts/dht/ - -### Circuit Relay - -| **Protocol id** | spec | | implementations | -|-------------------------------|----------------------------------|----------------|-----------------| -| `/libp2p/circuit/relay/0.1.0` | [circuit relay spec][spec_relay] | [go][relay_go] | [js][relay_js] | - -[spec_relay]: https://github.com/libp2p/specs/tree/master/relay -[relay_js]: https://github.com/libp2p/js-libp2p-circuit -[relay_go]: https://github.com/libp2p/go-libp2p-circuit - -As described in the [Circuit Relay article](/concepts/circuit-relay/), libp2p provides a protocol -for tunneling traffic through relay peers when two peers are unable to connect to each other -directly. See the article for more information on working with relays, including notes on relay -addresses and how to enable automatic relay connection when behind an intractable NAT. - - +{{% notice "note" %}} +Check out the [libp2p implementations page](https://libp2p.io/implementations/) for +updates on all the libp2p implementations. +{{% /notice %}} -[definition_switch]: /reference/glossary/#switch -[definition_multiaddr]: /reference/glossary/#multiaddr +| **name** | **protocol id** | **spec** | +|---------------|-------------------------------|------------------------------------------------------------------------------------------------| +| ping | `/ipfs/ping/1.0.0` | N/A | +| identity | `/ipfs/id/1.0.0` | [spec_identity](https://github.com/libp2p/specs/blob/master/identify/README.md) | +| identify/push | `/ipfs/id/push/1.0.0` | [spec_identity](https://github.com/libp2p/specs/blob/master/identify/README.md) | +| kad-dht | `/ipfs/kad/1.0.0` | [spec_kad](https://github.com/libp2p/specs/blob/master/kad-dht/README.md) | +| noise | `/libp2p/noise/1.0.0` | [spec_noise](https://github.com/libp2p/specs/blob/master/noise/README.md) | +| tls | `/libp2p/tls/1.0.0` | [spec_tls](https://github.com/libp2p/specs/blob/master/tls/tls.md) | +| plaintext | `/libp2p/plaintext/2.0.0` | [spec_plaintext](https://github.com/libp2p/specs/blob/master/plaintext/README.md) | +| rendezvous | `/libp2p/rendezvous/1.0.0` | [spec_rendezvous](https://github.com/libp2p/specs/blob/master/rendezvous/README.md) | +| mdns | `/libp2p/mdns/1.0.0` | [spec_mdns](https://github.com/libp2p/specs/blob/master/discovery/mdns.md) | +| autonat | `/libp2p/autonat/1.0.0` | [spec_autonat](https://github.com/libp2p/specs/blob/master/autonat/README.md#autonat-protocol) | +| circuit relay | `/libp2p/circuit/relay/0.1.0` | [spec_relay](https://github.com/libp2p/specs/blob/master/relay/README.md) | +| dcutr | `/libp2p/dcutr/1.0.0` | [spec_dcutr](https://github.com/libp2p/specs/blob/master/relay/DCUtR.md) | +| mplex | `/libp2p/mplex/1.0.0` | [spec_mplex](https://github.com/libp2p/specs/blob/master/mplex/README.md) | +| pnet | `libp2p/key/swarm/psk/1.0.0/` | [spec_pnet](https://github.com/libp2p/specs/blob/master/pnet/Private-Networks-PSK-V1.md) | -[repo_multistream-select]: https://github.com/multiformats/multistream-select From e50fc1ae82d499c107f332ddfc1b1eabf35d95b0 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Wed, 28 Sep 2022 05:58:57 -0400 Subject: [PATCH 02/11] addr pr feedback: update table --- content/concepts/protocols.md | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index fa708fb6..6bad1daa 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -143,20 +143,21 @@ Check out the [libp2p implementations page](https://libp2p.io/implementations/) updates on all the libp2p implementations. {{% /notice %}} -| **name** | **protocol id** | **spec** | -|---------------|-------------------------------|------------------------------------------------------------------------------------------------| -| ping | `/ipfs/ping/1.0.0` | N/A | -| identity | `/ipfs/id/1.0.0` | [spec_identity](https://github.com/libp2p/specs/blob/master/identify/README.md) | -| identify/push | `/ipfs/id/push/1.0.0` | [spec_identity](https://github.com/libp2p/specs/blob/master/identify/README.md) | -| kad-dht | `/ipfs/kad/1.0.0` | [spec_kad](https://github.com/libp2p/specs/blob/master/kad-dht/README.md) | -| noise | `/libp2p/noise/1.0.0` | [spec_noise](https://github.com/libp2p/specs/blob/master/noise/README.md) | -| tls | `/libp2p/tls/1.0.0` | [spec_tls](https://github.com/libp2p/specs/blob/master/tls/tls.md) | -| plaintext | `/libp2p/plaintext/2.0.0` | [spec_plaintext](https://github.com/libp2p/specs/blob/master/plaintext/README.md) | -| rendezvous | `/libp2p/rendezvous/1.0.0` | [spec_rendezvous](https://github.com/libp2p/specs/blob/master/rendezvous/README.md) | -| mdns | `/libp2p/mdns/1.0.0` | [spec_mdns](https://github.com/libp2p/specs/blob/master/discovery/mdns.md) | -| autonat | `/libp2p/autonat/1.0.0` | [spec_autonat](https://github.com/libp2p/specs/blob/master/autonat/README.md#autonat-protocol) | -| circuit relay | `/libp2p/circuit/relay/0.1.0` | [spec_relay](https://github.com/libp2p/specs/blob/master/relay/README.md) | -| dcutr | `/libp2p/dcutr/1.0.0` | [spec_dcutr](https://github.com/libp2p/specs/blob/master/relay/DCUtR.md) | -| mplex | `/libp2p/mplex/1.0.0` | [spec_mplex](https://github.com/libp2p/specs/blob/master/mplex/README.md) | -| pnet | `libp2p/key/swarm/psk/1.0.0/` | [spec_pnet](https://github.com/libp2p/specs/blob/master/pnet/Private-Networks-PSK-V1.md) | - +| **Specification** | **Protocol ID** | +|-------------------------------------------------------------------------------------------|-----------------------------------| +| Ping | `/ipfs/ping/1.0.0` | +| [Identify](https://github.com/libp2p/specs/blob/master/identify/README.md) | `/ipfs/id/1.0.0` | +| [Identify (push)](https://github.com/libp2p/specs/blob/master/identify/README.md) | `/ipfs/id/push/1.0.0` | +| [Kademlia DHT](https://github.com/libp2p/specs/blob/master/kad-dht/README.md) | `/ipfs/kad/1.0.0` | +| [QUIC](https://datatracker.ietf.org/doc/html/rfc9000) | `/libp2p/quic/1.0.0` | +| [WebTransport](https://github.com/libp2p/specs/pull/404) | `/libp2p/quic/webtransport/1.0.0` | +| [Noise](https://github.com/libp2p/specs/blob/master/noise/README.md) | `/libp2p/noise/1.0.0` | +| [TLS 1.3](https://github.com/libp2p/specs/blob/master/tls/tls.md) | `/libp2p/tls/1.0.0` | +| [Plaintext](https://github.com/libp2p/specs/blob/master/plaintext/README.md) | `/libp2p/plaintext/2.0.0` | +| [Rendezvous](https://github.com/libp2p/specs/blob/master/rendezvous/README.md) | `/libp2p/rendezvous/1.0.0` | +| [mDNS](https://github.com/libp2p/specs/blob/master/discovery/mdns.md) | `/libp2p/mdns/1.0.0` | +| [AutoNAT](https://github.com/libp2p/specs/blob/master/autonat/README.md#autonat-protocol) | `/libp2p/autonat/1.0.0` | +| [Circuit relay](https://github.com/libp2p/specs/blob/master/relay/README.md) | `/libp2p/circuit/relay/0.1.0` | +| [DCUtR](https://github.com/libp2p/specs/blob/master/relay/DCUtR.md) | `/libp2p/dcutr/1.0.0` | +| [mplex](https://github.com/libp2p/specs/blob/master/mplex/README.md) | `/libp2p/mplex/1.0.0` | +| [pnet](https://github.com/libp2p/specs/blob/master/pnet/Private-Networks-PSK-V1.md) | `libp2p/key/swarm/psk/1.0.0/` | From ab036a226a6cc1a197e991fa421c57cfebe8a364 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Wed, 28 Sep 2022 06:16:43 -0400 Subject: [PATCH 03/11] addr pr feedback: update table and add missing protocols --- content/concepts/protocols.md | 42 +++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index 6bad1daa..2885a11e 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -94,7 +94,7 @@ or connection is called **protocol negotiation**. There are two methods available to register a protocol handler. -1. **The first takes two arguments: a protocol ID and a handler function. If an incoming +1. The first takes two arguments: a protocol ID and a handler function. If an incoming stream request sends an exact match for the protocol ID; the handler function invokes the new stream as an argument. @@ -143,21 +143,25 @@ Check out the [libp2p implementations page](https://libp2p.io/implementations/) updates on all the libp2p implementations. {{% /notice %}} -| **Specification** | **Protocol ID** | -|-------------------------------------------------------------------------------------------|-----------------------------------| -| Ping | `/ipfs/ping/1.0.0` | -| [Identify](https://github.com/libp2p/specs/blob/master/identify/README.md) | `/ipfs/id/1.0.0` | -| [Identify (push)](https://github.com/libp2p/specs/blob/master/identify/README.md) | `/ipfs/id/push/1.0.0` | -| [Kademlia DHT](https://github.com/libp2p/specs/blob/master/kad-dht/README.md) | `/ipfs/kad/1.0.0` | -| [QUIC](https://datatracker.ietf.org/doc/html/rfc9000) | `/libp2p/quic/1.0.0` | -| [WebTransport](https://github.com/libp2p/specs/pull/404) | `/libp2p/quic/webtransport/1.0.0` | -| [Noise](https://github.com/libp2p/specs/blob/master/noise/README.md) | `/libp2p/noise/1.0.0` | -| [TLS 1.3](https://github.com/libp2p/specs/blob/master/tls/tls.md) | `/libp2p/tls/1.0.0` | -| [Plaintext](https://github.com/libp2p/specs/blob/master/plaintext/README.md) | `/libp2p/plaintext/2.0.0` | -| [Rendezvous](https://github.com/libp2p/specs/blob/master/rendezvous/README.md) | `/libp2p/rendezvous/1.0.0` | -| [mDNS](https://github.com/libp2p/specs/blob/master/discovery/mdns.md) | `/libp2p/mdns/1.0.0` | -| [AutoNAT](https://github.com/libp2p/specs/blob/master/autonat/README.md#autonat-protocol) | `/libp2p/autonat/1.0.0` | -| [Circuit relay](https://github.com/libp2p/specs/blob/master/relay/README.md) | `/libp2p/circuit/relay/0.1.0` | -| [DCUtR](https://github.com/libp2p/specs/blob/master/relay/DCUtR.md) | `/libp2p/dcutr/1.0.0` | -| [mplex](https://github.com/libp2p/specs/blob/master/mplex/README.md) | `/libp2p/mplex/1.0.0` | -| [pnet](https://github.com/libp2p/specs/blob/master/pnet/Private-Networks-PSK-V1.md) | `libp2p/key/swarm/psk/1.0.0/` | +| **Specification** | **Protocol ID** | +|--------------------------------------------------------------------------------------------|------------------------------------| +| [AutoNAT](https://github.com/libp2p/specs/blob/master/autonat/README.md#autonat-protocol) | `/libp2p/autonat/1.0.0` | +| [Circuit Relay v2 (hop) ](https://github.com/libp2p/specs/blob/master/relay/circuit-v2.md) | `/libp2p/circuit/relay/0.2.0/hop` | +| [Circuit Relay v2 (stop)](https://github.com/libp2p/specs/blob/master/relay/circuit-v2.md) | `/libp2p/circuit/relay/0.2.0/stop` | +| [DCUtR](https://github.com/libp2p/specs/blob/master/relay/DCUtR.md) | `/libp2p/dcutr/1.0.0` | +| [Fetch](https://github.com/libp2p/specs/tree/master/fetch) | `/libp2p/fetch/0.0.1` | +| [GossipSub v1.0](https://github.com/libp2p/specs/tree/master/pubsub/gossipsub) | `/libp2p/gossipsub/1.0.0` | +| [GossipSub v1.1](https://github.com/libp2p/specs/tree/master/pubsub/gossipsub) | `/libp2p/gossipsub/1.1.0` | +| [Identify](https://github.com/libp2p/specs/blob/master/identify/README.md) | `/ipfs/id/1.0.0` | +| [Identify (push)](https://github.com/libp2p/specs/blob/master/identify/README.md) | `/ipfs/id/push/1.0.0` | +| [Kademlia DHT](https://github.com/libp2p/specs/blob/master/kad-dht/README.md) | `/ipfs/kad/1.0.0` | +| [mDNS](https://github.com/libp2p/specs/blob/master/discovery/mdns.md) | `/libp2p/mdns/1.0.0` | +| [mplex](https://github.com/libp2p/specs/blob/master/mplex/README.md) | `/libp2p/mplex/1.0.0` | +| [Noise](https://github.com/libp2p/specs/blob/master/noise/README.md) | `/libp2p/noise/1.0.0` | +| Ping | `/ipfs/ping/1.0.0` | +| [Plaintext](https://github.com/libp2p/specs/blob/master/plaintext/README.md) | `/libp2p/plaintext/2.0.0` | +| [pnet](https://github.com/libp2p/specs/blob/master/pnet/Private-Networks-PSK-V1.md) | `libp2p/key/swarm/psk/1.0.0/` | +| [QUIC](https://datatracker.ietf.org/doc/html/rfc9000) | `/libp2p/quic/1.0.0` | +| [Rendezvous](https://github.com/libp2p/specs/blob/master/rendezvous/README.md) | `/libp2p/rendezvous/1.0.0` | +| [TLS 1.3](https://github.com/libp2p/specs/blob/master/tls/tls.md) | `/libp2p/tls/1.0.0` | +| [WebTransport](https://github.com/libp2p/specs/pull/404) | `/libp2p/quic/webtransport/1.0.0` | From 0c0228423d15891c73b661bd3d560cf67c5c1389 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 29 Sep 2022 04:00:25 -0400 Subject: [PATCH 04/11] Apply suggestions from code review Co-authored-by: Marten Seemann --- content/concepts/protocols.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index 2885a11e..944854eb 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -17,9 +17,9 @@ the key features mentioned below. #### Protocol IDs libp2p protocols have unique string identifiers, which are used in the -[protocol negotiation](#protocol-negotiation) process when connections are first opened. +[protocol negotiation](#protocol-negotiation) process when streams are first opened. -By convention, protocol IDs have a path-like structure, with a version number as the +Historically, the IDs of many protocols in use by libp2p have a path-like structure, with a version number as the final component: ``` @@ -76,11 +76,11 @@ internal protocols are outlined below. ## Protocol Negotiation -When dialing out to initiate a new stream, libp2p will send the protocol ID of the -protocol you want to use. The listening peer on the other end will check the incoming +When dialing out to initiate a new stream, libp2p sends the protocol ID of the +protocol you want to use. The listening peer on the other end checks the incoming protocol ID against the registered protocol handlers. -If the listening peer does not support the requested protocol, it will end the stream, +If the listening peer does not support the requested protocol, it will send "na" on the stream and the dialing peer can try again with a different protocol or possibly a fallback version of the initially requested protocol. From 4ca86d1d55c954fc654a3d568f63c0bcf39c31c9 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 29 Sep 2022 04:01:41 -0400 Subject: [PATCH 05/11] rmv transport-related protocols from table --- content/concepts/protocols.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index 944854eb..d231a444 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -155,13 +155,5 @@ updates on all the libp2p implementations. | [Identify](https://github.com/libp2p/specs/blob/master/identify/README.md) | `/ipfs/id/1.0.0` | | [Identify (push)](https://github.com/libp2p/specs/blob/master/identify/README.md) | `/ipfs/id/push/1.0.0` | | [Kademlia DHT](https://github.com/libp2p/specs/blob/master/kad-dht/README.md) | `/ipfs/kad/1.0.0` | -| [mDNS](https://github.com/libp2p/specs/blob/master/discovery/mdns.md) | `/libp2p/mdns/1.0.0` | -| [mplex](https://github.com/libp2p/specs/blob/master/mplex/README.md) | `/libp2p/mplex/1.0.0` | -| [Noise](https://github.com/libp2p/specs/blob/master/noise/README.md) | `/libp2p/noise/1.0.0` | | Ping | `/ipfs/ping/1.0.0` | -| [Plaintext](https://github.com/libp2p/specs/blob/master/plaintext/README.md) | `/libp2p/plaintext/2.0.0` | -| [pnet](https://github.com/libp2p/specs/blob/master/pnet/Private-Networks-PSK-V1.md) | `libp2p/key/swarm/psk/1.0.0/` | -| [QUIC](https://datatracker.ietf.org/doc/html/rfc9000) | `/libp2p/quic/1.0.0` | | [Rendezvous](https://github.com/libp2p/specs/blob/master/rendezvous/README.md) | `/libp2p/rendezvous/1.0.0` | -| [TLS 1.3](https://github.com/libp2p/specs/blob/master/tls/tls.md) | `/libp2p/tls/1.0.0` | -| [WebTransport](https://github.com/libp2p/specs/pull/404) | `/libp2p/quic/webtransport/1.0.0` | From d01340fc498f6046d74bc724f65992be84d82b21 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 29 Sep 2022 04:03:55 -0400 Subject: [PATCH 06/11] incorporate pr feedback --- content/concepts/protocols.md | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index d231a444..117b5122 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -30,21 +30,11 @@ Breaking changes to a protocol's wire format or semantics should result in a new version number. See the [protocol negotiation section](#protocol-neotiation) for more information about how version selection works during the dialing and listening. -{{% notice "info" %}} - -While libp2p can accept any string as a valid protocol ID, -using the recommended path structure with a version component is both -developer-friendly and enables [easier matching by version](#match-using-semver). - -{{% /notice %}} - #### Handler functions -A libp2p application will register handler functions -for protocols using their protocol ID with the -[switch][definition_switch] (aka "swarm"), or a higher level interface such as -[go's Host interface](https://github.com/libp2p/go-libp2p-core/blob/master/host/host.go) -to accept connections. +A libp2p application will define a stream handler that takes over the +stream after protocol negotiation. Everything sent and received after the +negotiation phase is then defined by the application protocol. The handler function invokes when an incoming stream is tagged with the registered protocol ID. If you register a handler with a From 24f7524c5281c8f4e8cd5b99cabce03ff107692a Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 29 Sep 2022 05:45:40 -0400 Subject: [PATCH 07/11] Apply suggestions from code review Co-authored-by: Marten Seemann --- content/concepts/protocols.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index 117b5122..3b6d51ac 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -36,7 +36,7 @@ A libp2p application will define a stream handler that takes over the stream after protocol negotiation. Everything sent and received after the negotiation phase is then defined by the application protocol. -The handler function invokes when an incoming stream is tagged with +The handler function is invoked when an incoming stream is received with the registered protocol ID. If you register a handler with a [match function](#matching-protocol-ids-and-versions), you can choose whether to accept non-exact string matches for protocol ids, for example, to match @@ -98,11 +98,11 @@ the new stream as an argument. ### Dialing a specific protocol -When dialing a remote peer to open a new stream, the initiating peer sends the protocol -ID they'd like to use. The remote peer will use the matching logic described above to +When opening a new stream on a connection, the initiating peer sends the protocol +ID it would like to use. The remote peer will use the matching logic described above to accept or reject the protocol. If the protocol is rejected, the dialing peer can try again. -When dialing, you can optionally provide a list of protocol IDs instead of a single ID. +When opening a new stream, the initiator can optionally provide a list of protocol IDs instead of a single ID. When you provide multiple protocol IDs, they will each is tried in succession, and the first successful match will be used if at least one of the protocols is supported by the remote peer. This can be useful if you support a range of protocol versions since you can From edcf5c975a3d84a0f44d768dfdefcd7339739656 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 29 Sep 2022 05:56:13 -0400 Subject: [PATCH 08/11] incorporate PR feedback --- content/concepts/protocols.md | 27 +++++++++------------------ 1 file changed, 9 insertions(+), 18 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index 3b6d51ac..99af506c 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -19,8 +19,8 @@ the key features mentioned below. libp2p protocols have unique string identifiers, which are used in the [protocol negotiation](#protocol-negotiation) process when streams are first opened. -Historically, the IDs of many protocols in use by libp2p have a path-like structure, with a version number as the -final component: +Historically, the IDs of many protocols in use by libp2p have a path-like structure, +with a version number as the final component: ``` /my-app/amazing-protocol/1.0.1 @@ -82,19 +82,10 @@ or connection is called **protocol negotiation**. ### Matching protocol IDs and versions -There are two methods available to register a protocol handler. - -1. The first takes two arguments: a protocol ID and a handler function. If an incoming -stream request sends an exact match for the protocol ID; the handler function invokes -the new stream as an argument. - -2. **Using a match function**: The second kind of protocol registration takes three arguments: - the protocol ID, a protocol match function, and the handler function. - - When a stream request comes in whose protocol ID doesn't have any exact matches, - the protocol ID will pass through all the registered match functions if any returns - `true`, the associated handler function invokes. - - This offers a lot of flexibility to do your own "fuzzy matching" and define - whatever rules for protocol matching make sense of your application. +There are different ways to register a protocol handler. For instance, a match function +is a flexible method for protocol registration. A match function +takes three arguments: the protocol ID, a protocol match function, +and the handler function as arguments. ### Dialing a specific protocol @@ -102,9 +93,9 @@ When opening a new stream on a connection, the initiating peer sends the protoco ID it would like to use. The remote peer will use the matching logic described above to accept or reject the protocol. If the protocol is rejected, the dialing peer can try again. -When opening a new stream, the initiator can optionally provide a list of protocol IDs instead of a single ID. -When you provide multiple protocol IDs, they will each is tried in succession, and the -first successful match will be used if at least one of the protocols is supported by the +When opening a new stream, the initiator can optionally provide a list of protocol IDs instead +of a single ID. When you provide multiple protocol IDs, they will each is tried in succession, +and the first successful match will be used if at least one of the protocols is supported by the remote peer. This can be useful if you support a range of protocol versions since you can propose the most recent version and fallback to older versions if the remote hasn't adopted the latest version yet. From 31702b3132f53c1748d8b093ec4d77cea6903b81 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 29 Sep 2022 06:29:05 -0400 Subject: [PATCH 09/11] enhance protocol neg. section --- content/concepts/protocols.md | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index 99af506c..04327b3a 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -14,7 +14,7 @@ abstractions to name a few. A libp2p protocol is a network protocol that includes the key features mentioned below. -#### Protocol IDs +### Protocol IDs libp2p protocols have unique string identifiers, which are used in the [protocol negotiation](#protocol-negotiation) process when streams are first opened. @@ -30,7 +30,7 @@ Breaking changes to a protocol's wire format or semantics should result in a new version number. See the [protocol negotiation section](#protocol-neotiation) for more information about how version selection works during the dialing and listening. -#### Handler functions +### Handler functions A libp2p application will define a stream handler that takes over the stream after protocol negotiation. Everything sent and received after the @@ -42,7 +42,7 @@ the registered protocol ID. If you register a handler with a to accept non-exact string matches for protocol ids, for example, to match on [semantic major versions](#match-using-semver). -#### Binary streams +### Binary streams The "medium" over which a libp2p protocol transpires is a bi-directional binary stream with the following properties: @@ -66,13 +66,21 @@ internal protocols are outlined below. ## Protocol Negotiation +Libp2p protocols can have sub-protocols or protocol-suites. It is difficult +to select and route protocols, or even know which protocols are available. + +[Multistream-select](https://github.com/multiformats/multistream-select) +is a protocol multiplexer negotiates the application-layer protocol. When a new +stream is created, it is ran through `multistream-select` to route the stream to the +ppropriate protocol handler. + When dialing out to initiate a new stream, libp2p sends the protocol ID of the protocol you want to use. The listening peer on the other end checks the incoming protocol ID against the registered protocol handlers. -If the listening peer does not support the requested protocol, it will send "na" on the stream -and the dialing peer can try again with a different protocol or possibly a fallback -version of the initially requested protocol. +If the listening peer does not support the requested protocol, it will send "na" +on the stream and the dialing peer can try again with a different protocol or possibly +a fallback version of the initially requested protocol. If the protocol is supported, the listening peer will echo back the protocol ID as a signal that future data sent over the stream will use the agreed protocol semantics. @@ -80,6 +88,12 @@ a signal that future data sent over the stream will use the agreed protocol sema This process of reaching an agreement about what protocol to use for a given stream or connection is called **protocol negotiation**. +{{% notice "note" %}} +Not to be mistaken with [stream multiplexers](/concepts/stream-multiplexing), +which are solely responsible for the internal streams on a connection and aren't +concerned with underlying protocol being used. +{{% /notice %}} + ### Matching protocol IDs and versions There are different ways to register a protocol handler. For instance, a match function From 2e6073c18b21bf4e97b368c8160ccddba818b278 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 29 Sep 2022 06:30:26 -0400 Subject: [PATCH 10/11] tweaks --- content/concepts/protocols.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index 04327b3a..3b02e012 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -66,20 +66,20 @@ internal protocols are outlined below. ## Protocol Negotiation -Libp2p protocols can have sub-protocols or protocol-suites. It is difficult +Libp2p protocols can have sub-protocols or protocol-suites. It isn't easy to select and route protocols, or even know which protocols are available. [Multistream-select](https://github.com/multiformats/multistream-select) -is a protocol multiplexer negotiates the application-layer protocol. When a new -stream is created, it is ran through `multistream-select` to route the stream to the -ppropriate protocol handler. +is a protocol multiplexer that negotiates the application-layer protocol. When +a new stream is created, it is run through `multistream-select`, which routes the +stream to the appropriate protocol handler. When dialing out to initiate a new stream, libp2p sends the protocol ID of the protocol you want to use. The listening peer on the other end checks the incoming protocol ID against the registered protocol handlers. If the listening peer does not support the requested protocol, it will send "na" -on the stream and the dialing peer can try again with a different protocol or possibly +on the stream, and the dialing peer can try again with a different protocol or possibly a fallback version of the initially requested protocol. If the protocol is supported, the listening peer will echo back the protocol ID as From 78b6279e005cd1c9205660cfd2ca6ab4ca367278 Mon Sep 17 00:00:00 2001 From: Danny Salman Date: Thu, 13 Oct 2022 15:33:04 -0400 Subject: [PATCH 11/11] incorporate pr feedback --- content/concepts/protocols.md | 39 +++-------------------------------- 1 file changed, 3 insertions(+), 36 deletions(-) diff --git a/content/concepts/protocols.md b/content/concepts/protocols.md index 3b02e012..8de5a49e 100644 --- a/content/concepts/protocols.md +++ b/content/concepts/protocols.md @@ -34,7 +34,7 @@ information about how version selection works during the dialing and listening. A libp2p application will define a stream handler that takes over the stream after protocol negotiation. Everything sent and received after the -negotiation phase is then defined by the application protocol. +negotiation phase is then defined by the application protocol. The handler function is invoked when an incoming stream is received with the registered protocol ID. If you register a handler with a @@ -64,8 +64,9 @@ The protocol determines the binary data format and the transport mechanics. For inspiration, some [common patterns](#common-patterns) that are used in libp2p's internal protocols are outlined below. -## Protocol Negotiation +## Life cycle of a stream +Libp2p streams are an abstraction Libp2p protocols can have sub-protocols or protocol-suites. It isn't easy to select and route protocols, or even know which protocols are available. @@ -94,45 +95,11 @@ which are solely responsible for the internal streams on a connection and aren't concerned with underlying protocol being used. {{% /notice %}} -### Matching protocol IDs and versions - -There are different ways to register a protocol handler. For instance, a match function -is a flexible method for protocol registration. A match function -takes three arguments: the protocol ID, a protocol match function, -and the handler function as arguments. - -### Dialing a specific protocol - -When opening a new stream on a connection, the initiating peer sends the protocol -ID it would like to use. The remote peer will use the matching logic described above to -accept or reject the protocol. If the protocol is rejected, the dialing peer can try again. - -When opening a new stream, the initiator can optionally provide a list of protocol IDs instead -of a single ID. When you provide multiple protocol IDs, they will each is tried in succession, -and the first successful match will be used if at least one of the protocols is supported by the -remote peer. This can be useful if you support a range of protocol versions since you can -propose the most recent version and fallback to older versions if the remote hasn't adopted -the latest version yet. - ## Core libp2p protocols In addition to the protocols written when developing a libp2p application, libp2p defines several foundational protocols used for core features. -### Common patterns - -The protocols described below all use -[protocol buffers](https://developers.google.com/protocol-buffers/) -(aka protobuf) to define message schemas. - -Messages are exchanged over the wire using a straightforward convention that prefixes -binary message payloads with an integer representing the payload's length in bytes. The -length is encoded as a -[protobuf varint](https://developers.google.com/protocol-buffers/docs/encoding#varints) -(variable-length integer). - -### Protocol list - {{% notice "note" %}} Check out the [libp2p implementations page](https://libp2p.io/implementations/) for updates on all the libp2p implementations.