From c75e81a7d6859e35adf58c63136695e787925d1c Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Tue, 30 May 2023 10:35:15 +0400 Subject: [PATCH 01/10] Add draft DONTSEND Gossip control message spec --- pubsub/gossipsub/gossipsub-v1.2.md | 102 +++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100644 pubsub/gossipsub/gossipsub-v1.2.md diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md new file mode 100644 index 000000000..023b93d85 --- /dev/null +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -0,0 +1,102 @@ +# Gossipsub v1.2 + +# Overview + +This document aims to provide a minimal extension to the [gossipsub +v1.1](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md) +protocol. + +The proposed extensions are backwards-compatible and aim to enhance the +efficiency (minimize amplification/duplicates and decrease message latency) of +the gossip mesh networks for larger messages. + +In more specific terms, a new control message is introduced: `DONTSEND`. It's primarily +intended to notify mesh peers that the node already received a message and there is no +need to send its duplicate. + +# Specification + +## Protocol Id + +Nodes that support this Gossipsub extension should additionally advertise the +version number `1.2.0`. Gossipsub nodes can advertise their own protocol-id +prefix, by default this is `meshsub` giving the default protocol id: +- `/meshsub/1.2.0` + +## Parameters + +This section lists the configuration parameters that needs to agreed on across clients to avoid + peer penalizations + +| Parameter | Description | Reasonable Default | +|-------------------------|------------------------------------------------------------------|--------------| +| `max_dontsend_messages` | The maximum number of `DONTSEND` messages per heartbeat per peer | ??? | + + +## DONTSEND Message + +### Basic scenario + +When the peer receives the first message instance it immediately broadcasts +(not queue for later piggybacking) `DONTSEND` with the `messageId` to all its mesh peers. +This could be performed prior to the message validation to further increase the effectiveness of the approach. + +On the other side a node maintains per-peer `dont_send_message_ids` set. Upon receiving `DONTSEND` from +a peer the `messageId` is added to the `dont_send_message_ids` set. +When later relaying the `messageId` message to the mesh the peers found in `dont_send_message_ids` could be skipped. + +Old entries from `dont_send_message_ids` could be pruned during heartbeat processing. +The prune strategy is outside of the spec scope and can be decided by implementations. + +`DONTSEND` message is supposed to be _optional_ for both receiver and sender. I.e. the sender may or may not utilize +this message. The receiver in turn may ignore `DONTSEND`: sending a message after the corresponding `DONTSEND` +should not be penalized. + +The `DONTSEND` may have negative effect on small messages as it may increase the overall traffic and CPU load. +Thus it is better to utilize `DONTSEND` for messages of a larger size. +The exact policy of `DONTSEND` appliance is outside of the spec scope. Every implementation may choose whatever +is more appropriate for it. Possible options are either choose a message size threshold and broadcast `DONTSEND` +on per message basis when the size is exceeded or just use `DONTSEND` for all messages on selected topics. + +To prevent DoS the number of `DONTSEND` control messages is limited to `max_dontsend_messages` per heartbeat + +### Relying on `IHAVE`s + +Another potential additional strategy could be as follows. If a node receives `IHAVE` (from one or more peers) +before the message is appeared in the mesh the node may request the message with `IWANT` and notify all mesh +peers that it don't want that message from them. + +### Sending `IHAVE` to mesh peers who choked that particular message + +Reasonable addition to the later scenario would be to _immediately_ send `IHAVE` instead of a full message +to those mesh peers who reported `DONTSEND`. That would notify mesh peers that the node has this message +and they could request it from you in case their `IWANT` requests fail in the previous scenario + +### Cancelling `IWANT` + +If a node requested a message via `IWANT` and then occasionally receives the message from other peer it may +try to cancel its `IWANT` requests with the corresponding `DONTSEND` message. It may work in cases when a +peer delays/queues `IWANT` requests and the `IWANT` request would be removed from the queue if not processed yet + +## Protobuf Extension + +The protobuf messages are identical to those specified in the [gossipsub v1.0.0 +specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md) +with the following control message modifications: + +```protobuf +message RPC { + // ... see definition in the gossipsub specification +} + +message ControlMessage { + // messages from v1.0 + repeated ControlDontSend dontSend = 5; +} + +message ControlDontSend { + required bytes messageID = 1; +} + +``` + From 4fafe66a1d6ccbfbeeb6039259aa944fd7611b8a Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 31 May 2023 15:25:20 +0400 Subject: [PATCH 02/10] Rename DONTSEND to IDONTWANT --- pubsub/gossipsub/gossipsub-v1.2.md | 38 +++++++++++++++--------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index 023b93d85..846362700 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -10,7 +10,7 @@ The proposed extensions are backwards-compatible and aim to enhance the efficiency (minimize amplification/duplicates and decrease message latency) of the gossip mesh networks for larger messages. -In more specific terms, a new control message is introduced: `DONTSEND`. It's primarily +In more specific terms, a new control message is introduced: `IDONTWANT`. It's primarily intended to notify mesh peers that the node already received a message and there is no need to send its duplicate. @@ -28,37 +28,37 @@ prefix, by default this is `meshsub` giving the default protocol id: This section lists the configuration parameters that needs to agreed on across clients to avoid peer penalizations -| Parameter | Description | Reasonable Default | -|-------------------------|------------------------------------------------------------------|--------------| -| `max_dontsend_messages` | The maximum number of `DONTSEND` messages per heartbeat per peer | ??? | +| Parameter | Description | Reasonable Default | +|--------------------------|------------------------------------------------------------------|--------------| +| `max_idontwant_messages` | The maximum number of `IDONTWANT` messages per heartbeat per peer | ??? | -## DONTSEND Message +## IDONTWANT Message ### Basic scenario When the peer receives the first message instance it immediately broadcasts -(not queue for later piggybacking) `DONTSEND` with the `messageId` to all its mesh peers. +(not queue for later piggybacking) `IDONTWANT` with the `messageId` to all its mesh peers. This could be performed prior to the message validation to further increase the effectiveness of the approach. -On the other side a node maintains per-peer `dont_send_message_ids` set. Upon receiving `DONTSEND` from +On the other side a node maintains per-peer `dont_send_message_ids` set. Upon receiving `IDONTWANT` from a peer the `messageId` is added to the `dont_send_message_ids` set. When later relaying the `messageId` message to the mesh the peers found in `dont_send_message_ids` could be skipped. Old entries from `dont_send_message_ids` could be pruned during heartbeat processing. The prune strategy is outside of the spec scope and can be decided by implementations. -`DONTSEND` message is supposed to be _optional_ for both receiver and sender. I.e. the sender may or may not utilize -this message. The receiver in turn may ignore `DONTSEND`: sending a message after the corresponding `DONTSEND` +`IDONTWANT` message is supposed to be _optional_ for both receiver and sender. I.e. the sender may or may not utilize +this message. The receiver in turn may ignore `IDONTWANT`: sending a message after the corresponding `IDONTWANT` should not be penalized. -The `DONTSEND` may have negative effect on small messages as it may increase the overall traffic and CPU load. -Thus it is better to utilize `DONTSEND` for messages of a larger size. -The exact policy of `DONTSEND` appliance is outside of the spec scope. Every implementation may choose whatever -is more appropriate for it. Possible options are either choose a message size threshold and broadcast `DONTSEND` -on per message basis when the size is exceeded or just use `DONTSEND` for all messages on selected topics. +The `IDONTWANT` may have negative effect on small messages as it may increase the overall traffic and CPU load. +Thus it is better to utilize `IDONTWANT` for messages of a larger size. +The exact policy of `IDONTWANT` appliance is outside of the spec scope. Every implementation may choose whatever +is more appropriate for it. Possible options are either choose a message size threshold and broadcast `IDONTWANT` +on per message basis when the size is exceeded or just use `IDONTWANT` for all messages on selected topics. -To prevent DoS the number of `DONTSEND` control messages is limited to `max_dontsend_messages` per heartbeat +To prevent DoS the number of `IDONTWANT` control messages is limited to `max_idontwant_messages` per heartbeat ### Relying on `IHAVE`s @@ -69,13 +69,13 @@ peers that it don't want that message from them. ### Sending `IHAVE` to mesh peers who choked that particular message Reasonable addition to the later scenario would be to _immediately_ send `IHAVE` instead of a full message -to those mesh peers who reported `DONTSEND`. That would notify mesh peers that the node has this message +to those mesh peers who reported `IDONTWANT`. That would notify mesh peers that the node has this message and they could request it from you in case their `IWANT` requests fail in the previous scenario ### Cancelling `IWANT` If a node requested a message via `IWANT` and then occasionally receives the message from other peer it may -try to cancel its `IWANT` requests with the corresponding `DONTSEND` message. It may work in cases when a +try to cancel its `IWANT` requests with the corresponding `IDONTWANT` message. It may work in cases when a peer delays/queues `IWANT` requests and the `IWANT` request would be removed from the queue if not processed yet ## Protobuf Extension @@ -91,10 +91,10 @@ message RPC { message ControlMessage { // messages from v1.0 - repeated ControlDontSend dontSend = 5; + repeated ControlIDontWant iDontWant = 5; } -message ControlDontSend { +message ControlIDontWant { required bytes messageID = 1; } From c6cfac969ed622cc91af679e028e84832be293cc Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 31 May 2023 15:33:10 +0400 Subject: [PATCH 03/10] Use the proper SHOULD and MAY wordings --- pubsub/gossipsub/gossipsub-v1.2.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index 846362700..f77a1352e 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -43,18 +43,18 @@ This could be performed prior to the message validation to further increase the On the other side a node maintains per-peer `dont_send_message_ids` set. Upon receiving `IDONTWANT` from a peer the `messageId` is added to the `dont_send_message_ids` set. -When later relaying the `messageId` message to the mesh the peers found in `dont_send_message_ids` could be skipped. +When later relaying the `messageId` message to the mesh the peers found in `dont_send_message_ids` SHOULD be skipped. -Old entries from `dont_send_message_ids` could be pruned during heartbeat processing. +Old entries from `dont_send_message_ids` SHOULD be pruned during heartbeat processing. The prune strategy is outside of the spec scope and can be decided by implementations. -`IDONTWANT` message is supposed to be _optional_ for both receiver and sender. I.e. the sender may or may not utilize -this message. The receiver in turn may ignore `IDONTWANT`: sending a message after the corresponding `IDONTWANT` +`IDONTWANT` message is supposed to be _optional_ for both receiver and sender. I.e. the sender MAY NOT utilize +this message. The receiver in turn MAY ignore `IDONTWANT`: sending a message after the corresponding `IDONTWANT` should not be penalized. The `IDONTWANT` may have negative effect on small messages as it may increase the overall traffic and CPU load. Thus it is better to utilize `IDONTWANT` for messages of a larger size. -The exact policy of `IDONTWANT` appliance is outside of the spec scope. Every implementation may choose whatever +The exact policy of `IDONTWANT` appliance is outside of the spec scope. Every implementation MAY choose whatever is more appropriate for it. Possible options are either choose a message size threshold and broadcast `IDONTWANT` on per message basis when the size is exceeded or just use `IDONTWANT` for all messages on selected topics. From 9f9314ca82488129ec539451314deb3fdd8d3959 Mon Sep 17 00:00:00 2001 From: vyzo Date: Fri, 14 Jul 2023 21:29:50 +0300 Subject: [PATCH 04/10] gossipsub v1.2 scaffolding --- pubsub/gossipsub/gossipsub-v1.2.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pubsub/gossipsub/gossipsub-v1.2.md diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md new file mode 100644 index 000000000..ad07c5bac --- /dev/null +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -0,0 +1,19 @@ +# gossipsub v1.2: TODO + +| Lifecycle Stage | Maturity | Status | Latest Revision | +|-----------------|---------------------------|--------|-----------------| +| 1A | Working Draft | Active | r1, 2023-07-14 | + +Authors: + +Interest Group: [@vyzo] + +[@vyzo]: https://github.com/vyzo + +See the [lifecycle document][lifecycle-spec] for context about maturity level and spec status. + +[lifecycle-spec]: https://github.com/libp2p/specs/blob/master/00-framework-01-spec-lifecycle.md + +## Overview + +TODO From 70642035acee6306662c085a87c3a9944f886db3 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Mon, 17 Jul 2023 12:18:53 +0400 Subject: [PATCH 05/10] Add authors and interest group members --- pubsub/gossipsub/gossipsub-v1.2.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index 646cdf12e..61305a74d 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -4,11 +4,13 @@ |-----------------|---------------------------|--------|-----------------| | 1A | Working Draft | Active | r1, 2023-07-14 | -Authors: +Authors: [@Nashatyrev], [@Menduist] -Interest Group: [@vyzo] +Interest Group: [@vyzo], [@Nashatyrev], [@Menduist] [@vyzo]: https://github.com/vyzo +[@Nashatyrev]: https://github.com/Nashatyrev +[@Menduist]: https://github.com/Menduist See the [lifecycle document][lifecycle-spec] for context about maturity level and spec status. From 31706a846223c236a311ed5704a005bcb0683cf1 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Thu, 7 Sep 2023 16:13:56 +0400 Subject: [PATCH 06/10] Remove potential use cases which may potentially lead to amplification --- pubsub/gossipsub/gossipsub-v1.2.md | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index 61305a74d..bd904115b 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -76,18 +76,6 @@ on per message basis when the size is exceeded or just use `IDONTWANT` for all m To prevent DoS the number of `IDONTWANT` control messages is limited to `max_idontwant_messages` per heartbeat -### Relying on `IHAVE`s - -Another potential additional strategy could be as follows. If a node receives `IHAVE` (from one or more peers) -before the message is appeared in the mesh the node may request the message with `IWANT` and notify all mesh -peers that it don't want that message from them. - -### Sending `IHAVE` to mesh peers who choked that particular message - -Reasonable addition to the later scenario would be to _immediately_ send `IHAVE` instead of a full message -to those mesh peers who reported `IDONTWANT`. That would notify mesh peers that the node has this message -and they could request it from you in case their `IWANT` requests fail in the previous scenario - ### Cancelling `IWANT` If a node requested a message via `IWANT` and then occasionally receives the message from other peer it may From 5c946efda4a0a22e4ca2d86bcacdebf4005532c4 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Thu, 7 Sep 2023 16:14:42 +0400 Subject: [PATCH 07/10] Minor rephrasing with MAY/SHOULD --- pubsub/gossipsub/gossipsub-v1.2.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index bd904115b..1d9925cce 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -78,9 +78,9 @@ To prevent DoS the number of `IDONTWANT` control messages is limited to `max_ido ### Cancelling `IWANT` -If a node requested a message via `IWANT` and then occasionally receives the message from other peer it may +If a node requested a message via `IWANT` and then occasionally receives the message from other peer it MAY try to cancel its `IWANT` requests with the corresponding `IDONTWANT` message. It may work in cases when a -peer delays/queues `IWANT` requests and the `IWANT` request would be removed from the queue if not processed yet +peer delays/queues `IWANT` requests and the `IWANT` request SHOULD be removed from the queue if not processed yet ## Protobuf Extension From b9202a8aeebbbd252d432818736a24f26db1b7b8 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 10 Jan 2024 14:46:20 +0400 Subject: [PATCH 08/10] Change SHOULD to MUST according to discussion in comments --- pubsub/gossipsub/gossipsub-v1.2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index 1d9925cce..5dec4696c 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -59,7 +59,7 @@ This could be performed prior to the message validation to further increase the On the other side a node maintains per-peer `dont_send_message_ids` set. Upon receiving `IDONTWANT` from a peer the `messageId` is added to the `dont_send_message_ids` set. -When later relaying the `messageId` message to the mesh the peers found in `dont_send_message_ids` SHOULD be skipped. +When later relaying the `messageId` message to the mesh the peers found in `dont_send_message_ids` MUST be skipped. Old entries from `dont_send_message_ids` SHOULD be pruned during heartbeat processing. The prune strategy is outside of the spec scope and can be decided by implementations. From 1fdaeb81773c3cb82357cf4eabcf9f61d382008c Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Fri, 23 Feb 2024 15:38:10 +0300 Subject: [PATCH 09/10] Make the ControlIDontWant.messageIDs plural to be consistent with IHAVE and IWANT messages Co-authored-by: Pop Chunhapanya --- pubsub/gossipsub/gossipsub-v1.2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index 5dec4696c..75b92e7a7 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -99,7 +99,7 @@ message ControlMessage { } message ControlIDontWant { - required bytes messageID = 1; + repeated bytes messageIDs = 1; } ``` From c7dfbe00294dc7fcaa654088ac673dd205cf54f1 Mon Sep 17 00:00:00 2001 From: Anton Nashatyrev Date: Wed, 20 Mar 2024 09:30:52 +0300 Subject: [PATCH 10/10] Change filed name to lower case for consistency with original spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: João Oliveira --- pubsub/gossipsub/gossipsub-v1.2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index 75b92e7a7..e81ded5fc 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -95,7 +95,7 @@ message RPC { message ControlMessage { // messages from v1.0 - repeated ControlIDontWant iDontWant = 5; + repeated ControlIDontWant idontwant = 5; } message ControlIDontWant {