From f7e4291ae2bea31da3e52251f8b00588849b8eed Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 16 Feb 2021 17:35:47 -0800 Subject: [PATCH 1/5] Reliable Mutability Primitive pitch --- proposals/Reliable Mutability Primitive.md | 162 +++++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 proposals/Reliable Mutability Primitive.md diff --git a/proposals/Reliable Mutability Primitive.md b/proposals/Reliable Mutability Primitive.md new file mode 100644 index 00000000..1bdc629b --- /dev/null +++ b/proposals/Reliable Mutability Primitive.md @@ -0,0 +1,162 @@ + +# Reliable Mutability Primitive + +### Purpose & impact +##### Background & intent +_Describe the desired state of the world after this project? Why does that matter?_ + + + +Today ecosystem has to work around the lack of functional mutability primitives. [IPNS][] does not provide reliable solution forcing community to adopt different custom solutions. + +- [Pinata][] gets asked about how to chaneg content for CID onec every few weeks _(Most of the time they find that is not what the users actually need)_. +- [fission.codes](https://fission.codes/) found IPNS persistance unreliable and had to build custom solution that updates [DNSLink][] records (**TODO**: Get a quote / link from Boris) +- **TODO**: Survey textile, last time we talked they found IPNS to be unreliable for propagating thread updates. +- ... +- Protocol Labs uses dns simple API to [publish site updates](https://github.com/filecoin-project/specs/blob/71f37208a1f4f56b33ea307d7cbdb4b06996b115/.github/workflows/main.yml#L40). + + +This gap in the stack makes building software which runs on end user computers and meets modern expectations more difficult than traditional hosted solutions. This creates a hardle for adoption of the technology. + + + +##### Assumptions & hypotheses +_What must be true for this project to matter?_ + + +- Applications want to provide a canonical address for a current state. +- Application authors will prefer built-in mutability primitive over custom solutions. +- Developed solution would provide simpler mechanism for publishing updates (than DNSLink updates) +- Developed solution would work provide reliable mechanism for producing updates on not always connected end user computers (think browsers, phones) +- Solution should not require shipping secret keys or tokens with devices. + +##### User workflow example +_How would a developer or user use this new capability?_ + + +Team builds an application enabling user to write and share notes with each other. They chose to use IPFS as that enabled them to focus on the applications writing exprience free from having to write and maintain backend infrastructure or ensuring data safety. Application stores encrypted notes in the embedded IPFS node and publishes CID to the latest shared notes DAG root via IPNS API. This enables collaborating users to fetch updated notes via embedde IPFS. Application also leverage [Pinning Services API][], enabling their users to choose from growing number of service providers or to bring their own endpoint for better availability. + +Choosing IPFS not only reduced an engineering effort, but also made their product fully off-grid capable (thanks to IPNS update propagation over mDNS) with no additional effort on their end. Team often likes to points out that their product has an edge over the competition, not only all user e2e encrypted, but their users can also choose where that data is persisted. + +##### Impact +_How directly important is the outcome to our top-level mission?_ + + +🔥🔥🔥 This would enable building competitive software that does not introduce custom server components (Standard [Pinning Services API][] provides data persistance and standard mutability primitive provides updating mechanism). + +It would also give an edge to the whole ecosystem, because [there is no step three](https://www.youtube.com/watch?v=YHzM4avGrKI) (as in no servers to run or maintain). + + + + +##### Leverage +_How much would nailing this project improve our knowledge and ability to execute future projects?_ + + + + +🎯🎯 - This would enable projects that need to have a canonical identifier for evolving dataset. It would also provide a greater interop across products built around IPFS by removing need for custom incompatibile solutions that teams need to implemente today. + + +##### Confidence +_How sure are we that this impact would be realized? Label from [this scale](https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15)_. + + + +3 - As high confidence as we can have without doing actual user studies. We know teams that have tried IPNS but found it unrelaible, ultimately rolling out an alternative solution. We also know teams that have evaluated IPFS but chose alternative due to lack of reliable mutability story. + + + +### Project definition +##### Brief plan of attack + + + +- Implement third party IPNS republishing (TODO: Find a relevant issue link) +- Collaborate with Pinning Services to make this part of the API. +- Fix IPNS publishing in browsers. +- Enable pubsub by default + IPNS over pubsub. +- Implement & deploy strategic republishing system on PL hosted nodes that will republish popular content and drop stalled one. So that IPNS works even thought pinning services. +- (Nice to have) Implement and rollout delegated publishing to enable updating from multiple devices. +- (Nice to have) Implement and rollout versioning system so that nodes can see full update history. + +##### What does done look like? +_What specific deliverables should completed to consider this project done?_ + +Publishing IPNS record takes just a few miliseconds. Shutting laptop and loading `https://ipfs.io/ipns/{key}` week later on other device loads the content published as fast as loading corresponding `https://ipfs.io/ipfs/{cid}`. + + +##### What does success look like? +_Success means impact. How will we know we did the right thing?_ + + +People and teams are storing IPNS keys in [DNSLink][] records as opposed to CIDs, because that provides a more effective and simpler way to publish updates. We see new projects are leveraging IPNS over working around it via things like [DNSLink][]. + + +##### Counterpoints & pre-mortem +_Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ + +- IPNS has bad reputation, changing it may prove to be a challenge. +- Teams may find that general primitive is not adequate for custom needs. +- Required key management (e.g. key recovery, key compromise) still may prove it impractical. +- Higher level alternatives like textile threads may prove to be better layer of abstraction. + +##### Alternatives +_How might this project’s intent be realized in other ways (other than this project proposal)? What other potential solutions can address the same need?_ + +- Integrating textile threads as core component of the ecosystem. +- Standardizing DNS record updates mechanism and intergating into IPFS. +- Work with Pinning Services to implement stardized DNSLink updating mechanism. + +##### Dependencies/prerequisites + + +- Functioning IPNS in browsers +- Enabling pubsub in go-ipfs + +##### Future opportunities + + +- [Petname][] system for memorable but non canonical names (kind of how our phones allow us to not remember numbers). +- Subscribtion based deny lists (node can subscribe to any endpoint) +- RSS like systems (In RSS readers memorable namse were not a concern) +- DAGs revisions carrying IPNS key so that last latest revision or any past revision could be discovered. + +### Required resources + + +##### Effort estimate + + +Large. Requires lot of coordinated changes across implementations and collabs. Uncertainty in the scope until concrete actionable plan is in place. Uncertenty in collab buy-in. + +##### Roles / skills needed + + +- go-ipfs development +- js-ipfs development +- cross org coordinator (to get pinning services onboard) + +[Pinata]:https://pinata.cloud/ +[DNSLink]:https://docs.ipfs.io/concepts/dnslink/ +[IPNS]:https://docs.ipfs.io/concepts/ipns/ +[Pinning Services API]:https://ipfs.github.io/pinning-services-api-spec/ +[Petname]:https://en.wikipedia.org/wiki/Petname From d87bd834aed9df78d4f4dcd2aade4aacbb851114 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Wed, 17 Feb 2021 09:03:58 -0800 Subject: [PATCH 2/5] Apply suggestions from code review Co-authored-by: Rod Vagg Co-authored-by: Alan Shaw --- proposals/Reliable Mutability Primitive.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/proposals/Reliable Mutability Primitive.md b/proposals/Reliable Mutability Primitive.md index 1bdc629b..5322f315 100644 --- a/proposals/Reliable Mutability Primitive.md +++ b/proposals/Reliable Mutability Primitive.md @@ -1,8 +1,12 @@ +# Reliable Mutability Primitive + +Authors: [@gozala](https://github.com/gozala) + +Initial PR: https://github.com/protocol/web3-dev-team/pull/19/ -# Reliable Mutability Primitive ### Purpose & impact ##### Background & intent @@ -14,14 +18,14 @@ Outline the status quo, including any relevant context on the problem you’re s Today ecosystem has to work around the lack of functional mutability primitives. [IPNS][] does not provide reliable solution forcing community to adopt different custom solutions. -- [Pinata][] gets asked about how to chaneg content for CID onec every few weeks _(Most of the time they find that is not what the users actually need)_. +- [Pinata][] gets asked about how to change content for CID once every few weeks _(Most of the time they find that is not what the users actually need)_. - [fission.codes](https://fission.codes/) found IPNS persistance unreliable and had to build custom solution that updates [DNSLink][] records (**TODO**: Get a quote / link from Boris) - **TODO**: Survey textile, last time we talked they found IPNS to be unreliable for propagating thread updates. - ... - Protocol Labs uses dns simple API to [publish site updates](https://github.com/filecoin-project/specs/blob/71f37208a1f4f56b33ea307d7cbdb4b06996b115/.github/workflows/main.yml#L40). -This gap in the stack makes building software which runs on end user computers and meets modern expectations more difficult than traditional hosted solutions. This creates a hardle for adoption of the technology. +This gap in the stack makes building software which runs on end user computers and meets modern expectations more difficult than traditional hosted solutions. This creates a hurdle for adoption of the technology. @@ -32,7 +36,7 @@ _What must be true for this project to matter?_ - Applications want to provide a canonical address for a current state. - Application authors will prefer built-in mutability primitive over custom solutions. - Developed solution would provide simpler mechanism for publishing updates (than DNSLink updates) -- Developed solution would work provide reliable mechanism for producing updates on not always connected end user computers (think browsers, phones) +- Developed solution would provide a reliable mechanism for producing updates on not always connected end user computers (think browsers, phones) - Solution should not require shipping secret keys or tokens with devices. ##### User workflow example From 059bc2982e844fe2da61ec060396a8554ebf3d88 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 23 Feb 2021 11:32:25 -0800 Subject: [PATCH 3/5] Update proposals/Reliable Mutability Primitive.md --- proposals/Reliable Mutability Primitive.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/Reliable Mutability Primitive.md b/proposals/Reliable Mutability Primitive.md index 5322f315..6241119d 100644 --- a/proposals/Reliable Mutability Primitive.md +++ b/proposals/Reliable Mutability Primitive.md @@ -85,7 +85,7 @@ _How sure are we that this impact would be realized? Label from [this scale](htt -- Implement third party IPNS republishing (TODO: Find a relevant issue link) +- Implement third party IPNS republishing (see [ipfs/go-ipfs#4435](https://github.com/ipfs/go-ipfs/issues/4435)) - Collaborate with Pinning Services to make this part of the API. - Fix IPNS publishing in browsers. - Enable pubsub by default + IPNS over pubsub. From e7d25420f5e5b6db2e3624ea897d6295cc217d6a Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 23 Feb 2021 11:45:57 -0800 Subject: [PATCH 4/5] Update proposals/Reliable Mutability Primitive.md --- proposals/Reliable Mutability Primitive.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/Reliable Mutability Primitive.md b/proposals/Reliable Mutability Primitive.md index 6241119d..b80bb70e 100644 --- a/proposals/Reliable Mutability Primitive.md +++ b/proposals/Reliable Mutability Primitive.md @@ -105,7 +105,9 @@ _Success means impact. How will we know we did the right thing?_ -People and teams are storing IPNS keys in [DNSLink][] records as opposed to CIDs, because that provides a more effective and simpler way to publish updates. We see new projects are leveraging IPNS over working around it via things like [DNSLink][]. +- People and teams are storing IPNS keys in [DNSLink][] records as opposed to CIDs, because that provides a more effective and simpler way to publish updates. +- We see people storing IPNS addresses in ENS to save on blockchain transactions costs. +- We see new projects leveraging IPNS (when human readable name is not a concern) instead of working around it wit [DNSLink][]. ##### Counterpoints & pre-mortem From 68055dd0451ed0d119809df3e78ab8e9e531a014 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Mon, 1 Mar 2021 21:37:51 -0800 Subject: [PATCH 5/5] Apply suggestions from code review Co-authored-by: Marcin Rataj --- proposals/Reliable Mutability Primitive.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/proposals/Reliable Mutability Primitive.md b/proposals/Reliable Mutability Primitive.md index b80bb70e..da0e5bba 100644 --- a/proposals/Reliable Mutability Primitive.md +++ b/proposals/Reliable Mutability Primitive.md @@ -96,7 +96,12 @@ _How sure are we that this impact would be realized? Label from [this scale](htt ##### What does done look like? _What specific deliverables should completed to consider this project done?_ -Publishing IPNS record takes just a few miliseconds. Shutting laptop and loading `https://ipfs.io/ipns/{key}` week later on other device loads the content published as fast as loading corresponding `https://ipfs.io/ipfs/{cid}`. +- Publishing IPNS record takes just a few miliseconds. +- Shutting laptop and loading `https://ipfs.io/ipns/{key}` week later on other device loads the content published as fast as loading corresponding `https://ipfs.io/ipfs/{cid}`. +- It is possible to pin IPNS address to keep it around and ensure it resolves even when original publisher is gone (AKA solving IPNS rot). + - Node follows IPNS record updates and keeps re-publishing latest one if original author disappears. + - Node follows content updates behind the IPNS record, and update content pin every time IPNS record changes. + - It is possible to opt-out from this behavior to re-publish IPNS record for `en.wikipedia-on-ipfs.org` without pinning all the data. ##### What does success look like? @@ -107,7 +112,8 @@ Provide success criteria. These might include particular metrics, desired change --> - People and teams are storing IPNS keys in [DNSLink][] records as opposed to CIDs, because that provides a more effective and simpler way to publish updates. - We see people storing IPNS addresses in ENS to save on blockchain transactions costs. -- We see new projects leveraging IPNS (when human readable name is not a concern) instead of working around it wit [DNSLink][]. +- We see new projects leveraging IPNS (when human readable name is not a concern) instead of working around it with [DNSLink][]. +- We see more requests for `/ipns/{libp2p-key}` paths on our public gateway (dweb.link & ipfs.io). ##### Counterpoints & pre-mortem @@ -135,6 +141,7 @@ _How might this project’s intent be realized in other ways (other than this pr - [Petname][] system for memorable but non canonical names (kind of how our phones allow us to not remember numbers). +- IPNS-based autoupdate mechanisms for our own stack (eg. go-ipfs, ipfs-desktop etc) - Subscribtion based deny lists (node can subscribe to any endpoint) - RSS like systems (In RSS readers memorable namse were not a concern) - DAGs revisions carrying IPNS key so that last latest revision or any past revision could be discovered.