Proposal: Reliable Mutability Primitive #19
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,168 @@ | ||||||
| # Reliable Mutability Primitive | ||||||
|
|
||||||
| Authors: [@gozala](https://github.com/gozala) | ||||||
|
|
||||||
| Initial PR: https://github.com/protocol/web3-dev-team/pull/19/ | ||||||
| <!-- | ||||||
Gozala marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| This template is intended to be used by those who would like to pitch a new project for one of the Web3 Dev project teams to take on. It should contain sufficient detail that others can understand how this project contributes to our team’s mission of product-market fit for our unified stack of protocols, what is included in scope of the project, where to get started if a project team were to take this on, and any other information relevant for prioritizing this project against others. | ||||||
| Good project scope aims for ~3-5 engineers for 1-3 months (though feel free to suggest larger-scoped projects anyway). Projects do not include regular day-to-day maintenance and improvement work, e.g. on testing, tooling, validation, code clarity, refactors for future capability, etc. | ||||||
| --> | ||||||
|
|
||||||
| ### Purpose & impact | ||||||
| ##### Background & intent | ||||||
| _Describe the desired state of the world after this project? Why does that matter?_ | ||||||
| <!-- | ||||||
| Outline the status quo, including any relevant context on the problem you’re seeing that this project should solve. Wherever possible, include pains or problems that you’ve seen users experience to help motivate why solving this problem works towards top-line objectives. | ||||||
| --> | ||||||
|
|
||||||
|
|
||||||
| Today ecosystem has to work around the lack of functional mutability primitives. [IPNS][] does not provide reliable solution forcing community to adopt different custom solutions. | ||||||
|
|
||||||
| - [Pinata][] gets asked about how to change content for CID once every few weeks _(Most of the time they find that is not what the users actually need)_. | ||||||
| - [fission.codes](https://fission.codes/) found IPNS persistance unreliable and had to build custom solution that updates [DNSLink][] records (**TODO**: Get a quote / link from Boris) | ||||||
| - **TODO**: Survey textile, last time we talked they found IPNS to be unreliable for propagating thread updates. | ||||||
|
There was a problem hiding this comment. I should also mention something else to provide further context: We're currently managing 1000s (yes, that's right) of IPNS updates for buckets. This currently takes an extremely long time to republish. We are actively conducting tests to determine exactly how long this takes, and how often it fails to complete. In some cases, it is taking hours per day. |
||||||
| - ... | ||||||
| - Protocol Labs uses dns simple API to [publish site updates](https://github.com/filecoin-project/specs/blob/71f37208a1f4f56b33ea307d7cbdb4b06996b115/.github/workflows/main.yml#L40). | ||||||
|
|
||||||
|
|
||||||
| This gap in the stack makes building software which runs on end user computers and meets modern expectations more difficult than traditional hosted solutions. This creates a hurdle for adoption of the technology. | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| ##### Assumptions & hypotheses | ||||||
| _What must be true for this project to matter?_ | ||||||
| <!--(bullet list)--> | ||||||
|
|
||||||
| - Applications want to provide a canonical address for a current state. | ||||||
| - Application authors will prefer built-in mutability primitive over custom solutions. | ||||||
| - Developed solution would provide simpler mechanism for publishing updates (than DNSLink updates) | ||||||
| - Developed solution would provide a reliable mechanism for producing updates on not always connected end user computers (think browsers, phones) | ||||||
| - Solution should not require shipping secret keys or tokens with devices. | ||||||
|
|
||||||
| ##### User workflow example | ||||||
| _How would a developer or user use this new capability?_ | ||||||
| <!--(short paragraph)--> | ||||||
|
|
||||||
| Team builds an application enabling user to write and share notes with each other. They chose to use IPFS as that enabled them to focus on the applications writing exprience free from having to write and maintain backend infrastructure or ensuring data safety. Application stores encrypted notes in the embedded IPFS node and publishes CID to the latest shared notes DAG root via IPNS API. This enables collaborating users to fetch updated notes via embedde IPFS. Application also leverage [Pinning Services API][], enabling their users to choose from growing number of service providers or to bring their own endpoint for better availability. | ||||||
|
|
||||||
| Choosing IPFS not only reduced an engineering effort, but also made their product fully off-grid capable (thanks to IPNS update propagation over mDNS) with no additional effort on their end. Team often likes to points out that their product has an edge over the competition, not only all user e2e encrypted, but their users can also choose where that data is persisted. | ||||||
|
|
||||||
| ##### Impact | ||||||
| _How directly important is the outcome to our top-level mission?_ | ||||||
|
|
||||||
|
|
||||||
| 🔥🔥🔥 This would enable building competitive software that does not introduce custom server components (Standard [Pinning Services API][] provides data persistance and standard mutability primitive provides updating mechanism). | ||||||
|
|
||||||
| It would also give an edge to the whole ecosystem, because [there is no step three](https://www.youtube.com/watch?v=YHzM4avGrKI) (as in no servers to run or maintain). | ||||||
|
|
||||||
|
|
||||||
| <!-- | ||||||
| Explain why you have chosen this rating | ||||||
| What awesome potential impact/outcomes/results will we see if we nail this project? | ||||||
| --> | ||||||
|
|
||||||
| ##### Leverage | ||||||
| _How much would nailing this project improve our knowledge and ability to execute future projects?_ | ||||||
|
|
||||||
|
|
||||||
| <!-- Explain the opportunity or leverage point for our subsequent velocity/impact (e.g. by speeding up development, enabling more contributors, etc) | ||||||
| --> | ||||||
|
|
||||||
| 🎯🎯 - This would enable projects that need to have a canonical identifier for evolving dataset. It would also provide a greater interop across products built around IPFS by removing need for custom incompatibile solutions that teams need to implemente today. | ||||||
|
|
||||||
|
|
||||||
| ##### Confidence | ||||||
| _How sure are we that this impact would be realized? Label from [this scale](https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15)_. | ||||||
|
|
||||||
| <!--Explain why this rating--> | ||||||
|
|
||||||
| 3 - As high confidence as we can have without doing actual user studies. We know teams that have tried IPNS but found it unrelaible, ultimately rolling out an alternative solution. We also know teams that have evaluated IPFS but chose alternative due to lack of reliable mutability story. | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| ### Project definition | ||||||
| ##### Brief plan of attack | ||||||
|
|
||||||
| <!--Briefly describe the milestones/steps/work needed for this project--> | ||||||
|
|
||||||
| - Implement third party IPNS republishing (see [ipfs/go-ipfs#4435](https://github.com/ipfs/go-ipfs/issues/4435)) | ||||||
| - Collaborate with Pinning Services to make this part of the API. | ||||||
| - Fix IPNS publishing in browsers. | ||||||
|
There was a problem hiding this comment. (New person comment): is this essential for unblock the core use-case of enabling customers to do ipns updates instead of DNS updates? There was a problem hiding this comment. Yes if those updates were to made from browser. It is also however useful on it's own merit as it enables apps to pull updated content even if such content has no human readable name. |
||||||
| - Enable pubsub by default + IPNS over pubsub. | ||||||
|
There was a problem hiding this comment. Note: Textile threads utilize pubsub which means that any issues with enabling pubsub by default are things that would be experienced by Textile threads users. They also utilize IPNS over PubSub for buckets. |
||||||
| - Implement & deploy strategic republishing system on PL hosted nodes that will republish popular content and drop stalled one. So that IPNS works even thought pinning services. | ||||||
|
There was a problem hiding this comment. Interesting, I'm not sure this is really required as we can give IPNS the same guarantees as IPFS. However, since the amount of data is smaller this idea of a public IPNS pinning service seems pretty cool. |
||||||
| - (Nice to have) Implement and rollout delegated publishing to enable updating from multiple devices. | ||||||
|
There was a problem hiding this comment. This is sort of doable already "for free" if we have some way to pin/follow IPNS updates. I can just share the private key with my multiple devices via There are alternative approaches too, but figured I'd point this one out. There was a problem hiding this comment. Yeah I meant getting it to a state where we can comfortably tell users this is how you go about it. I do have some reservations towards key sharing, but that's probably discussion worth having when we get there. |
||||||
| - (Nice to have) Implement and rollout versioning system so that nodes can see full update history. | ||||||
|
|
||||||
| ##### What does done look like? | ||||||
| _What specific deliverables should completed to consider this project done?_ | ||||||
|
|
||||||
| Publishing IPNS record takes just a few miliseconds. Shutting laptop and loading `https://ipfs.io/ipns/{key}` week later on other device loads the content published as fast as loading corresponding `https://ipfs.io/ipfs/{cid}`. | ||||||
Gozala marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
|
|
||||||
| ##### What does success look like? | ||||||
| _Success means impact. How will we know we did the right thing?_ | ||||||
|
|
||||||
| <!-- | ||||||
| Provide success criteria. These might include particular metrics, desired changes in the types of bug reports being filed, desired changes in qualitative user feedback (measured via surveys, etc), etc. | ||||||
| --> | ||||||
| - People and teams are storing IPNS keys in [DNSLink][] records as opposed to CIDs, because that provides a more effective and simpler way to publish updates. | ||||||
| - We see people storing IPNS addresses in ENS to save on blockchain transactions costs. | ||||||
| - We see new projects leveraging IPNS (when human readable name is not a concern) instead of working around it wit [DNSLink][]. | ||||||
Gozala marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
|
|
||||||
| ##### Counterpoints & pre-mortem | ||||||
| _Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_ | ||||||
|
|
||||||
| - IPNS has bad reputation, changing it may prove to be a challenge. | ||||||
|
There was a problem hiding this comment. How do we get more data on this? Is this a question to PM's? There was a problem hiding this comment. I think it's well know fact, evidence been that teams have tried and ultimately end up going with DNSLink solution. I'm pretty sure PL did that too, but someone with longer history here might know better. We could also get specific collabs to comment here to either validate the claim or dispute it. I have invited folks from textile & fission to validate. We could reach out to more teams. @olizilla do you by a chance know if we used to have IPNS in DNSLink past or am I making wrong assumptions here. There was a problem hiding this comment. @BigLep good pulse check in this issue (2017–today): ipfs/kubo#3860 |
||||||
| - Teams may find that general primitive is not adequate for custom needs. | ||||||
|
There was a problem hiding this comment. Have we been presented any usecases where ipns (once it has a reliable mutability primitive) isn't sufficient? There was a problem hiding this comment.
I suspect that despite all this usable IPNS will address enough use cases to be worth it. That said we might find that a higher level primitive that address more use cases out of the box will provide a better market fit. |
||||||
| - Required key management (e.g. key recovery, key compromise) still may prove it impractical. | ||||||
| - Higher level alternatives like textile threads may prove to be better layer of abstraction. | ||||||
|
|
||||||
| ##### Alternatives | ||||||
| _How might this project’s intent be realized in other ways (other than this project proposal)? What other potential solutions can address the same need?_ | ||||||
|
|
||||||
| - Integrating textile threads as core component of the ecosystem. | ||||||
|
There was a problem hiding this comment. @carsonfarmer: Obviously I like this idea. But it would end up being something threads-like I think. A benefit here is that threads supports multi-writer updates, which I think its a feature lacking in IPNS right now. The downside is that threads "as is" won't scale. Additionally, the encryption layer in threads is too heavy handed right now for a general solution like this. Having said all that, these things are all fixable implementation details. |
||||||
| - Standardizing DNS record updates mechanism and intergating into IPFS. | ||||||
|
There was a problem hiding this comment. Is this even possible? Doesn't it depend on each DNS' API for mutating actions? There was a problem hiding this comment. Hypothetically speaking we can start a standardization effort form a committee and get enough big players onboard to get it done. I imagine there might be some value for DNS providers to make their products more accessible. |
||||||
| - Work with Pinning Services to implement stardized DNSLink updating mechanism. | ||||||
|
There was a problem hiding this comment. There was a problem hiding this comment. (new person coming up to speed): what would this look like? I think I'll be able to grok it with a tiny sketch or a mock sample code example. There was a problem hiding this comment. We have recently created pinning service API that allows you to add pinning services like pinata to your IPFS node and then remote pin things. Kind of like adding git remote and pushing changes to it. Idea here is to extend that API in a way that you could update DNSLink provided by the pinning service to the new CID through extended API. Kind of like when you push to remote it updates corresponding branch to new commit. In our case branch names would be DNSLinks, that you could update to point to new CID. @BigLep does this answer your question ? There was a problem hiding this comment. @Gozala : thanks - this helps. The only open area for me then is where is this DNSLink stored? Is this a DNSLink in a DNS record on a (sub)domain that the Pinning Service owns? Basically would the chain be There was a problem hiding this comment. @BigLep This is great question – it makes us think about all implications. To give you a real life example, let's say we want to host https://cid.ipfs.io on some pinning service (pin + take care of updating dnslink + TLS). (B) is easy. Pinning service simply give you a name on some domain they own. Fleek does that already, they give https://cid-utils-website.on.fleek.co/ for free + they set up DNSLink for those names (because they own the top domain). (A) means we want to delegate updates of DNS TXT record on
|
||||||
|
|
||||||
| ##### Dependencies/prerequisites | ||||||
| <!--List any other projects that are dependencies/prerequisites for this project that is being pitched.--> | ||||||
|
|
||||||
| - Functioning IPNS in browsers | ||||||
|
There was a problem hiding this comment. Note: likely includes an up-to-date IPNS over PubSub implementation There was a problem hiding this comment. @aschmahmann do you mean something like
Suggested change
Or a separate item ? I There was a problem hiding this comment. Well IPNS over PubSub has a few tweaks over just PubSub itself to give some reliability. It's specced here. @hugomrdias would probably know more about other work required to get IPNS functioning in a browser I just know that without the IPNS over PubSub changes in the spec that it won't be fast enough. There was a problem hiding this comment. Sorry for the basic question, but I'm still confused by what is meant by "functioning IPNS in browsers". What isn't working browsers today? There was a problem hiding this comment. I should probably capture this in the document. Currently pubsub is disabled in go-ipfs so all the IPNS records are published and resolved from DHT. Nodes in browsers do not have necessary capabilities to query/publish DHT. Instead they do publish and resolve IPNS records over pubsub. Result is fragmented network where go and js nodes see each others records. Furthermore because because there is not enough nodes on the network with pubsub enabled published records from browser don't always make to nodes subscribed to them. P.S.: Some of this may not be completely accurate, @hugomrdias @aschmahmann could you please confirm I'm not misrepresenting current state. |
||||||
| - Enabling pubsub in go-ipfs | ||||||
|
|
||||||
| ##### Future opportunities | ||||||
| <!--What future projects/opportunities could this project enable?--> | ||||||
|
|
||||||
| - [Petname][] system for memorable but non canonical names (kind of how our phones allow us to not remember numbers). | ||||||
Gozala marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| - Subscribtion based deny lists (node can subscribe to any endpoint) | ||||||
| - RSS like systems (In RSS readers memorable namse were not a concern) | ||||||
| - DAGs revisions carrying IPNS key so that last latest revision or any past revision could be discovered. | ||||||
|
|
||||||
|
There was a problem hiding this comment. Also, more advanced mutability systems could be built (e.g. where instead of a single public key, we can have M of N signers required on records). But it all starts from having the UX and process around pinning mutable names and having a performant and efficient mutability system (e.g. Persistent PubSub, like what IPNS over PubSub uses) |
||||||
| ### Required resources | ||||||
|
|
||||||
|
|
||||||
| ##### Effort estimate | ||||||
| <!--T-shirt size rating of the size of the project. If the project might require external collaborators/teams, please note in the roles/skills section below). | ||||||
| For a team of 3-5 people with the appropriate skills: | ||||||
| - Small, 1-2 weeks | ||||||
| - Medium, 3-5 weeks | ||||||
| - Large, 6-10 weeks | ||||||
| - XLarge, >10 weeks | ||||||
| Describe any choices and uncertainty in this scope estimate. (E.g. Uncertainty in the scope until design work is complete, low uncertainty in execution thereafter.) | ||||||
| --> | ||||||
|
|
||||||
| Large. Requires lot of coordinated changes across implementations and collabs. Uncertainty in the scope until concrete actionable plan is in place. Uncertenty in collab buy-in. | ||||||
|
There was a problem hiding this comment. True, although just getting things stable + performant in go-ipfs and web browsers would likely go along way while collabs conversations are underway. There was a problem hiding this comment. @Gozala : can we put some high-level estimates (in terms of dev weeks) around each item in the "Brief plan of attack" above? I think that will help give a bit more clarity on the size of this project. There was a problem hiding this comment. I'm afraid I'm not well qualified to do this. I've mostly put this proposal together from prior discussions I had with @aschmahmann and @hugomrdias (still in outsider capacity) in an attempt to rally folks who have much better understanding of internal details. Help trying to scope it out further, would be greatly appreciated. P.S: Me & @hugomrdias plan no going over this proposal tomorrow to flush it out further. |
||||||
|
|
||||||
| ##### Roles / skills needed | ||||||
| <!--Describe the knowledge/skill-sets and team that are needed for this project (e.g. PM, docs, protocol or library expertise, design expertise, etc.). If this project could be externalized to the community or a team outside PL's direct employment, please note that here.--> | ||||||
|
|
||||||
| - go-ipfs development | ||||||
| - js-ipfs development | ||||||
| - cross org coordinator (to get pinning services onboard) | ||||||
|
There was a problem hiding this comment. Sorry to be slow, but why is it required to get pinning services onboard? I'm not saying we should skip doing this, I just want to make sure I understand the inter-dependency. For example, in my head right now I understand that Pinning Services are critical for persistence but I'm less clear on their role in mutability. There was a problem hiding this comment. That is a valid point and calling out pinning services specifically might be projecting too far of how things would work. This was partially influenced by the fact that IPNS (at least in its current form) requires records to be republished otherwise they'll become unsolvable. Implicit assumption here, is that, unless there are motivated nodes to do this, system will end up impractical. Pinning services just seem well positions to do this given that they are motivated nodes keep content that IPNS will point to available (insert git branch analogy here). That said we could and probably should consider different strategies to meet reliability goals. There was a problem hiding this comment. In other words I think we still need someone to help coordinate efforts beyond just leading implementation efforts, to ensure that we do not end up with a system that is directly tied to PL infrastructure. I am assuming this involves getting other teams onboard. There was a problem hiding this comment. +1 on at least thinking about pinning services in the context of IPNS:
|
||||||
|
|
||||||
| [Pinata]:https://pinata.cloud/ | ||||||
| [DNSLink]:https://docs.ipfs.io/concepts/dnslink/ | ||||||
| [IPNS]:https://docs.ipfs.io/concepts/ipns/ | ||||||
| [Pinning Services API]:https://ipfs.github.io/pinning-services-api-spec/ | ||||||
| [Petname]:https://en.wikipedia.org/wiki/Petname | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add a note that this is part of the larger story in #42 ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure this would be accurate. I think there is a merit to reliable mutability primitive whether we decide to do human readable names or not.