Lotus Light mode #40
Lotus Light mode #40
Conversation
magik6k
force-pushed
the
proposal/lotus-light
branch
from
b39a1cc
to
0cd93fd
Compare
| #### Background & intent | ||
|
|
||
| Currently, running lotus nodes requires dedicating lots of resources. Users should | ||
| be able to talk to the blockchain without validating it fully |
There was a problem hiding this comment.
| be able to talk to the blockchain without validating it fully | |
| be able to talk to the blockchain without validating it fully. |
| Lotus would be much easier to run for non-miner users | ||
|
|
||
| #### Leverage | ||
| _How much would nailing this project improve our knowledge and ability to execute future projects?_ |
There was a problem hiding this comment.
| _How much would nailing this project improve our knowledge and ability to execute future projects?_ | |
| The current difficulty of running a Lotus node is a frequent painpoint for developers and users. The typical solution is to abstract Lotus away behind some service, but we stand to gain a lot from also lowering the barrier to entry to the Filecoin network itself. This would reduce the space reqs needed to be connected to the network (no more running out of disk space every few weeks), the time to get started (no more gigabytes of data to download), and the computing power needed (no more hearing your laptop make angry sounds when it runs Lotus). |
| --> | ||
|
|
||
| #### Confidence | ||
| _How sure are we that this impact would be realized? Label from [this scale](https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15)_. |
There was a problem hiding this comment.
| _How sure are we that this impact would be realized? Label from [this scale](https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15)_. | |
| - Medium impact: It doesn't transform the user experience, but it is at the very heart of interacting with the network (running Lotus) | |
| - High confidence: We know that the difficulty of running Lotus is a barrier to entry for many | |
| - Medium ease: It's a non-trivial amount of work, but it is low-risk -- the probability that we run into some insurmountable challenge is basically zero, since most obstacles will just need some refactoring to be overcome. |
| gigabytes of data, and having to dedicate significant resources for the node | ||
|
|
||
| #### What does success look like? | ||
| _Success means impact. How will we know we did the right thing?_ |
There was a problem hiding this comment.
| _Success means impact. How will we know we did the right thing?_ | |
| - The number of people in our own team running Lotus increases thanks to the new light mode | |
| - Partners report fewer issues with getting started with the network |
| Describe any choices and uncertainty in this scope estimate. (E.g. Uncertainty in the scope until design work is complete, low uncertainty in execution thereafter.) | ||
| --> | ||
|
|
||
| Medium |
There was a problem hiding this comment.
Depending on "appropriate skills" this might even be a Small tbh.
| --> | ||
|
|
||
| #### Counterpoints & pre-mortem | ||
| * Even basic validation of the chain, pulling the state over Bitswap might still be very resource intensive |
There was a problem hiding this comment.
- We should be able to pull actor state on-demand. Although we're likely going to need part of the power table state at every epoch. Without graphsync, this could be slow. It may be faster to just keep the entire power table in-sync assuming it's reasonably small.
- We can re-use the same session for all requests. That'll let us ask the same peers instead of broadcasting.
- We don't need messages.
Really, we want a pretty complex graphsync selector.
|
This depends on having a solid way to IPLD GC state doesn't it? I know @ribasushi has been doing reference counting with his custom block store (or something?) that might be useful for that. Or is there already an existing way to efficiently discard blocks not part of the current state tree without walking the whole lot every time a you get a new block and need to discard block+X's state? |
Not for the initial version. The dataset will grow, but not terribly fast.
A hot/cold store (riba's?) would cover this.
Although, in this case, there's a much easier approach: delete everything. That is, whenever we hit some storage limit, we can just delete everything and re-download anything we actually need. |
|
Worth noting - I have a hack-weekend POC of this idea, which works really well with caught-up mode, and within-finality sync, but is still computing state in some places, and is fetching all messages from the network when syncing (changing that requires some deeper changes in the syncer code). The POC patch is really short - https://gist.github.com/magik6k/4d0603dcaf266b300b69af89b0599235, but it proves that it's really feasible, even without graphsync |
@Stebalien Are you referring to the splitstore developed by @vyzo? |
|
All in this thread: please review the splitstore PR I linked above. It would probably benefit from package-level godocs, but it already does what seems to be needed (or at least enough, such that it can potentially be extended to accommodate further requirements). |
This is a read-through cache with a 24h window expiry. It doesn't solve:
(assuming this is what's actually needed) |
You're right, they're two different things.
Right. But we don't really need to keep that. Really, we don't need to keep anything. It would just be nice to keep blocks we've accessed within the last 24 hours because we're likely to use them again.
I probably was but that doesn't help here (as far as I can tell). From what I can tell, that just makes accessing the "hot" store faster. |
|
|
||
| Authors: @magik6k | ||
|
|
||
| Initial PR: TBD <!-- Reference the PR first proposing this document. Oooh, self-reference! --> |
There was a problem hiding this comment.
| Initial PR: TBD <!-- Reference the PR first proposing this document. Oooh, self-reference! --> | |
| Initial PR: https://github.com/protocol/web3-dev-team/pull/40 |
|
Per 2021-05-26 EM discussion, we're closing for now and will reevaluate the need here after miner-runtime rearchitecture: #98 |
No description provided.