Decoupling FIL+ term from storage marketplaces (FIP-0045)

[anorth]

Update: FIP draft in progress on #432

With some changes we’ll be making to support more open programmability on the FVM, we have an opportunity to extend the term associated with FIL+ data cap allocations, possibly indefinitely. This would mean that a provider provably storing a piece of useful data could enjoy the boosted power and rewards for much longer, independently of limitations of deals with a storage market actor.

This post is a starter for discussion around whether long, extensible terms for verified data are positive for the network, and what restrictions or caveats we might need. I’ll follow up with concrete technical design soon.

Background

First, a brief overview of how the Filecoin Plus verified data program works today. The program’s operation is coupled with the built-in storage market actor.

• FIL+ notaries assign data cap to verified clients. Data cap is measured in bytes and is single-use.
• A verified client may indicate that a deal has verified data. In that case, when the deal is activated the corresponding data cap is consumed, and the deal’s space-time contributes to the “quality” of the sector in which it’s stored, increasing its reward.
• Clients and providers make deals via the built-in storage market actor. A deal specifies a term (start and end epoch).
• Each sector has a committed lifetime, which can be extended up to a 5 year maximum. The storage market implementation prohibits any deal with a term that exceeds the sector’s commitment. Today, there is no way to extend a deal if the sector into which the data is committed is also extended. There is no mechanism to transfer a deal (and hence its data cap) to another sector, thus no way to have any deal beyond the sector’s maximum life. Today, there is not even a way to extend a deal even if the sector’s life is extended longer than its initial commitment.
• Therefore, implicitly, each data cap allocation has a term bounded by a deal, in turn bounded by a sector’s maximum life (5 years). In practise the upper bound is 540 days, the maximum initial lifespan commitment for a sector.

So, the duration for which a FIL+ verified deal provides boosted rewards to a storage provider is limited by the deal term, in turn limited by the maximum sector lifespan for which we believe SDR PoRep is secure (i.e. a reason totally unrelated to FIL+ incentives). The Filecoin Plus data cap mechanism has no intrinsic notion of the term for which data cap, and hence the network-subsidised rewards, should be granted.

Protocol direction toward decoupling

The storage market actor’s prohibition of deal terms exceeding their sector’s commitment is a limitation of its simplified implementation. So is the inability to extend a deal. We would like to resolve both of these, and could do so by allowing the deal’s data to be moved/added to a new sector with a farther-out expiration. We could do so if not for some complications with Filecoin Plus quality-adjusted power. But there’s not technical constraint on arbitrarily long deal terms.

Separately, but pushing in the same direction, we are planning architectural changes to open up the platform for development of alternative market implementations on the FVM. Other markets might structure deals quite differently, and the built-in one should not have a monopoly on brokering FIL+ verified data. Thus, the verified registry actor will be decoupled from the built-in storage market so that verified data is accessible to all markets.

Each of these shifts—either fixing the built-in market or allowing new markets—would result in a FIL+ data cap allocation effectively having an indefinite term. Data cap has no intrinsic term, and the constraints on built-in storage market deal terms would be either resolved or irrelevant.

Network-subsidised data

I think that this decoupling, and subsequent realisation of arbitrary terms for verified data, is a positive change. It simplifies and clarifies many things. We can lean in to the potentially-indefinite term for verified data and make it a feature of the network.

By decoupling verified data cap from the built-in market, and tracking allocations of data cap to pieces of data explicitly in the verified registry actor, we can view a FIL+ allocation as an explicit data subsidy. The network will fund the ongoing, proveable storage of verified pieces through increased reward, independently of any client wishing to pay an additional incentive. Data that a verified client has blessed remains useful regardless of the terms or technical limitations of particular markets or of sector proof-of-replication algorithms, or even of the term for which a client wishes to pay extra for its storage. The Filecoin Plus program incentivises the storage of useful data on Filecoin and, if a verified client indicates, that data might be useful for far longer than a single sector’s lifetime, possibly indefinitely.

Simplicity

Minimising coupling is a key principle in software and systems engineering. Decoupling the verified data rewards from storage deals provides great opportunities for simplifying Filecoin’s core mechanics in order to provide a better platform for development.

One benefit we could immediately realise is simplifying the calculation of quality-adjusted power. The current mechanism, which is coupled to deal terms, imposes significant limits on the flexibility of sector storage, essentially rendering it write-once. Decoupling these neatly resolves the calculation difficulties to give full flexibility, and is much simpler than the Filecoin Plus premium proposal that also attempted this.

Proposed functionality

I’ll follow with a detailed design proposal in a FIP (work-in-progress here), but in the meantime here is a sketch of the properties I think we’d reach:

• A verified client allocates data cap to a piece of data and a provider, rather than a deal. The data cap allocation is independent of any storage markets.
• An allocation specifies a minimum commitment duration the provider must meet, which can be up to the maximum sector commitment (540 days). An allocation also specifies a maximum term, which can be arbitrarily larger than the minimum.
• The nominated provider can prove commitment of the data into a sector with a committed lifetime between the minimum and maximum verified data allocation term, and gain the boosted power and rewards immediately. As the sector approaches expiration, the provider can commit the same data into a new sector, with a longer expiration, and roll the power/reward boost into that new sector. They can repeat this until the allocation’s maximum term.
• The client and provider do not need to also do a deal via a storage market if the QA power rewards are sufficient compensation to the provider (e.g. for all zero-priced deals today). But they can do a deal too if the client wants to pay more.
• The verified registry notaries can set some policies that limit the maximum term a verified client can offer. Such a bound would provide a safety/backstop against unexpected system properties, but could be lifted in the future.

Stages to realisation

We might realise fully unlimited terms in a few stages, the latter of which need not land this year.

1. Decouple data cap from deals, and support data cap terms up to the maximum sector lifespan (5 years). Simplify the QA power calculations to award full 10x power to each byte of verified data immediately. Support retaining full boosted power and rewards when a sector’s lifespan is extended up to that maximum.
2. Support transfer of verified pieces into a new sector with the same provider, and a longer remaining lifetime. This would support an indefinite term, and unlock update-ability of sector data.
3. Support transfer of verified pieces to other providers (with client consent). A provider’s claim on a data cap allocation is a restricted-transfer non-fungible token.

Reasons for caution

There are some reasons to be cautious about this change, and I’d like to invite a full discussion. I can start with a few possible concerns here.

• Indefinitely-extended QA power over a long period could allow a provider to accumulate a large amount of consensus power relative to their storage commitment.
  • This is probably not an issue if many other providers also accumulate power through long-term storage of verified useful data. That’s probably healthy network behaviour.
  • The verified registry notaries could set a conservative maximum term for data cap allocations to limit term extension until we’re confident the behaviour is healthy.
• Data cap is too powerful if indefinite.
  • Each unit of data cap becomes less powerful with each incremental unit of data cap granted elsewhere. As verification procedures and automation improve, we might expect data cap to become increasingly abundant over time.
  • Verified registry limits could bound the long-term power initially.
• The network might end up carrying a bunch of data that’s not very useful.
  • In an environment of increasing abundance of storage capacity and/or demand, this should be a relatively minor concern. Eventually the subsidy earned by verified-but-not-useful data will fall below the opportunity cost of a paid-for storage deal; the situation can only last until the market-clearing price for a verified piece of data rises above zero.
  • Verified registry limits could bound the worst case, as we gain confidence
• Longer FIL+ terms reward early participants more than later ones.
  • This is true, but this reward to being early is usually seen as compensation for the risk of committing to a new network.
  • Abundant data cap will reward providers that commit to storing useful data even more than those who are early but less aligned.

Alternatives

If we don’t implement something like these bounds on explicit verified data allocation term, we would need to:

• Deny alternative storage deal markets/mechanisms on the FVM from brokering FIL+ data
• Refrain from implementing deal extension or transfer in the built-in market (for verified deals)

Or possibly there is another alternative model that would allow the flexibility we seek.

[jennijuju]

Great proposal!

How does this work for data/deal aggregation? - rn a couple data brokers / many clients aggregates smaller data pieces into a 32gib deal, so when we “assign datacap to the piece”, is it assigned to the aggregated whole piece, or datacap is tracked individually according to each small pieces?

[anorth]

With the current storage market, nothing will change there and so datacap must be for the top-level deal, which is the definition of a piece. After the architectural changes for programmable markets, what is a "deal" becomes very open and so tracking FIL+ for smaller pieces within a 32GB deal should become possible. It would kinda work against the efficiency gains of aggregation, though, to track many more smaller data cap allocations. This proposal doesn't magically help things scale smaller.

It will help things scale larger, though. One thing that I think we'll be able to build in is data cap allocations (and hence deal) larger than a sector.

[AxCortesCubero]

I was composing a comment, but it got way too long, so I made it its own document, here

https://hackmd.io/@R02mDHrYQ3C4PFmNaxF5bw/r1ITc9u-c

My main issue is in elaborating more on some of the potential problems you have already pointed out, in particular your last three "reasons for caution", and I add one which is that this proposal increases the incentive for SP's to seal for the shortest possible sector lifetimes. I also disagree that "Network-subsidised data" is a good thing we should do, or at least it is a separate question we could address intentionally, but if it can be avoided as a by product of FIL+ indefinite that would be good.

I propose and elaborate that the solution to all these issues is having a separate FIL+ collateral, that is not coupled to the sector, but corresponds to the FIL+ status of the data itself.

[anorth]

Thanks for taking the time to think about this and write a detailed comment. Some responses:

I would note that the first motivation here, in the title, is to decouple FIL+ from storage market deals. Currently, FIL+ status is restricted by initial deal terms, usually much shorter than sector lifetimes. Your premise "Currently FIL+ status is restraint by the sector lifetime" is not correct: in fact it would be a huge boon to reach that state.

The first stage under "stages to realisation" exactly addresses the first part of your document, tying FIL+ status to the lifetime of the sector into which the data is sealed. It introduces an "incentive to [maintain a sector] for the longest lifetime possible, if the sector contains FIL+ deals" that does not exist today. I would certainly advocate and expect that this first stage is implemented first, and policy discussion about the subsequent stages deferred until after that. Would you have a different response to only the first stage?

Even after (if) stage 2 is implemented, policy settings in the verified registry permit bounding the maximum FIL+ term. "Forever" is perhaps an unlikely value in practise. But coupling to sector lifetime seems somewhat arbitrary. Is 5 years exactly the right value? Maybe 3 or 7 is better. What about when we develop sealing tech that supports 20-year or 100-year sectors—now what's the right term limit?

I propose and elaborate that the solution to all these issues is having a separate FIL+ collateral

Your proposal is a subset of the FIL+ premium proposal, decoupling FIL+ term and pledge from sector lifetimes (while your proposal leaves rewards coupled to power). You didn't specify a mechanism for specifying FIL+ terms or accounting for the allocations and claims, so I assume it's the same as I've proposed here. I too think that parts of the FIL+ Premium proposal will still have merit after the initial decoupling of FIL+ from deal markets. The verified registry's explicit representation of claims will be a good place to do the accounting. But it would add a lot of complexity to try to roll it all in now. Again, perhaps you are only addressing the stage-2 onwards part?

I also disagree that "Network-subsidised data" is a good thing we should do

This isn't the right place for that discussion. This proposal takes the FIL+ program goals as given, and is oriented towards realising a better version of it. Data cap is intended to become abundant, and verified data dominate the use of capacity (with a corollary that the early-investor advantage will decrease significantly).

[AxCortesCubero]

Ok, the bulk of my comments were about the late-stage scenario where FIL+ is made "forever". My feelings are much milder about the first stage of making FIL+ sector bound to 5 years. I'll think more about that, but initially that doesn't seem like huge red flags for me.

For later stages if FIL+ goes independent of sector lifetimes, then I just think the default setting that makes sense to me is that it should have its own collateral independent of sector lifetimes too. It feels to me like this should be the "default" minimal choice, like I would like to hear some more arguments why this should not be the case.

[anorth]

Noting for followers that we're picking this up again soon. The next step is to prototype the implementation, which should help us find and resolve any uncertainties ahead of a well-defined FIP.

[jennijuju]

💙💙

[kaitlin-beegle]

Hi @anorth! Any update as to when a PR for this FIP draft will be opened?

[jennijuju]

Hi @anorth! Any update as to when a PR for this FIP draft will be opened?

Zen from lotus team will start prototyping later this month, and work with Alex to finalize the details. It may take 1-2 months for the whole to be fully figured out

[anorth]

As Jenn says. If we'd rather a FIP sooner, but with some chance of significant changes, I can write it sooner. But I was leaning towards waiting for implementation to reduce any uncertainty. That would mean that when published, we'd be concrete and also minimal implementation effort remaining.

That said, I'll probably start drafting soon anyway, to aid decisions about scope. There is a detailed design here: https://pl-strflt.notion.site/FIL-indefinite-term-limits-08079d7ec25c4cae839a3bf95a82df26

[kaitlin-beegle]

Thanks @anorth! Since implementers are already tracking this FIP, it's not too critical that a FIP draft be added.

For community visibility it's best to have a draft sooner rather than later, even if significant changes are necessary. It sounds like you're moving towards pushing one to the repo in the next few weeks, which sounds good. If there's any pressure to move faster I/we/someone will certainly let you know, but I don't expect this to be an issue.

[kuangzhiyu37]

We have the same wish that more active data should be introduced into the FIL network as all ,which will expand Filecoin ecosystem and make the real application landed.
However,with the limition of notary number and the highly centralised notaries' right of DC applications and approvals, we also extremely concerned that some notaries will unite for monopoly and manipulate the market,even profit through their right.
It breaks the original wish of decentralize network,and prevent more user and data into network.

[jennijuju]

Thanks for raising the flag! However, what you’ve described seems to be a FIP+ governance concern, and is not directly related to this FIP. I’d encourage you to repost here https://github.com/filecoin-project/notary-governance/discussions

[dkkapur]

@anorth thanks for bringing this one back! I'm supportive of the decoupling of the subsidy step. I have some concerns about this bit:

The nominated provider can prove commitment of the data into a sector with a committed lifetime between the minimum and maximum verified data allocation term, and gain the boosted power and rewards immediately. As the sector approaches expiration, the provider can commit the same data into a new sector, with a longer expiration, and roll the power/reward boost into that new sector. They can repeat this until the allocation’s maximum term.

In general, control over whether or not a piece of data is stored on Filecoin should be on the client / data owner side, not the side of the SP. I.e., there are use cases today where the data is only useful for n time, i.e., 1 year video backup, after which the data is not useful for the owner. Allowing an SP to continually extend its lifetime and serve it in a deal does not present a useful/valuable outcome. This is flagged in reasons for caution in the proposal, but I think this needs to be weighted higher and addressed from a philosophical standpoint of data ownership as well.

Secondly - what's the solution for if a CID is identified to be not worthy of DataCap? I.e., someone initially gets away with abusing the system to get their data verified, but future checks/retrieval sampling/etc. result in this being flagged as not worthy of receiving DataCap? Is there a path other than a FIP to remove the verification status from a piece?

[anorth]

In general, control over whether or not a piece of data is stored on Filecoin should be on the client / data owner side

The allocation's maximum term, which is specified by the client, prevents the SP gaining rewards for committing it any longer. The client has control, although terms shorter than the length of a profitable sector commitment won't be attractive to SPs.

what's the solution for if a CID is identified to be not worthy of DataCap

[Edit - misread the question the first time] This proposal doesn't change that. There is no proposal here for a mechanism to remove verification status from a piece. In general that would be quite complicated (although slightly less so after this decoupling). There might be a path to retro-actively reduce the TermMaximum for a claim, but it would only be checked when the sector in which a piece is currently committed expires. Supporting long sector commitments (#421) would limit the utility of that. And yes you'd need a FIP.

[dkkapur]

@anorth thanks - I also misread something in the proposal and your response helped my understanding. Thank you.

I still think it is worth us discussing removing verification status from a piece post this getting merged, but agree that it does not need to block this FIP from proceeding to implementation. We've talked about mechanisms to remove QAP from a sector in the past, and this feels like the natural evolution of that conversation with this FIP, and would be a very useful lever in ensuring Fil+ allocations can move faster in the future.

[Fatman13]

As the sector approaches expiration, the provider can commit the same data into a new sector, with a longer expiration,

How does commit same data into a new sector work? Do SPs snap the data to a new sector and send another message to the chain?

[anorth]

@Fatman13 yes, but I would note we probably won't have the actual methods hooked up to do that from day 1, due to complications in miner actor sector accounting. But #298 will realise it.

[tmellan]

Support transfer of verified pieces into a new sector with the same provider, and a longer remaining lifetime

@anorth can I confirm this will be possible for existing verified pieces, as well as new ones?

[anorth]

@tmellan after initial analysis, I'm pretty confident we won't be able to do a one-shot migration of all verified pieces into the new world, because some sectors would see a power drop from doing so. We might be able to do an opt-in migration, where a miner nominates sectors to migrate.

[anorth]

And, to clarify, I don't think we'll be able to support transfer into a new sector, but may be able to support extension of the sector already hosting the FIL+ deal, with the FIL+ power also being extended.

[jennijuju]

because some sectors would see a power drop from doing so.

could you please provide an example? 🙏

[geoff-vball]

@jennijuju (I think we talked about this yesterday, but throwing it here for others)

If a 1-year sector has a a 6-month fil+ deal in it, the power gets spread over the entire year. Applying the new rules to this sector after the deal expires would see the power of the sector drop to that of a CC Sector, which is unfair because they earned that power by storing the fil+ deal for 6 months already.

[jennijuju]

@tmellan fyi @geoff-vball and I have discussed this yesterday, and the above statement is not aarcurate, cuz with this fip they would actually earn more power across longer time.

[dannyob]

There's a lot to like about this proposal, but I wonder how it speaks to the tensions we're feeling in the Fil+ network, between acceptable deals, and acceptable data. For instance, notaries spend a lot of time trying to identify whether deals are appropriately determined, so that storage providers aren't simply creating a deal internally, in order to obtain datacap, so we try to ascertain that a deal is legitimately made with an external partner. Then there's appropriate data, which is the idea that there's some data that publicly beneficial for the Filecoin network to host. II believe aspects need to be in place to a greater or lesser degree for the process to work -- maybe it matters less if a SP has gone out and arranged the storage of data that is genuinely a public good. Alternatively, it's given that uploading data that is supposedly beneficial doesn't really mean anything if nobody wants it, perhaps genuine deals with external parties point to it being sufficiently "useful" data (part of the impetus for the Fil Enterprise project).

So my concern here is that this proposal leans a lot on the data itself being "useful", permanently, which is something it's pretty hard to determine in advance.

This may need a narrowing of the scope of what Fil+ notaries generally verify, which is fine and possibly a direction we should be going in -- but it also means that verified data becomes a much higher hurdle. The demands and current economics of the ecosystem point in the other direction, though: verified deals are now a very high percentage of our overall deals, mostly because the increase in power makes them extremely attractive -- and, while I haven't checked, for some SPs, at their current pricing, it may be the difference between a profitable deal and an unprofitable one.

If verified deals are a large percentage of total deals, now or in the future, then obtaining datacap -- very much especially permanent datacap! -- really does determine what level of return you get for your storage, and it goes without saying that as more datacapped content fills the network, not only does the value of particular datacap itself shrink (because it is an increasingly small percentage of the total datacap), the non-datacap data's value shrinks incredibly low.

I guess what I'm saying here is that I would love to see some economic modelling of the consequences of this, under conditions where the allocation of datacap becomes harder, but the length of time datacap is held by an SP grows. I ask this for two reasons: what does that means to the decentralizing of the network, and how strict does it point to us needing to be to guard datacap provision?

[anorth]

Thank you for your comments.

I don't think this proposal should change too much about the role and goals of FIL+ notaries. I concur that both aspects of the deal and aspects of the data itself are relevant considerations, and with different weights depending on the circumstances like whether the data is a public good, or the client genuinely unaffiliated with the provider.

My phrasing in the proposal, when I wrote it in March, does emphasise the potential for permanence and indefinite terms. This largely comes from the realisation that prompted this, which is that FIL+ allocations lacked any explicit term and if we were to open things up for third-party market actors, the default would be unbounded unless we made an explicit change. But the actual change which is proposed here decouples FIL+ from markets but adds back an explicit term (as a range from min to max) for the data cap allocation itself. The practical immediate consequence will be the data cap can be allocated for up to 5 years (stage 1), and renewed further by spending more datacap (stage 2) and transferring the data to new sectors. So it won't be permanent.

In the long term, I believe that the original concept for datacap is that it should become increasingly common, rather than a higher hurdle. You are correct that, over time, increasing abundance of datacap will reduce the value and network power accorded to each allocation. This should reduce the governance burden and strictness of allocating each marginal GiB. Eventually the network subsidy to storage will decrease in value, and clients will need to pay some non-zero fee for data storage. In the long run, we need demand to meet supply at a price.

Ultimately this proposal changes the network parameter of the max duration of datacap from 1.5y to 5y (or whatever value we decide), with technical possibility to make it longer in the future. The modelling you seek would of course be valuable, and perhaps the CryptoEconLab team (who designed FIL+) could assist. @zixuanzh?

[anorth]

I have begun drafting a formal FIP for this proposal in #432. I'm starting with the specification, as much of the motivation etc will be copied from this discussion. @ZenGround0 and I are sketching the implementation now, which should flush out any considerations I have missed.

[anorth]

should we define the TermMaximum according to the proof type

For any individual allocation, TermMaximum is set by the client. There's a policy maximum value for this (that I haven't given a name), but no I don't see any reason to couple it to proof type.

a client allocated the datacap to the same piece data to the same provider

Do you mean a client makes two allocations with the same data and same provider? It's fine, the provider can store it twice. The Claim will record which sector is storing which allocation (they have IDs).

[jennijuju]

policy maximum value for this

ah yes - I was wondering about this lolol

t no I don't see any reason to couple it to proof type.

I guess I should be asking: what's the policy maximum value? is it bounded by the sector Max life time?

[anorth]

Its in the FIP: 5 years

[dkkapur]

Do you mean a client makes two allocations with the same data and same provider? It's fine, the provider can store it twice. The Claim will record which sector is storing which allocation (they have IDs).

From a Fil+ standpoint, there is no upside / usefulness to this though. Would be interesting to actually only cap that data, provider tuple to have 1 instance per client.

[jennijuju]

@dkkapur other than "interesting", is there any requirements from FIL+ to must have unique (cliend ID, data that has cap, provider ID)?

[jennijuju]

This representation permits a claim to span pieces of data in multiple sectors.
// Ranges are over the allocation data, and are non-overlapping. (0, 0) means all.
// This structure initially un-used (just a single entry) but supports claims larger
// than a single sector.

@ZenGround0 could you please help me understand how this may work? so far it sounds like

• if a piece of data, with data CID_A, gto break into mutiple pieces, with CID_B, CID_C, CID_D.
• say a provide publish a deal and claim allocation with CID_B (with defined range)
• we will go check the verifreg, and confirm CID_B is a part of the CID_A?

on the storage provider implementation side, when they publish a deal to make the claim, how can they get the right range for CID_B? or is that something that needs to be provided by the client?

[jennijuju]

why not have client define range[], range[]... in allocation? and sp may claim accordingly?

[anorth]

A client can break their data up into multiple sector-sized allocations if they want. The proposed mechanism is to give the provider flexibility about how to arrange data in sectors (32GiB? 64GiB? Snapped into some free half-sector? Future 1TiB sectors?).

[jennijuju]

This maintains the incentive to host a complete replica of the allocation. ....Other ranges may be committed after the expiration epoch.

Whats the mechanism to ensure that all rangers are committed?

[ZenGround0]

Forgetting about ranges this is the case for a single cid allocation that doesn't get committed. The builtin market actor that facilitates datacap exchange has a penalty for providers that don't commit deals they have promised in exchange for datacap. All markets including those that use FIL+ can do the same to protect clients.

[dkkapur]

@ZenGround0 how is the penalty calculated / what is it? Also what happens to the DataCap that was used by the client on that particular cid, minerID combination? Do we automatically "refund" it so it can be re-used, or is there a need to go apply for more DataCap?

[anorth]

The penalty for failed deal consummation is specified by the client in the deal proposal. But note that this is just the current built-in storage market's policy. Future storage markets can have different policies.

No, data cap cannot be refunded for partially-satisifed allocations with current mechanisms.

[jennijuju]

do you mean the ProviderCollateral in the deal proposals?

[jennijuju]

data cap cannot be refunded for partially-satisifed allocations with current mechanisms.

this is a change of behaviour of the current FIL+ process, and might require client to apply additional datacap to get the full data stored.

[jnthnvctr]

To echo @dannyob I'm pretty concerned with the direction this conversation is going re: explicitly verifying data on the network. I'd like to piece apart the functionality we'd like to see and the practical implementation, but I'd call out that the moment we start talking about the network verifying explicit bytes (and proposals for how to automate renewal for those bytes) we are opening up a philosophical and logistical can of worms.

I feel pretty strongly that the network should not be in the position that it is making any sort of claim about the legitimacy of data (nor that it could be interpreted that the network is making those claims). The flow we have today is the network endowing individuals with datacap, who then make their own decisions about data. While subtly different, it feels substantially less problematic than saying explicitly that a specific dataset should be subsidized vs. another that might not be. You can imagine for a sensitive dataset the act of including a subsidy (or not including a subsidy) to be seen as a political one.

Practically speaking this proposal could be interpreted as the network making some sort of endorsement of the data itself, which given the differences across global jurisdictions and laws could be seen as problematic.

IMHO we should instead be focused on how we can automate the notary process (via incentives, where it is irrational to engage with self dealing) and migrate to more permissionless systems post FVM. I can elaborate on some ideas here (believe Nicola has some as well) - but dont want to derail from this topic.

[ZenGround0]

The flow we have today is the network endowing individuals with datacap, who then make their own decisions about data. While subtly different, it feels substantially less problematic than saying explicitly that a specific dataset should be subsidized vs. another that might not be.

I don't understand how this proposal departs from that model. The verifier -> verified client -> verified data pipeline is the same. It might help to look at the latest iteration of this proposal as this discussion captures a longer design process. Verified clients allocate datacap to a (provider, piece CID) pair. While this is now decoupled from builtin storage market deal terms this does not mean the network is providing some qualitatively different blessing to data. This fip only tweaks the capabilities of the verified client in a way that allows generalized markets to make use of FIL+.

[jnthnvctr]

ack - this is my bad! i'd missed the latest revision, my bad

we should talk about the idea about automatic notaries more though, think this will unlock quite a lot :D

[anorth]

This and other changes (#413) have been designed with automated notaries in mind.

[dkkapur]

@anorth, I was just chatting with @jennijuju and a question came up - what is the plan for handling changes introduced by FIP-0028 here? I assume the mechanism has to change somehow, but IIRC we need to account for that in this FIP before this can move forward. Here's the PR for that FIP: https://github.com/filecoin-project/FIPs/pull/226/files

[jennijuju]

@dkkapur - I have confirmed the with @ZenGround0 , as of this upgrade, allocation is can only be made by market actor. So Anorth is right, there is no new step.

[anorth]

is MakeAllocation only callable by market actor?

No, others can call it. Especially other user-programmed market actors. But in the immediate aftermath of this FIP, there will be no point because we need subsequent changes to the miner actor APIs to enable claiming the allocation without using the built-in market.

[dkkapur]

ACK but this new step is harder to build safety around, right? i.e., if someone runs rampant with DataCap today, they're still somewhat blocked by their sealing rate + PSD costs + other miner side constraints, so essentially its a race for us to round up 2 notaries and get RKH approval vs. how much DataCap they are able to lock in day by day. It seems like with the allocation process - it could be pretty close to instantaneous and then redeemable at any point for a verified deal by the SP once they have access to collateral/resources with no recourse?

[Kubuxu]

@dkkapur this doesn't change, today the Miner actor goes through the Market actor to get the deal weight/Quality Adjustment for a given piece of data/sector based on whether it is FIL+ or not.
The proposal here is for the Miner to essentially bypass the Market for purpose of verifying that FIL+ Allocation for that piece of data can be claimed.

The SP still has to expend sealing+PSD so on to onboard a sector.

[anorth]

if someone runs rampant with DataCap today, they're still somewhat blocked by their sealing rate + PSD costs + other miner side constraints

I don't think so. A party can create deals with the market as fast as they like, independent of later activating those deals by sealing sectors.

[anorth]

This is a new thread for discussion of the capability for larger-than-sector FIL+ allocations, and the ranged claim approach to satisfy them. A number of the questions above are about that, so let's discuss it here.

This proposal presents and opportunity to implement the data structures that will permit larger-than-sector allocations. The inability to do deals larger than a sector is apparently a pain at the moment. This proposal addresses the FIL+ part of that, but won't actually make such deals possible yet without further changes to the miner actor for #298.

There are significant constraints on what it's possible to implement here. This could provide a huge scalability boost to FIL+, but can't come with much in the way of enforced policy mechanisms. The general design pattern here is to make FIL+ simple, and delegate all deal-related policy to market actors, which can act as delegates. If a client wants particular assurances, they should use a market that provides them. An example would be some collateral posted that would be lost if not all ranges are sealed.

Note that the built-in market cannot support larger-than-sector deals and I have no intentions of doing the work to change that. Instead, I expect other, far superior market actors to emerge on the FVM.

If there is significant opposition to making this possible, I'll just remove it from the proposal and then larger-than-sector allocations will have to wait some time (years?) to rise in priority enough to motivate a dedicated FIP.

@jennijuju @dkkapur

[dkkapur]

Thanks. @jennijuju just to confirm, your main concern here is still that there may not be an incentive for an SP to follow through on the full set of sectors thereby potentially both (1) rendering some DataCap useless which has already been spent by the client and (2) rendering used DataCap potentially useless as well, if partial storage is of no value.

My main concern is around enabling clients to get DataCap back in the scenario where they could not use it as intended. IMO this is non blocking here, but could result some follow up proposals?

[jennijuju]

@mr-spaghetti-code is the an urgent need from the clients to be able to make larger-than-sector size deals?

unless there is an urgent user requirement, I'd suggest to descope this from this FIP:

• given the additional deal packing complexity introduced by this FIP, the incentive for SP to actually commit a sector with a small piece of data, that might be way less than a sector size, is not clear.
• will simplify the actor implementation
• engineering time wise, its hard (almost impossible) to support this feature from the client impl side, given the desired upgrade timeline

[jennijuju]

related #313 (comment)

[anorth]

After more consideration, I'll remove all the details about this from the FIP. I may leave a placeholder for the data so that we don't need to do a migration while implementing the other pieces of this. I think we'll be able to do it all in the subsequent API-changing FIP.

[jennijuju]

Thank you!

[dkkapur]

@anorth from Fil+ side of things - we're going to need time to update all the tooling on our end if this FIP goes through. Specifically:

• flows for RKH and notary allocations are not really affected
• flows for dashboards / tracking / notary due diligence -> significantly impacted, need to redesign how we track DataCap flow and what we can do with it?
• flows for clients -> content changes to how DataCap is to be used, substantial changes to bots that help monitor DataCap balances and prompt requesting additional DataCap when they run low

I'm sure there is more than this. If this FIP passes, would love to find a path where we can play around with the implementation in test/calib with enough lead time to ensure a smooth transition with the network upgrade cc @raghavrmadya @galen-mcandrew

[anorth]

Since this FIP continues to use the built-in market actor as a delegate, leaving all actor exported APIs intact, I expect very minimal changes to tooling that interacts with the chain. However, the state representation will change, so yes tooling that inspects datacap balances on chain will have to look up a different table of balances. And I guess there might be some tooling about the implied active datacap in deals in the built-in market, which could instead look at the explicit representation of data cap in claims (possibly both, depending on a migration into the latter).

Implementation is in progress now so I expect testnets will be available with at least the usual 6-8 weeks lead time (but this is more @jennijuju and @arajasek's area than mine)

[jennijuju]

base on my experience with the tooling lolol (assuming its still similar than the one we used to look at tgt

flows for RKH and notary allocations are not really affected

I will follow back on this one when I got a confirmation on this

flows for dashboards / tracking / notary due diligence... significantly impacted, need to redesign how we track DataCap flow and what we can do with it?

&

inspects datacap balances on chain will have to look up a different table of balances.

how are you getting the stats today? if you are simply calling lotus api StateVerifierStatus & StateVerifiedClientStatus for getting datacap allocation stat, then we will actually have to make the changes within API tho the end user experience wont change.

and what about datacap flow? do you track AddVerifier & AddVerifiedClient messages & market for deal info to get datacap from client -> SP? if so.. I think they likely wont change much.

flows for clients

yah you'd at least need to add RevokeAllocations

[jennijuju]

Implementation is in progress now so I expect testnets will be available with at least the usual 6-8 weeks lead time

to be more precise, you will get aleast** 4 weeks if your testing point starting on calib upgrade. tho if you test in other
devnets you will get even more time

[dkkapur]

thanks @anorth @jennijuju! We primarily look at publish deal messages to track when verified deals are actually on chain and update our understanding of an entity's DataCap balances based on this, rather than just checking verified client status. This also lets us make assumptions more easily on rate of utilization of DataCap and attempt to draw links between entities when looking at potential abuse by assuming FIFO flow of DataCap. This bit will need to change to track allocations.

[anorth]

publish deal messages won't change. If that's what you're using, you have the whole period until the next big update here (nv19?) to change tooling.

[dkkapur]

Are there any examples of scenarios where differently architected market actors could be useful?

[dkkapur]

Having this outlined would also help motivate support for the FIP. We've got a Fil+ governance call coming up next Tues at which I'd like to flag this FIP and keep people updated.

[anorth]

Basically every feature that people want out of filecoin deals. See #241

Features like deal extension, multi-sector deals, re-negotiation, deal transfer, capacity deals, auction markets, repair markets, insurance, derivatives etc should all be possible in smart contracts.

[anorth]

I did attend a FIL+ gov call back in ~March and outlined this proposal at the time. It was a while ago, so I would not be surprised if it has since slipped from consciousness.

[dkkapur]

@anorth yes exactly 😅 will re-flag at upcoming calls on Aug 23/24 and re motivate some attention here.

[xinaxu]

Great proposal. I have a few questions regarding technical details

1. AllowRanges - I know it is not going to be used today, but how is it possible in the future? If a client want to make a 1PiB allocation, does he needs to provide 32000 CIDs in the allocation object?
2. Is it feasible to allow the provide to extend the sector without dropping any claim, so that SPs do not need to go through the sealing procedure to move it to another sector? It's greener and will allow SP to utilize their sealing capacity to onboard more data.
3. Is allocation going to be a new message type sent from the client? Does that mean now the client needs to pay some gas to perform the allocation?

[anorth]

1. No, client will provide 1 CID and each sub-piece (from the SP) will need to carry an inclusion proof. But I don't want to specify much more now
2. If the allocations permit a longer max term, yes the SP can extend. But they must drop claims with a max term that would be exceeded. This is a step towards full re-use of storage without resealing
3. No, the market actor will do it for them.

[xinaxu]

@anorth Could you elaborate a bit more on the 3rd one?
The current flow is

1. Notary grants datacap to a verified client via an on-chain message
2. Client sends a deal proposal to the provider off-chain
3. The provider publishs the proposed deal on the chain

With this FIP, how does the flow change? since you mention that the market actor will perform the allocation, does that mean

1. Notary grants datacap to a verified client via an on-chain message
2. Client sends an allocation proposal to the provider off-chain
3. The provider publishes the proposed allocation on the chain on behalf of the client

[anorth]

No the flow is literally the same as the first version, but during (3), for a deal with verified=true, the provider will also create an allocation with the verified registry on behalf of the client.

[dkkapur]

@anorth and all others involved in designing this FIP proposal, thanks for all the discussion over the last few weeks. Helped me clarify my understanding of this one substantially and I'm generally supportive of this. However, as part of this design change - based on what we have observed in the Fil+ community so far and where gaps lie on the governance front today, I'd like to propose the following amendment.

Create a mechanism to change an allocation for a piece through a Fil+ governance mechanism.

Primary motivations:

• Enable a client the opportunity to actually get useful value out of DataCap. Today, if a client or a client representative applies to store data of size X, they end up making a request for 5X+ DataCap and ideally spend that with 5+ SPs that they can trust to correctly store replicas of their data over time. If over time their knowledge of the SP and their services changes, an option to replace/change an allocation gives an out to either ensure allocated DataCap can be redirected or removed. This helps with some amount of unnecessary dilation of the value of DataCap, helping reward SPs and clients working together to make Filecoin more productive / useful (stated mission of Fil+). This case is unfortunately quite common today - given changes in the Fil+ ecosystem, SP economics, complexity in accurate setup, etc. Up front due diligence in a young network today without strong reputational systems to rely on, is challenging for clients.
• Given this FIP introduces an architecture change in how DataCap is allocated and used, this amendment will provide an additional option for consequences that the Fil+ community can levy in the case of abuse, in this new system. Though the risk introduced by this FIP is theoretically none (since the implementation actually creates no new risk), practically, it is likely to result in abuse being perceived as easier. Specifically, though clients and SPs that collude today could theoretically lock up DataCap to be expended in future deals, the gap between DataCap in client addresses vs. used in published storage deals is not substantive (<5%, usually 1-2% or so). This change, with a future of using DataCap without proposing/publishing deals via the built-in storage market actor, will highlight the ease with which DataCap can effectively be "locked up" for future sealing by entities looking to misuse DataCap.

Proposed design:
For the initial implementation, we could rely on similar functionality as what was built to support removing DataCap from client addresses today. I.e., explicitly needing 2 individual notaries to sign messages to the chain requesting a change in status + approval from the RKH multisig (2 signers on the multisig).

We can design and implement the governance process and tooling to make this process easier on the Fil+ in the few cases where it will be needed.

Additional considerations:

• proposing this a separate FIP is challenging since the changes introduced by this FIP are going to be dependencies for implementing this amendment. We'll want to develop on the same branch.
• Since deals will initially go through the built-in market actor, the risk is somewhat mitigated. This amendment is still being proposed for this FIP since we'll likely have several more changes introduced in the next big network upgrade (with FVM), and it makes sense to address potential new sources of risk with the shipping of the initial change.

[dkkapur]

cc @jennijuju @ribasushi @xinaxu @Fatman13 and others from the Fil+ community, would love to hear your thoughts as well.

[anorth]

Could you please be more concrete about what "change an allocation" means? Change what about it? Only unclaimed allocations?

Note to your second motivation, we can and probably should add a maximum expiration period for an allocation, so it can't sit around on unused datacap forever. Yes an SP a client could re-allocate it, but we can make it more difficult to squat on.

In discussion I know I suggested a change like this could tag on this FIP, but I no longer like that idea. I think you should probably propose a new one, with FIP-0045 as pre-requisite so you can phrase it in terms of these concepts. A governance mechanism like you propose is pretty easy to add later, and would immediately be effective.

[jennijuju]

Yes an SP could re-allocate it

do you mean client?

[dkkapur]

@anorth thanks - adding the following:

• change unclaimed allocations - ideally both "refund" back to the client or remove entirely
• agree. will flag to Fil+ community and get a sense for what that could be. perhaps 6mo or some dynamic variable based on the min/max durations put forth by the client

what's the logic change on not adding this to the current FIP? issues with implementation or something else?

[anorth]

what's the logic change on not adding this to the current FIP? issues with implementation or something else?

• it's adding complexity to an already complex proposal
• it's a net new idea (you could equally propose a governance action to flip the verified flag on a pending deal in the market today)
• it builds on this proposal but isn't required for it, and could be added later and still work: it should be accepted or rejected independently

[jennijuju]

some notes from the CN SPWG call today

on manual opt-in migration existing DC -> new QAP accounting:

• without FIP0036 & understood that more collateral will be needed according to the longer period QAP, vast majority of the group think its likely they'll consider the manual migration, and commit longer storage service for those data
• with FIP0036 - if the migration term max is fixed and > 3yr, some of them raised that they will very unlikely to do the migration given the potential impact that additional 30x-50x collateral will provide

SPs raised that they'd prefer flexibility, meaning they might prefer the possibility to define the term max themselves upon the opt-in migration. They will circle back after more analysis.

[jennijuju]

An opt-in per-existing-dc-sector migration by SP is being proposed in the exiting FIP draft, tl'dr: SP can extend any active datacap-ed deal up to current+540days, despite the current deal duration thats set by the client, by triggering a sector migration.

We should try to reach consensus soon, as a community, on whether this is desirable:

• @dkkapur: you once raised concern where, there might be some abuse in FIL+ program, and giving more power to the spent datacap might amplify the abusement of datacap and it is undesired from the FIL+ governance perspective. Is that still the case? does the fil+ community has a strong preference on whether existing dc migration should be enabled or not?
• @AxCortesCubero @misilva73 what's CEL's take on this proposal? any comment on the potential QAP growth impact?
• @dkkapur @mr-spaghetti-code : any concern from the verified client SLA perspective on SP may take the lead on how long they are incentivized to store the data, instead of the clients?
• @anorth - @geoff-vball (plz correct me if im wrong) was raising: is it fair that when clients wants to extend the term, more minted tokens needs to be spent. However - no more token needs to be minted for existing sectors?

[anorth]

@jennijuju the linchpin for whether to support opt-in migration is FIP-0036. Without that migration, FIP-0036 is "unfair" to existing sectors with deals (if existing CC sectors can be immediately extended for a multiplier), which encompasses the most aligned providers, even if it also probably includes some fraud.

If FIP-0036 were not in the picture, I would be more or less happy to drop the migration and focus only on new sectors and deals. It would remove a lot of complexity.

@geoff-vball the proposal right now explicitly handles this, and allows the original client to increase term up to the 5y max as if they had done so in the first place, without spending more datacap. However, one practical result of this is that past fraudulent datacap can then be extended for 5y too. @dkkapur had suggested not supporting unilateral client extension, and instead requiring new datacap so that new, better governance processes could reduce the impact of abuse. I could accept this most easily if the built-in market is configured to set a long term maximum for new deals.

[steven004]

It is the right direction to decouple FIL+ term from storage marketplaces. Very supportive of this. But we may not need to solve all the problems with this, to really solve the deal extension and other related issues, I would like to see decoupling deals from sectors, #442 looks interesting.

[anorth]

FIP-0045 separates FIL+ maximum term from a deal’s term, so that, after first making a deal (limited to 1.5y by the built-in market), a client could increase their FIL+ term to 5 years if desired. Later, a client will be able to specify a 5y term up front, but that requires more technical and API changes. The FIP also changes the QA power calculation to be a simple 10x on the verified bytes for the life of the sector, but requires a sector to expire prior to the client-specified FIL+ term limit being reached.

In the simple approach, this capability would only apply to new sectors and deals. This raises a question of what to do about existing ones. Hand-wavy motivations to do something more are are:

• Many clients of those deals probably wanted to specify a longer term but couldn’t - can we let them?
• It would be nice to incentivise extending existing FIL+ sectors with a continued QA power boost

It’s possible to migrate old sectors into the new model, but that migration has limitations:

• Any deal that’s already expired can’t be migrated, but its full QA rewards have not yet been earned due to the spreading out effect of the old QA power calculation. For some sectors, migration would cause a power drop. So we can’t do it network-wide, it must be opt-in from SPs.
• Any deal that hasn’t already expired would change to a full 10x multiplier for the remaining life of the sector, which may mean FIL+ incentives last beyond the deal’s initial term. This behaviour is a necessary invariant in the new system.
• An SP can unilaterally extend a sector’s expiration to 540 days prior to migration, which means that any non-expired deals then will then earn a full 10x rewards for a further 540 days, regardless of the deal’s term (which was certainly shorter, though many clients might have wanted a longer term, if they could have specified it). Since this is possible, for simplicity we’d just add that 540 days automatically at migration.
• After migration, an SP must get their client to extend their allocation’s max term (an on-chain operation) before they can extend their sector (beyond the 540 days they can get for free, in the above point). SPs who can’t reach their clients won’t be able to take full advantage.
• Implementing the migration capability is complex, and adds a lot to the necessary scope of testing nv17.

This all became more intense with the introduction of FIP-0036, because then the inability to extend existing FIL+ sectors (without diluting QA power) makes the sector duration multiplier far more accessible to CC providers than deal providers. Deal providers’ share of revenues would likely fall significantly against CC providers and this is perceived very poorly by community. So FIP-0036 wants a migration to fix equality.

On the other side, it’s likely that some proportion of datacap granted in the past was fraudulent (fake “clients”). As FIL+ governance improves, we’d like to decrease the impact of past grants and direct participants to new ones. So FIL+ governance wants no migration, so as not to amplify the impact of fraudulent QA power.

So it’s a big compromise. If we don’t implement migration:

• Existing providers of verified deals are disadvantaged w.r.t. access to FIP-0036 duration multipliers (if FIP-0036 is accepted)
• Existing deals will expire within 1.5y and those sectors more likely to be dropped, even if clients might have desired a longer term

If we do implement migration:

• Existing deals will earn 10x rewards for at least 540 more days, overriding client’s deal term
• The impact of fraudulent data cap will be extended at least 540 more days, but more likely ~5 years since fraudulent clients would presumably always increase their term to 5 years [1]
• It’s complex and adds risk to nv17 correctness and timelines.

I have been working to try to make FIP-0036 possible, via making migration possible. But it’s not a great story. After some consideration, I would prefer not implementing the migration, and just look to the future. I would prefer FIP-0036 addressed inequality by making multipliers only available to net new sectors, or some other means. But if it’s the only way, it’s probably better than the inequality.

A key tradeoff is how much to preserve existing power, knowing that some of the QA boost is inauthentic? Do we want to preserve the top-line while sending disproportional rewards to bad actors, or cleanse the system by letting that power cycle out over the next 1.5 years?

[1] A possible mitigation to the potential for 5 years of fraudulent data cap is to disallow clients extending their FIL+ term after the fact, requiring instead to spend new data cap, which might be better governed. But this conflicts with the equality of access needs for FIP-0036, and would put FIL+ notaries in the role of adjudicators of who can get duration multipliers or not.

[anorth]

I have removed the opt-in migration from the FIP in #455. Old sectors with verified deals will stay in their own model. New sectors, or snap-deals into CC sectors, can use the new model.

[misilva73]

The current version of FIP 36 has a section that discussed this inequality.

The TLDR is that the FIP has softened this inequality by reducing the maximum duration to 3.5 years and by gradually phasing in the increase to maximum commitment. In addition, in order for CC sectors to take advantage of a higher pie of rewards, they need to bring capital into the network, which is another smoothing factor.

[jennijuju]

The current proposal (which only enables storage market actor as a delegator of datacap allocator) sets the default term min = deal duration, and term max is deal duration + .

We shall seek consensus on the follow product decision:

• what's a reasonable value for
  • @AxCortesCubero @misilva73 any suggestion from CE perspective ?
• Should the protocol define the duration on how long the SP should be incentivized for storing the data, despite the duration set by client during deal proposal?
  • tagging some folks I know thats been working on client onboarding and growth, @LaurenSpiegel @mr-spaghetti-code any insights?

[LaurenSpiegel]

Should the protocol define the duration on how long the SP should be incentivized for storing the data, despite the duration set by client during deal proposal?

Why would the SP get incentives beyond the duration set by client?

Generally, the objectives we should try to achieve (or at least not move away from)--

1. the client should control the duration of data storage
2. extension should be easy (should not have to move data around).

[anorth]

For practical implementation reasons, the incentive duration must align exactly with a sector commitment duration. In the general case, a client will be able to specify a min and max term in the future. However, in the interim, the built-in market has no means to receive the min and max term from the client (it get's only a single duration number). If it specifies only an exact term (min == max), then those deals will be basically impossible pack into sectors.

So I understand the objectives, I'm asserting it's not practical to hit them exactly in this interim phase, and thus opening discussion for a "good enough" policy that the built-in market can implement.

[anorth]

In core devs call, a value of 3-6 months seemed reasonable. I've specced 90 days in #455

[misilva73]

@kasteph has created a dashboard with the distribution of deal durations here. It seems that there are 3 modes in the distribution, namely, around 6 months, 1 year, and 1.5 years (although the last two seem to have more deals than the 6 months).

I think that for the last 2 groups, 3 months seems ok. But for the 6-months deals, I am not sure if 3 months is too much as it corresponds to an increase of 50%. However, the max term duration is only a mechanism to make deal packing easier, so I am guessing not all sectors will go to the max.

[jennijuju]

FYSA - @dkkapur & fil+ team, the current proposal states: A verified client can extend the term for a claim beyond the initial maximum term by spending new DataCap. meaning that the client will need to request more datacap for storing the data for longer term. This is similar to clients needs to request new datacap for renewal the deal by making new deals, or want to make extra replicas. just pointing out that you may see Datacap extension as a reason for datacap applications with this FIP

[MegTei]

HI @anorth @dkkapur @mr-spaghetti-code I have to admit Im struggling to see, and (possibly this is why there is low community engagement):

1. What are the problems to solve and for who clients/ SPs/ Governance?
2. What are the viable and feasible options and their pros and cons on the table as a result of this discussion?
3. What does the end to end flow of datacap look like over the life of the deal?

This complex FIP would benefit from a service design mapping or mapping out the flow, if there is one can you share it?

[jennijuju]

What are the problems to solve

Did you at least read the FIP? Specifically the motivation session?

[MegTei]

Yes, it doesnt address these Qs

[ZenGround0]

1. From the FIP:

The core motivation behind this proposal is to enable the development of markets and market-like actors
on the FVM, once user-programability is supported.

3. Datacap flow is not changing significantly. Conceptually datacap still flows from verifier to verified client and then transformed to miner actor power multiplier. The implementation is changing a bit to use a standard token contract to represent datacap which you can read more about in the fip.

[jennijuju]

Spec hook/method for a different client extending the claim by spending DataCap

per @dkkapur - this feature is not required/desired from Fil+ atm. However, if this is supported on the protocol level, there shall not exist major issues given currently, a verified client can simply spend datacap on any CID they'd like to without having actual data via offline deals.

[anorth]

👍 I'll leave it in scope. It will be hard to implement later because it would break an API that user actors will use.

[jennijuju]

The market will accept verified deals with a StartEpoch beyond the maximum allowed allocation Expiration.
If such a deal is not activated before the allocation expires, it will not confer QA power.

similar to deal packing, storage provider's decisision on when to force precommit(bacth)/provecommit(aggregate) now has a new contraint, allocation expiration.

using an extreme example, today, if

• deal_A with a size of 4gib landed(as in PSD) now with start epoch set at now + 100 days
• deal_B with a size of 4gib landed closely with start epoch set at now + 200 days
• SPs can totally wait up until say now+90 days so that more deals that fit in the sector can be packed before start sealing.
• but with this change, they have a new constraint of allocation expiration, in which puts a 60 days contraints

i personally dont think this is a huge deal given:

• 75% of deal sectors today only have one deal -> not waiting for packing more deals in one sector
• clients set way closer start epoch (<60 days) AFAIK, & most clients especially data brokers prefer deals to be packed and sealed as soon as possible -> less waiting for packing additional deals in the same sector

tho it is still a new constraint, thus wanna leave a call out.

[anorth]

Moving a question from @misilva73 from PR review

It is not clear to me what happens to Allocations and DataCap tokens when there is a termination.

For instance, imagine there is a 2-year sector with a FIL+ deal that has a minimum term of 2 years. Then, after 1 year, the sector terminates. what happens here? Can the remaining 1-year DataCap allocation be used? Should DataCap tokens be given back to the client? Should a new allocation for 1 year be created?

[anorth]

Nothing special happens here, and this preserves the current behaviour for now. The remaining allocation can't be used (yet) and datacap is not refunded to clients. However, this is a step towards better behaviour which I expect to realise in a follow-up change (but requires change to miner actor exported APIs)

An earlier draft specified that the provider could re-seal the claimed data and resume rewards for it, i.e. maintain the incentive. Everything from the datacap/verified registry side can support this. However to do it in practise requires changes to the miner actor APIs, and this FIP is leaving them unchanged for now. In a subsequent change we'll be able to resolve this.

[stuberman]

PiKNiK supports FIP-0045

[anorth]

Moving some questions from @dkkapur to the right forum:

1. Does the client address in the Allocation and Claim structs have any downstream impact in terms of deal/QAP extension for SPs? The way the FIP reads today (which is reasonable, just want to confirm) - anyone can create Allocations for the specific CID/SP and the SP could make a claim against it. Does the first client address need to be maintained in either of the Structs for future renewals? The only potential case I could find for this was in Extension of claim term by client where it looks like the original client listed in the claim struct is the only one able to increase a claim length up to the 5y max.

The case you have identified is the only on-chain use of the client address in the proposed implementation. Of course it's great metadata to keep transparent. It might also be useful for other things later on.

2. In the case that a client does extend a claim up to 5y but the sector expires / if the SP does not extend it? Is it returned to the allocations table or burned?

A claim remains valid until its max term is reached, regardless of what happens to sectors. However, in the concrete version being implemented, if the sector that first claimed the allocation expires, there is no mechanism for the SP to seal that data into a new sector and "re-claim" the power. The claim remains in state, but isn't doing anything. I hope to add such a mechanism in the follow-up #298.

3. There is no deduplication across clients on the allocations table since the client address is a field in the allocation, correct? Theoretically, as long as there are multiple identical allocations, an SP can make multiple deals with the same CID (serially, waiting to seal each one before initiating the next) and get QAP for each of them. This is the same behavior as today but could be a good opportunity for future optimization.

I'm not sure what you mean by "optimize", but your description of behaviour is mostly correct. A provider could claim all the allocations simultaneously (but with a unique copy of data for each).

4. Can multiple allocations can be made from the same client address to the same SP for the same piece CID?

Yes

6. Will MinimumVerifiedAllocationTerm need to be updated to 12 months if FIP 0036 passes?

Terms less than 12 months will be impossible to seal, so it would probably make sense to do that (but isn't critical).

7. Sanity check: is there any particular reason why MarketDefaultMaximumAllocationTerm needs to be 90d? Seems like anything between 60d and TermMin should work correct?

We discussed this parameter live in the core devs call. Yes a range of numbers could work, trading off the buffer that makes deal packing easier with a tighter adherence to the deal's term. I would like longer, you pressed for shorter, we landed at 90d.

8. With this FIP, (please correct me if this is wrong) a deal cannot be sealed into a sector that has a lifetime that is 90d longer than the deal length. SPs that proactively extend existing sectors beyond reasonable deal length terms (i.e., anything over 21 months today) may preclude them from being able to snap future deals into those sectors.

Yes, although I tend to take the perspective that clients making short deals will find a lower supply of sectors willing to take them.

If FIP-0036 doesn't pass, I will immediately propose a FIP to increase max sector duration to 5y, which would make this issue go away.

cc @ribasushi @ZenGround0

[xinaxu]

If FIP-0036 doesn't pass, I will immediately propose a FIP to increase max sector duration to 5y, which would make this issue go away.
@anorth Is this still on your radar?

[anorth]

#475

[rvagg]

FYI there's a retroactive update proposal for this FIP @ #961 to align type details to what has actually been deployed to the network.

[Fatman13]

FIP-0045 was referenced to be solving this legacy issue here, but it seems that 10x pledge is still locked when sector become CC from DC. Is this by design or overlooked? @anorth CC @rvagg

[anorth]

I'm afraid FIP-0045 was never going to be able to resolve this.

From #367 (comment), my last word on this was

you are correct that the network currently enforces a policy that a sector's collateral is monotonic: it can only increase. Even when we make sector storage much more flexible (deal extension, transferring deals, etc) the current plan is to maintain this collateral requirement at its high water mark.

I can certainly see why providers might wish for the collateral to be recalculated at various points (it almost always goes down, even if deals were extended). I've brought this up before, but forget the reasoning. We need the cryptoecon team to help explain why this policy is preferred.

I can see later comments from @jennijuju that FIP-0045 would resolve it, but I'm afraid that, as far as I know, they are not correct.

I do think we should reduce pledge requirements after FIL+ claims expire. My proposal for doing this is outlined in #734

Remove the monotonic requirement on a sector’s initial pledge, allowing it to decrease. When a sector is extended or updated, recalculate the initial pledge amount and refund to the SP if the new requirement is lower than the old requirement.