CHIP-0012: Reduce Plot Filter

[jmhands]

plot filter chip, open up at 8am 1/20

[SlowestTimelord]

Thanks for writing such a thorough explanation of the problem and proposed solution @jmhands! A few clarifying questions that I have already asked you in other channels, but adding here for completeness:

1. Does reducing the plot filter by half also double plot "decompression" farming workload? Or is it not exactly proportional?
2. Upon reaching size 32 plot filter, does that imply increasing minimum k-size to k33 is the logical next proactive step? At that time would plot filter reset to 512 or stay at 32?
3. Is increasing the minimum k-size a soft fork?
4. In the event of a hard fork, would that be an opportunity to introduce other pending changes that require a hard fork? What such changes might those be?
5. Would increasing the number of signage points per subslot (to decrease the time between infusion points) be a viable option as well or is that too drastic a change to the consensus?

[SlowestTimelord]

A few more thoughts to stir the pot in favor of Option 2 (k-size change):

• If there were ever a time to declare timelines for a minimum k-size, it's probably now while folks are considering replotting anyway.
• I don't think a proposal on plot filter decrease is complete without considering when/how minimum k-size requirement might fit into the ~10 year timeline.
• Plot filter changes + decompression considerations adds a dimension that many farmers may have trouble grasping or considering whereas a minimum k-size timeline is easier to grasp (say, k33 phases in 2025, k34 phases in 2028 etc.)
• Given the unpredictability of future hardware and plotting/farming advances, there's a chance another hard fork to change the timeline of plot filter changes may be necessary whereas as Bram has mentioned in the past a minimum a k-size change would be a soft fork (I'd like to confirm whether or not this is the case).

Increased hardware requirements for plotting. A k=34 plot would be very difficult to create in RAM without using a high-end server

Maybe it's my experience from early plotting days but I've always viewed the ability to plot all in RAM (and now to plot with GPU+RAM) as a nice-to-have and not something farmers should feel is necessary or feel entitled to be able to do.

[PanzerKunst]

Between:
A. Halve the plot filter every couple years
B. Increase minimum k value every couple years

As far as I understand, B will require many farmers to replot to k+1 every couple years, whereas A doesn't require such a periodic replot. If that is indeed the case, I think this CHIP is a great proposal, and I support it.

[digitalspaceport]

What is the plan after 2030?

[CryptoBlockchainTechnologies]

This is a good proposal to ensure there is a constant cadence in hardware requirements, however I think the two year implementation schedule is a bit aggressive. Why was two years chosen, was it arbitrary, or based on ASIC game theory?

I think it makes more sense to tie in the halving of the plot filter with the halving of coin emissions every three years to prevent them from occurring too close together. More specifically six months prior to the coin halving we would get a plot filter halving. This would allow farmers to upgrade their hardware (if necessary) prior to coin halving in what is typically a positive market environment. We could also do it six months after coin halving but I think prior to coin halving makes the most sense. Regardless, arbitrarily picking two years does not seem like a good idea and is a tad extreme.

So if we decide three years makes more sense and this CHIP passes, the first filter halving would be 10/1/2023, six months before coin emission halving on 4/1/2024, which is approximately eight months from today.

[mech2you]

To me it sounds like just pushing the problem into the future. What will happen if in a few years the network size is 2 times bigger and the nodes are mostly at k32? There is no incentive to plot on k33, k34 or kXX. However, reducing the size of the filter does not reduce the memory requirements for creating a plot. At some point, ASIC miners will come and replace the GPUs. Therefore, in my opinion, increasing the K size oder changing the plot format makes more sense, even if not everyone is willing to do it. By increasing the memory requirements when creating the plots, this ensures that no ASICs are developed in this direction because RAM memory is still expensive. Better to lose some network size now than face the same problem in 7 years. I myself have a 1.5PiB farm with about 70% k32 plots. Every farmer knew that the moment will come when the K32 size is not enough ;)

[Motophan]

Here is how I see it -> lets say chia corp goes bankrupt. I do not see considerations for this scenario.

Startup tech companies have a very high likelihood (>90%) of going bankrupt. Ones who have less revenue than expenses are more likely to fail.

This CHIP will send us to a plot filter of 32 and stay there forever.

I suggest doing a plot filter change on a as needed basis and not schedule them as its (extremely) likely chia corp will not be around to reset the plot filter and set it to k33. Chia corp will not have to live w/ the problems they create. We will.

[reythia]

Reducing plot filter towards 32 seems sensible, and preferable as the first lever, but is difficult to consider in isolation.

At some point a lower filter necessitates an increase in k.

Cumulative 16x decrease in farmable space with a given setup will catch out many farmers, especially those about to plot with compression levels that appear to have plenty of headroom. They'll either have to upgrade hardware with increasing aggression or replot in some form - to higher k or lower compression.

If k=34 by the time filter=32 that reduces the increase in farming demands to 4x over ~8 years. Feels like a sensible balance on multiple fronts and provides a predictable environment for farmers.

Perhaps a cadence of filter-filter-k-filter-filter-k? With an 18 month or "half halving" interval.

[SlowestTimelord]

I suggest doing a plot filter change on a as needed basis and not schedule them as its (extremely) likely chia corp will not be around to reset the plot filter and set it to k33. Chia corp will not have to live w/ the problems they create. We will.

The chain does not need Chia Network Inc to incorporate any changes. The client can be forked and the farmers can decide which version to run if CNI disappeared off the face of the earth tomorrow. Also the scheduled plot filter change is purposely more aggressive than needed because relaxing the plot filter schedule would be a soft fork while accelerating it would be a hard fork -- so you'd want to default to a more aggressive schedule to be conservative. It would be far more disruptive to make plot filter changes "as needed" since each would need to be a hard fork.

[danieljperry]

Thanks for writing such a thorough explanation of the problem and proposed solution @jmhands! A few clarifying questions that I have already asked you in other channels, but adding here for completeness:

1. Does reducing the plot filter by half also double plot "decompression" farming workload? Or is it not exactly proportional?

2. Upon reaching size 32 plot filter, does that imply increasing minimum k-size to k33 is the logical next proactive step? At that time would plot filter reset to 512 or stay at 32?

3. Is increasing the minimum k-size a soft fork?

4. In the event of a hard fork, would that be an opportunity to introduce other pending changes that require a hard fork? What such changes might those be?

5. Would increasing the number of signage points per subslot (to decrease the time between infusion points) be a viable option as well or is that too drastic a change to the consensus?

1. Yes, all other things being equal, this would double the farming workload. However, keep in mind that the workload is minuscule to begin with.
2. We have no plans for changing either the filter or the minimum k size after this proposal runs out. The reasons for this are that
   a. At a plot filter of 32 (keeping in mind that this will be 8 years from now), low end machines will still be able to farm Chia using HDD storage, and
   b. It's difficult to predict the state of the storage industry that far into the future. Conventional wisdom says that SSD storage will be cheaper than HDD storage at that time, which will mean that most of the netspace will be on SSDs as well. If that's the case, there may not be a need for a filter at all, but we simply don't know if this prediction will come true yet.
3. Yes. A soft fork is a tightening of the rules, and a hard fork is a loosening of the rules.

• Increasing k would mean that some plots (k-32) would no longer work, therefore the rules would have been tightened so it would be a soft fork.
• Reducing the filter would mean that some plots that did not pass the filter before the change would pass the filter after the change. This would be a relaxation of the rules and therefore would be a hard fork. Following this logic, if we were ever to increase the filter, it would be a soft fork.

4. Potentially yes, but we don't have any such plans at this time. If such plans did emerge, they would be proposed in a new CHIP. This is because each CHIP needs to focus on a single proposal.
5. To my knowledge changing the number of signage points per subslot wasn't discussed, though other consensus changes were. My initial thought about this idea is that it would make plot grinding more expensive and more difficult, but it wouldn't achieve significantly more than simply decreasing the filter. This would also be a large consensus change, it could affect the stability of the network, and it could potentially open the network to new attack vectors. While alternate proposals are certainly welcome, I don't see this one being viable (unless I'm mistaken about something fundamental).

[danieljperry]

A few more thoughts to stir the pot in favor of Option 2 (k-size change):

* If there were ever a time to declare timelines for a minimum k-size, it's probably now while folks are considering replotting anyway.

* I don't think a proposal on plot filter decrease is complete without considering when/how minimum k-size requirement might fit into the ~10 year timeline.

* Plot filter changes + decompression considerations adds a dimension that many farmers may have trouble grasping or considering whereas a minimum k-size timeline is easier to grasp (say, k33 phases in 2025, k34 phases in 2028 etc.)

* Given the unpredictability of future hardware and plotting/farming advances, there's a chance another hard fork to change the timeline of plot filter changes may be necessary whereas as Bram has mentioned in the past a minimum a k-size change would be a soft fork (I'd like to confirm whether or not this is the case).

Increased hardware requirements for plotting. A k=34 plot would be very difficult to create in RAM without using a high-end server

Maybe it's my experience from early plotting days but I've always viewed the ability to plot all in RAM (and now to plot with GPU+RAM) as a nice-to-have and not something farmers should feel is necessary or feel entitled to be able to do.

These are all fair points and the community will need to carefully weigh the options, just as CNI already has done.

[danieljperry]

What is the plan after 2030?

No plans yet. We have a decent idea of what the storage and GPU industries will look like in a few years, but it's difficult to predict what will happen 7+ years from now. However, the two primary mitigations we have against plot grinding are -- and likely will remain -- reducing the filter and increasing the minimum k.

[danieljperry]

passes, the first filter halving would be 10/1/2023, six months before coin emission halving on 4/1/2024, which is approximately eig

JM is the expert here, but I can say that the timeline from this proposal was not chosen arbitrarily. It was chosen based on the likely advancements in the GPU, memory, and storage industries over the next few years. The exact block heights where the changes will be implemented is something that can be discussed on Discourse, which will be set up soon.

[danieljperry]

To me it sounds like just pushing the problem into the future.

Correct, this proposal is meant to make plot grinding uneconomical over the next 8 years, but not forever.

What will happen if in a few years the network size is 2 times bigger and the nodes are mostly at k32?

The network size has nothing to do with this proposal. Most plots will likely still be k-32 because they are the easiest to create, though larger plots do require less resources while farming.

At some point, ASIC miners will come and replace the GPUs.

We don't foresee this happening (assuming you mean "plotters" and not "miners"). If you have any data to back this up, please share it.

[lord-icon]

When considering K32 on K33:
Is it technically possible to convert 2xK32 plots to a K33?
Replotting 10,000 K33s fails the CHIA policy.

[danieljperry]

It might be possible but it will require more time and energy than simply creating a new k33. However, this proposal does not require replotting. That decision remains entirely yours, regardless of whether the proposal succeeds or fails to gain adoption.

[RudolfAchter]

This is the Plot Filter proposal from GPU Plotting is Real – and Very Fast right?
Am i reading correct if this plot filter proposal is implemented then there will be a hard fork in the blockchain?
So farmers have to update to a specific software version before a certain block is reached?
Would have no problem with it. Just want to clarify.

I think also with blockchain there is always a need to upgrade software to adapt to technology development.

[danieljperry]

This is the Plot Filter proposal from GPU Plotting is Real – and Very Fast right? Am i reading correct if this plot filter proposal is implemented then there will be a hard fork in the blockchain? So farmers have to update to a specific software version before a certain block is reached? Would have no problem with it. Just want to clarify.

I think also with blockchain there is always a need to upgrade software to adapt to technology development.

This is correct, yes.

[keliew]

a. At a plot filter of 32 (keeping in mind that this will be 8 years from now), low end machines will still be able to farm Chia using HDD storage, and

Is the definition of a "low end machines" still a RPi4? Or will the minimum requirement increase due to this change. Obviously in this scenario, the farmer is not farming compressed plots.

[danieljperry]

This CHIP is now a Draft. If it is accepted, it will require a hard fork of Chia's blockchain. Therefore it is imperative that we have consensus from the community. Please continue to leave your reviews here.

You can also discuss this CHIP in the #chips channel on Keybase, or on Discourse:
https://developers.chia.net/t/chip-0012-plot-filter-reduction/810

Additionally, we will hold a public Zoom call to discuss this CHIP at 8 AM PST on Wednesday, February 22. See the #chips channel on our Keybase for details.
https://keybase.io/team/chia_network.public

[danieljperry]

a. At a plot filter of 32 (keeping in mind that this will be 8 years from now), low end machines will still be able to farm Chia using HDD storage, and

Is the definition of a "low end machines" still a RPi4? Or will the minimum requirement increase due to this change. Obviously in this scenario, the farmer is not farming compressed plots.

The definition of "low end machine" will likely remain within the same genre for the foreseeable future. For example, the min spec machine could become a RPi5, but it won't be a workstation.

[danieljperry]

Here is a recording of the discussion of this CHIP:
https://youtu.be/hf01z8Uhj1Y

Please continue to leave your reviews here.

[SlowestTimelord]

Thanks for the informative public zoom call today. Wanted to summarize some thoughts:

• Option 1 (plot filter reduction) makes much more sense to me than Option 2 (increase minimum k-size) if only because Option 2 likely still requires a plot filter reduction, which almost makes Option 1 strictly better. But I would still like to see documented guidance about what situations would warrant a k-size increase? If I understand correctly from the call today, it is when multi-GPU plotting can economically do k32s entirely in combined VRAM.

• There needs to be more definitive direction/education for farmers looking to do compressed plot farming at higher compression levels (C6+) because plot filter reduction will proportionally increase decompression workload and hence proportionally reduce maximum farm size (by up to 16x). Plotting and farming now requires longer term planning for farmers looking to min/max their compression.

• Probably a moot point but re: validity of proofs from older clients. Plots that pass the 512 filter will also pass lower plot filters. It was brought up on the call that clients that don't upgrade have a high likelihood of farming on the wrong chain due to rejecting valid proofs that the most of the network sees as valid.
  Seeing "Farm Sooner" on the 2023 roadmap is interesting because I think a "light wallet" farmer relying on full node peers might be able to continue farming 512 filter plots at their own disadvantage. This is only relevant if "Farm Sooner" is released well before a plot filter reduction.

• Given the expected eventual SSD flippening, wouldn't it make sense to also put in a plot filter reduction to 1 in the distant future to reduce likelihood of another hard fork? e.g. plot filter to 1 reduction in 2040. This may be aggressive but again, this can be pushed out via soft fork if we realize HDD farming is still a non-negligible portion of the netspace in 2040.

[danieljperry]

Thanks for these thoughtful comments. We will take them all into consideration.

[soloHeroo]

I have 100,000 k32 plots😓😓

[danieljperry]

I have 100,000 k32 plots😓😓

This CHIP does not affect the validity of k32 plots.

[SlowestTimelord]

Great points @Voodoo-Chia. There's also a high likelihood SSDs will make up much more of the netspace by the 3rd or 4th plot filter reduction, which also dampens the impact of look ups (though not decompression).

Now that we have established block height (date) proposed, I would like to see this proposal move forward in the CHIP process along with #57. I have not seen any objections to the CHIP that haven't already been addressed by the comments above.

[CryptoBlockchainTechnologies]

@SlowestTimelord
I will 2nd that motion. My original objection I stated here was to extend plot filter having to three years. Now that this has been done, it has my support.

[danieljperry]

Thanks everyone! Given your your feedback, along with projections for the advancement of hardware, we have updated the proposal to use a 3-year cadence for reduction, starting in one year. We will be moving this CHIP along in the review process soon. To make it absolutely clear, here is the current proposed schedule:

Block height	Month/Year (approx)	Filter size
5 496 000	June 2024	256
10 542 000	June 2027	128
15 592 000	June 2030	64
20 643 000	June 2033	32

Additional feedback and reviews are still welcome!

[danieljperry]

This CHIP is in Review. Please make your final reviews soon.

[digitalspaceport]

Full support, like the updated filter reduction cadence. Glad to see it in review with specific blocks. Could we get estimated dates for those blocks?

…

________________________________ From: Dan Perry ***@***.***> Sent: Monday, May 22, 2023 9:12:27 AM To: Chia-Network/chips ***@***.***> Cc: Digital Spaceport ***@***.***>; Comment ***@***.***> Subject: Re: [Chia-Network/chips] CHIP-12 -- Reduce Plot Filter (PR #53) This CHIP is in Review. Please make your final reviews soon. — Reply to this email directly, view it on GitHub<#53 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAOIVH632JYA6MHMM2PR2D3XHNX4XANCNFSM6AAAAAAUBDQTOA>. You are receiving this because you commented.Message ID: ***@***.***>

[SlowestTimelord]

The month estimates (especially for later reductions) might be as accurate as we can get. With the ASIC timelords coming online the exact corresponding date could have multiple days/weeks of variation?

[danieljperry]

We will minimize the impact of the ASIC timelords by introducing them slowly. However, other events such as a rapid increase in netspace could move the block heights by a few hours or days. We won't be able to give a more granular estimate than month and year.

[geraldneale]

Let's call this what it is, a tendency away from PoST and towards PoW.

[SlowestTimelord]

Let's call this what it is, a tendency away from PoST and towards PoW.

This is exactly the opposite. It's a lever to ensure PoST doesn't become PoW.

[danieljperry]

Let's call this what it is, a tendency away from PoST and towards PoW.

Can you explain in more detail why you hold this opinion?

[geraldneale]

PoST always had a POW component therein because there is no other way to make plots other than doing work. Also, doubling lookups requires doubling the work for this component as this CHIP clearly describes. Also, the new compression algorithms move work to the CPU compared to the old. Also, replotting means more work.

PoST advocates, as I still consider myself to be, should always recognize these facts. Proof of Space Time would have been better labelled Proof of Space Time and Work(PoSTW) from the beginning because work is NOT removed by Chia Consensus only managed better. The key to managing work is recognizing it imo.

There is clearly a precedent in Bitcoin that POW works well, but over-reliance on work tends to centralize and Chia is tending in that direction with this CHIP and other updates imo. I'm not offering an alternative, because perhaps it is inevitable, except enhanced disclosure so more folks can understand why at least a minimum amount of work is necessary, but even with incremental increases it is not necessary in the same abundance as Bitcoin. I think this is important for the sake of mass uptake and therefore security.

[danieljperry]

Thanks for the feedback!

Responses in line:

PoST always had a POW component therein because there is no other way to make plots other than doing work.

True, but irrelevant for this CHIP.

doubling lookups requires doubling the work for this component as this CHIP clearly describes.

Also true, but it only doubles the lookups a farmer must perform. No new hashes are generated. No new data is written. It's doubling the amount of work, but reading data from a disk is not the same as the Proof of Work consensus.

the new compression algorithms move work to the CPU compared to the old.

Irrelevant -- This CHIP is reacting to the new compression algorithms, but it is not the cause of those algorithms, which would exist regardless of the existence of this CHIP.

replotting means more work.

Irrelevant -- This CHIP does not necessitate any replotting.

There is clearly a precedent in Bitcoin that POW works well, but over-reliance on work tends to centralize and Chia is tending in that direction with this CHIP and other updates imo.

Do you have other reasons to believe that Chia is trending in the direction of PoW that are specific to this CHIP?

[...] enhanced disclosure so more folks can understand why at least a minimum amount of work is necessary, but even with incremental increases it is not necessary in the same abundance as Bitcoin. I think this is important for the sake of mass uptake and therefore security.

Agreed, full disclosure is very important here. That's why this CHIP has been open for five months and counting.

[hoffmang9]

PoST always had a POW component therein because there is no other way to make plots other than doing work. Also, doubling lookups requires doubling the work for this component as this CHIP clearly describes. Also, the new compression algorithms move work to the CPU compared to the old. Also, replotting means more work.

Speaking holistically and not to just this CHIP - on net the cost of plotting in both time and CPU power has fallen/is being amortized over farming. When looking at a new entrant, the amount of time and CPU/GPU that they have to allocate to be up and providing material additional security is greatly decreased by all these changes.

As Chia blockchain scales, that next 25 EiB will be brought on much more cheaply than before the new methods to create smaller plots on disk existed.

[geraldneale]

If there's one thing I "know" it is that nobody knows the future. Where does it say in this CHIP why the filter adjustment can't be more like Bitcoin where the difficulty adjustment is based on feedback from the network, not the solely the calendar(e.g. one year out and then every 3 years)? Sorry if I missed it. Since it requires a hard fork, perhaps this is an opportunity to build something that presumes less about the future.

[hoffmang9]

The designed from the start mechanism to defend against the ongoing increase in memory bus speed between parallel processors is the raising of the minimum K size. This was an opportunity to use a less blunt instrument, but if the plot filter decreases are not enough, then we revert to the original plan that's always been there.

[geraldneale]

• ass the 512 filter will also pass lower plot filters. It was brought up on the call that clients that don't upgrade have a high likelihood of farming on the wrong chain due to rejecting valid proofs that the most of the network sees as valid.

I was wondering why this wasn't programmed in from the start since it seems so systematic, but I used up all of my adversarial stance question points today ;) Thank you for answering the implied question.

[danieljperry]

This CHIP is now in Last Call. If no further changes are required in the next two weeks (roughly), it will be moved to Final.

[danieljperry]

This CHIP is now Final. No further changes are allowed.