Beta 18

[hoffmang9]

[1.0beta18] aka Beta 1.18 - 2020-12-03

Added

• F1 generation in the plotter is now fully parallel for a small speedup.
• We have bitfield optimized phase 2 of plotting. There is only about a 1% increase in speed from this change but there is a 12% decrease in writes with a penalty of 3% more reads. More details in PR 120. Note that some sorts in phase 2 and phase 3 will now appear "out of order" and that is now expected behavior.
• Partial support for Python 3.9. That includes new versions of Chia dependencies like chiabip158.

Changed

• We have moved from using gulrak/filesystem across all platforms to only using it on MacOS. It's required on MacOS as we are still targeting Mojave compatibility. This should resolve Windows path issues.
• We upgraded to cbor 5.2.0 but expect to deprecate cbor in a future release.

Fixed

• A segfault caused by memory leaks in bls-library has been fixed. This should end the random farmer and harvester crashes over time as outlined in Issue 500.
• Plotting could hang up retrying in an "error 0" state due to a bug in table handling in some edge cases.
• CPU utilization as reported in the plotter is now accurate for Windows.
• FreeBSD and OpenBSD should be able to build and install chia-blockchain and its dependencies again.
• Starting with recent setuptools fixes, we can no longer pass an empty string to the linker on Windows when building binary wheels in the sub repos. Thanks @jaraco for tracking this down.

[hoffmang9]

Bitfield backprop is causing Windows 10 and Catalina to run out of files when plotting a 128 bucket k27 with the GUI.

STDERR:

stderr: 17:05:51.931  src.plotting.create_plots        : INFO     Creating 1 plots of size 27, pool public key:  a550e1d5db57eb4eea4da60bdfbc2056bb3a243666b7da93ad00659b8a63de2db7e118cb077fadf4c35106746035959a farmer public key: a07b193d0b609eae469a683b598070481e8c22acd3167d293f8fc38aaebcb9c02a2452f49e30506a1b58b121b95f7fa2
stderr: 17:05:51.955  src.plotting.create_plots        : INFO     Starting plot 1/1
stderr: Traceback (most recent call last):
  File "/Users/hoffmang/beta/chia-blockchain/venv/bin/chia", line 33, in <module>
    sys.exit(load_entry_point('chia-blockchain', 'console_scripts', 'chia')())
  File "/Users/hoffmang/beta/chia-blockchain/src/cmds/chia.py", line 59, in main
    return chia(args, parser)
  File "/Users/hoffmang/beta/chia-blockchain/src/cmds/chia.py", line 53, in chia
    return args.function(args, parser)
  File "/Users/hoffmang/beta/chia-blockchain/src/cmds/plots.py", line 146, in handler
    create_plots(args, root_path)
  File "/Users/hoffmang/beta/chia-blockchain/src/plotting/create_plots.py", line 137, in create_plots
    plotter.create_plot_disk(
RuntimeError: std::exception

Plotter output:
k27-128buckets.log

Please take a look @arvidn

[arvidn]

my suspicion is that the regression is caused by there now being 2 SortManager objects alive at the same time. Whereas previously (I believe) we would just use one at a time and then "flatten" the tables in between sorting. Now we save some I/O by reading from one sort manager and writing directly into the next.

On MacOS, the file handle limit per process is unusually low, 256. And two sort managers with 128 Buckets sounds like it would just about exceed that limit (given there are probably a few other random file handles open too). I'm surprised about windows having an issue too, historically windows has had a very high limit of file handles per process (in the order of 32k, iirc).

However, when reading from a sort manager, we should only need one file handle open at a time.

[wjblanke]

AOK

[hoffmang9]

Merged by hand