proof-producer: introduce command chain

[oclaw]

Implemented resource subscription mechanism based on boost::signals + command_chain for achieving smart resource management. Now each step of the executing command keeps its own copy of shared pointer to the data it uses

Rewrote all prover command to the new syntax. Now each command is located into its own file and represented as queue of executing steps (each step releases references to the resources after popping from the queue)

New approach allows:

• keep different parts of proof-producer independently
• lazy-initialize large objects (LpcScheme, AssignmentTable)
• release these resources automatically based on the steps added to the command

Related changes in crypto3:

• common_data is now placed into shared_ptr as far as it is used as an independent data part in most cases
• Added reset of KZG scheme in unit tests

Also added ResultCode types for identifying the exact reason of failure externally

[makxenov]

Unused

[oclaw]

It still might be used, will remove if there will be no need for it

[github-actions]

Clang Test Results

  174 files  ± 0    174 suites  ±0   16m 15s ⏱️ +15s
8 545 tests +97  8 539 ✅ +97  6 💤 ±0  0 ❌ ±0 
8 611 runs  +97  8 605 ✅ +97  6 💤 ±0  0 ❌ ±0 

Results for commit 7cdb98d. ± Comparison against base commit 331cae7.

This pull request removes 6 and adds 103 tests. Note that renamed tests count towards both.

pack_imploder_test_suite ‑ be_to_be_1
pack_imploder_test_suite ‑ be_to_be_2
pack_imploder_test_suite ‑ be_to_le_1
pack_imploder_test_suite ‑ be_to_le_2
pack_imploder_test_suite ‑ be_to_le_3
pack_imploder_test_suite ‑ bubb_to_lulb_4

lpc_math_polynomial_suite ‑ lpc_basic_skipping_layers_test
lpc_math_polynomial_suite ‑ lpc_dfs_basic_test
lpc_params_test_suite ‑ lpc_batches_num_3_test
lpc_params_test_suite ‑ lpc_different_hash_types_test
pack_equal_test_suite ‑ bubb_to_bubb_1
pack_equal_test_suite ‑ bubb_to_bubb_2
pack_equal_test_suite ‑ bubb_to_bubb_3
pack_equal_test_suite ‑ bubb_to_bulb_1
pack_equal_test_suite ‑ bubb_to_bulb_2
pack_equal_test_suite ‑ bubb_to_lubb_1
…

♻️ This comment has been updated with latest results.

[notion-workspace]

Event based resource management

[github-actions]

Gcc Test Results

  174 files  ±0    174 suites  ±0   17m 18s ⏱️ -3s
8 545 tests ±0  8 539 ✅ ±0  6 💤 ±0  0 ❌ ±0 
8 611 runs  ±0  8 605 ✅ ±0  6 💤 ±0  0 ❌ ±0 

Results for commit 7cdb98d. ± Comparison against base commit 331cae7.

♻️ This comment has been updated with latest results.

[makxenov]

General question. We have 2 similar terms: proof producer (as top-level directory, Nix derivation name, name of binaries) and proof generator (namespace for source files inside proof-producer/, name of libraries). Do these terms have any differences in the meaning? If no, let's unify it to avoid possible confusions.
@akokoshn @oclaw @x-mass

[akokoshn]

nice to have print here name of the stage together with number

[akokoshn]

General question. We have 2 similar terms: proof producer (as top-level directory, Nix derivation name, name of binaries) and proof generator (namespace for source files inside proof-producer/, name of libraries). Do these terms have any differences in the meaning? If no, let's unify it to avoid possible confusions. @akokoshn @oclaw @x-mass

@martun , do you remember why was used proof_generator namespace but not proof_producer?

[akokoshn]

generally look good.
Let's wait for @martun review

[martun]

Such a complex way of adding the next step, why not accept a command_step here, and construct it outside?

[oclaw]

It is complicated only here (not in usage) and the pattern is much similar to make_shared . IMO such approach reduces possible resource leaks caused by unneeded copies of command_step instances outside the queue

[martun]

The rest of the project is throwing exceptions. You will still need to have a "catch block" at the start of each command to report those exceptions if there will be any. Maybe here too you could just throw standard std::exceptions.

[oclaw]

Last time I saw entire proof-producer is not throwing exceptions, maybe your take is more relevant to the crypto3 internals

I, actually, do not mind switching to an exception-based mechanism of error handling, but it is a topic for a separate discussion
cc @akokoshn

[martun]

Do we use this "Args" structure somewhere else? Can't we just accept the arguments here, and delete the "Args" structure from everywhere?

If we use the Args, I would suggest to mark somehow which arguments are inputs, which are the outputs, and when printing the error messages from the "execute" function of command_step.hpp, print the name of the command and the parameters it was called with as a part of the error message. This way will use the Args in some way.

[oclaw]

Separate args structs will help us to split command line options in future. I thought to do it during the initial refactoring too but amount of LoC made me to reconsider :)

[martun]

You can move the scope { to the previous line.

[martun]

It looks a bit strange that we pass the same object twice, because it contains inside it 2 object that class "Prover" will need to access. Why not extract those objects (the public data and LPC scheme) and pass them to this class? The benefit of creating class "resources::resource_provider" is to load the resource from a file only when required? Is it worth it writing so much code for that purpose?

[oclaw]

This is the part I am not proud of too :(

I think it could be improved with some another factory methods or some kind of dependency injection but it will bring much more complications to an already complicated logic

[akokoshn]

duplicate?

[makxenov]

Nitpick: file still has trailing whitespaces

[oclaw]

I actually wanted to include 'clang-format' run to this PR but it would turn the whole diff to a mess
I suggest to create a precommit hook with implicit call of it in further development :)