Rust bindings to OCaml native objects #6545

mrmr1993 · 2020-10-29T17:48:53Z

This PR implements rust bindings allocated directly on the OCaml heap.

Much of the code deals with conversions from rust types to OCaml types
- This is especially true for Vec<_> <-> _ array. We are careful to avoid passing non-OCaml-heap Vec<_>s as OCaml objects, since they appear 'small' to the OCaml GC (they are effectively a small wrapper around a pointer) but may point to an arbitrarily large amount of memory in the rust heap.
The types that we were previously converting to OCaml versions are now done on the rust side
- This avoids allocating large opaque objects in the OCaml heap, or passing rust heap pointers into the OCaml heap
- This should also be slightly more efficient, since we are 'closer to the metal' in rust and don't have to back-and-forth through OCaml's value kind
All of the exposed types are duplicated, on both the OCaml side and the rust side
- This is a safety measure: these types must be kept in sync! If the OCaml and rust versions desync, we may be misinterpreting types incorrectly, which would be catastrophic.
- This results in a lot of boilerplate code in rust to deal with the wrapper types that we need to progress.
- It would be useful one day to write a rust macro to generate the OCaml type corresponding to the rust type we are deriving for, and then lower the OCaml conversions to the types in the zexe/marlin repos. Today is not that day.
- In OCaml this results in some record wrapping and unwrapping to transfer the bindings' types into the pickles types. We can elide some of this with type aliasing between the different versions, but otherwise this seems like a small, acceptable cost for safety.
Rust error cases are converted to OCaml errors

Given the size of this PR, it may be valuable to review commit-by-commit. Excepting a few tidying up PRs, the vast majority add the bindings for a single type or type family (e.g. fields, groups, urs, etc.).

To test the bindings in utop with dune, I have found it useful to add printers for the fields, as below.

$ dune utop src/lib/marlin_plonk_bindings/
───────────────┬─────────────────────────────────────────────────────────────┬────────────────
               │ Welcome to utop version 2.3.0 (using OCaml version 4.07.1)! │
               └─────────────────────────────────────────────────────────────┘
Findlib has been successfully loaded. Additional directives:
  #require "package";;      to load a package
  #list;;                   to list the available packages
  #camlp4o;;                to load camlp4 (standard syntax)
  #camlp4r;;                to load camlp4 (revised syntax)
  #predicates "p,q,...";;   to set these predicates
  Topfind.reset();;         to force that packages will be reloaded
  #thread;;                 to enable threads


Type #utop_help for help about using utop.

─( 17:40:40 )─< command 0 >────────────────────────────────────────────────────{ counter: 0 }─
utop # open Marlin_plonk_bindings;;
─( 17:40:40 )─< command 1 >────────────────────────────────────────────────────{ counter: 0 }─
utop # let print_fp fmt x = Format.pp_print_string fmt (Tweedle_fp.to_string x);;
val print_fp : Format.formatter -> Marlin_plonk_bindings_tweedle_fp.t -> unit =
  <fun>
─( 17:40:52 )─< command 2 >────────────────────────────────────────────────────{ counter: 0 }─
utop # let print_fq fmt x = Format.pp_print_string fmt (Tweedle_fq.to_string x);;
val print_fq : Format.formatter -> Marlin_plonk_bindings_tweedle_fq.t -> unit =
  <fun>
─( 17:40:55 )─< command 3 >────────────────────────────────────────────────────{ counter: 0 }─
utop # #install_printer print_fp;;
─( 17:41:00 )─< command 4 >────────────────────────────────────────────────────{ counter: 0 }─
utop # #install_printer print_fq;;
─( 17:41:05 )─< command 5 >────────────────────────────────────────────────────{ counter: 0 }─
utop #

Sample output (verification key parsing bindings test):

utop # let urs = match Tweedle_fq_urs.read "/tmp/coda_cache_dir/tweedledum_18_v2" with | Some x -> x | None -> assert false;;
val urs : Marlin_plonk_bindings_tweedle_fq_urs.t = <abstr>
utop # let vk' = Tweedle_fq_verifier_index.read urs "/tmp/coda_cache_dir/vk-step-transaction-snark-0-4fdccb9b478f189064a91400df4667b7";;
val vk' : t =
  {domain =
    {Marlin_plonk_bindings_types.Plonk_domain.log_size_of_group = 16;
     group_gen =
      19044435143061987783228261618356440625338966967281119294414704748777318225301};
   max_poly_size = 262144; max_quot_size = 327685; urs = <abstr>;
   evals =
    {Marlin_plonk_bindings_types.Plonk_verification_evals.sigma_comm0 =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (877543031817688733121476498165209218248348666409573215356203686608453318525,
            17722669863408732729755214290497967447722480324566981412346229231382279785500)|]};
     sigma_comm1 =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (28509887837982487832067520103637713388947697988911054821407771458345756172637,
            24467550716972961200383840776987711883369782442833518454663295876712307474828)|]};
     sigma_comm2 =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (18961256850892043953373807845720638332683842568098228278847861270817640742559,
            13236189716195075157982480840193765749157712951210298944081877615899049983534)|]};
     ql_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (6660070206784024567224438279432569008481434342189153165499436418227784966901,
            23452470202699940846588717978251298978505290827234167534585170256745313298413)|]};
     qr_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (28698628638803889923446162350203408084593795905376999149869256388029031405272,
            9504731251592849976761625516961736508917878194882354076763000021407629641207)|]};
     qo_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (3903594666551180723715543631871075965254049333751811999321439467613176976205,
            27635301537242995152424283719812290380926837719998544991200669545609385673405)|]};
     qm_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (19846296767775445425661982727018641506349511604804061578187597184215454020516,
            10169030497955827579911624150038315151633183130034914609690595773093457907688)|]};
     qc_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (14442566999971039516539188635497984029567978036224519697773222173006962366959,
            26826424002574313332987177120969199915175636857747557265326853628196032179085)|]};
     rcm_comm0 =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (10596206016340999673967431792160845237995551364483621262403458198900831725152,
            290467027465180508021345970527291816781686329263630764037586860876233210186)|]};
     rcm_comm1 =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (23395410579898971357541456367876814299640738296658831535860710095374844957019,
            9357195542300446996632519060025870048947167674081534937490319797451500572523)|]};
     rcm_comm2 =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (21526761103340557017756230799652951080296301351742886224935360603069411594607,
            14989144201036375030821950563750665695138978972721137632609752952570039741094)|]};
     psm_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (23259765037161000206310261463227701647922328101290580745097393486586584412699,
            4341476780643537556036146734852568379112496163183246090319660257393995934914)|]};
     add_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (14777393917163206044219903296108058846097051771049847669598430559737992548830,
            2269475990671192482632721803840473667525908005190864960354219120257887504269)|]};
     mul1_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (23290844110063490122675681660994333189059483211179173028330571170596264407161,
            26906126276464352552304573409576859789823761145641631214301217702388494945153)|]};
     mul2_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (13746881869141654521832776044999144589077988007305980488192404856321424748368,
            14418734774138784417535874025381552658120039649570161491242161566407008680606)|]};
     emul1_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (25201537854411348770051631999511395686122628971152634897425797213492274905475,
            7367822245271661658628149826812829283558930398075055765486456935202317282871)|]};
     emul2_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (20040430717121254770434702842449121750771523797215608232525164177394651348874,
            3677051471782146313214844365222737313060526096915796068463789508484238557883)|]};
     emul3_comm =
      {Marlin_plonk_bindings_types.Poly_comm.shifted = None;
       unshifted =
        [|Marlin_plonk_bindings_types.Or_infinite.Finite
           (16376218848904891014596772514133259759624089788681670586960429891539631965239,
            13497633861762563118566917102457832888429893588964679302620439027756358494211)|]}};
   shifts =
    {Marlin_plonk_bindings_types.Plonk_verification_shifts.r =
      328286983623303317637963920346571898945724874896624808297627776768640590563;
     o =
      220790353665890403705559231885806581221301230221265349993193424985261418438}}

Checklist:

Document code purpose, how to use it
- Mention expected invariants, implicit constraints
Tests were added for the new behavior
- Document test purpose, significance of failures
- Test names should reflect their purpose
All tests pass (CI will check this if you didn't)
Serialized types are in stable-versioned modules
Does this close issues? List them:

These do some low-level fiddling to make sure that we strictly stick to the allocations necessary. The result is unpleasant, but works.

mrmr1993 · 2020-10-29T19:02:00Z

Updated to add a couple of bindings that were missing.

psteckler · 2020-10-29T19:24:16Z

src/lib/marlin_plonk_bindings/stubs/src/tweedle_dum.rs

+    CamlTweedleFp(endo_q)
+}
+
+#[no_mangle]


why does the annotation here differ from the one above?

Woops! Fixing.

imeckler

very cool -- had a few qs

imeckler · 2020-10-30T16:21:06Z

src/lib/marlin_plonk_bindings/stubs/src/tweedle_fp.rs

+}
+
+#[ocaml::func]
+pub fn caml_tweedle_fp_is_square(x: CamlTweedleFpPtr) -> bool {


I noticed some functions take CamlTweedleFp and some take CamlTweedleFpPtr -- what determines that?

CamlTweedleFp is the representation, CamlTweedleFpPtr is effectively a &CamlTweedleFp where the reference points to the OCaml heap. We can write CamlTweedleFps, but reading them has to go through CamlTweedlePtr to handle the fact that the OCaml GC can relocate the data.

imeckler · 2020-10-30T16:22:54Z

src/lib/marlin_plonk_bindings/stubs/src/tweedle_fp_urs.rs

@@ -0,0 +1,171 @@
+use crate::tweedle_dee::{


Why are the srses special?

This mirrors what we did in the previous bindings, where we have 1 long-lived SRS and its references are used in any keys loaded with it. IIRC it's several MBs too, so not ideal to store in the OCaml heap as a custom blob.

netlify · 2020-10-30T20:10:35Z

Deploy preview for mina-staging ready!

Built with commit 7c215a8

https://deploy-preview-6545--mina-staging.netlify.app

mrmr1993 added 21 commits October 26, 2020 21:29

Add Marlin_plonk_bindings library, Bigint_384 bindings

ad287fc

Add bindings for Bigint256

1601aa7

Update bigint bindings to borrow instead of copying

166e102

Add tweedle_fp bindings

9a09fde

Add bindings for tweedle_fq

46a7356

Add bn_382_fp and bn_382_fq bindings

07327f5

Reformat, make field contents public

e9d51f0

Add field element vector bindings

e8d233a

Add tweedle_dee bindings

505ecf2

Add tweedle_dum bindings

ad5528c

Add CamlTweedle{Dee,Dum}AffineVector wrappers

ffce8aa

These do some low-level fiddling to make sure that we strictly stick to the allocations necessary. The result is unpleasant, but works.

Add OCaml conversion support for PolyComm and Vec<PolyComm>

e28cd30

Add refcounted URS bindings

6a33d0e

Add verifier_index bindings

0a1bb21

Reformat bindings OCaml code

51e1afb

Add caml_vector helper library for bindings

90c2196

Rework caml_vector to be simpler, easier to use

dcf6228

Add index and gate vector bindings

f4817b2

Add proof bindings

61934ad

Add bindings for endo base/scalar

e69fa2a

Rename fields to match Pickles_types

9394c00

Rename Or_infinite to Or_infinity to match Pickles_types

7dbeadf

psteckler reviewed Oct 29, 2020

View reviewed changes

no_mangle -> ocaml::func

db83db3

imeckler reviewed Oct 30, 2020

View reviewed changes

imeckler added the ci-build-me Add this label to trigger a circle+buildkite build for this branch label Oct 30, 2020

imeckler approved these changes Oct 30, 2020

View reviewed changes

mrmr1993 added the ready-to-merge-into-develop label Oct 30, 2020

Add ppx stanzas required by lint

73fed09

Merge branch 'develop' into feature/rust-ocaml-heap

7c215a8

lk86 and others added 4 commits October 30, 2020 19:09

Merge branch 'develop' into feature/rust-ocaml-heap

262557d

Add preprocessing stanzas to test executables

970b340

Merge branch 'develop' into feature/rust-ocaml-heap

fb07bc4

Merge branch 'develop' into feature/rust-ocaml-heap

c22b84c

mergify bot merged commit 88f1c45 into develop Oct 31, 2020

mergify bot deleted the feature/rust-ocaml-heap branch October 31, 2020 06:04

mrmr1993 mentioned this pull request Nov 10, 2020

Uplift snarky_bn382 from the zexe repo #6464

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rust bindings to OCaml native objects #6545

Rust bindings to OCaml native objects #6545

mrmr1993 commented Oct 29, 2020 •

edited

Loading

mrmr1993 commented Oct 29, 2020

psteckler Oct 29, 2020

mrmr1993 Oct 29, 2020

imeckler left a comment

imeckler Oct 30, 2020

mrmr1993 Oct 30, 2020

imeckler Oct 30, 2020

mrmr1993 Oct 30, 2020

netlify bot commented Oct 30, 2020

Rust bindings to OCaml native objects #6545

Rust bindings to OCaml native objects #6545

Conversation

mrmr1993 commented Oct 29, 2020 • edited Loading

mrmr1993 commented Oct 29, 2020

psteckler Oct 29, 2020

Choose a reason for hiding this comment

mrmr1993 Oct 29, 2020

Choose a reason for hiding this comment

imeckler left a comment

Choose a reason for hiding this comment

imeckler Oct 30, 2020

Choose a reason for hiding this comment

mrmr1993 Oct 30, 2020

Choose a reason for hiding this comment

imeckler Oct 30, 2020

Choose a reason for hiding this comment

mrmr1993 Oct 30, 2020

Choose a reason for hiding this comment

netlify bot commented Oct 30, 2020

mrmr1993 commented Oct 29, 2020 •

edited

Loading