Merge remote-tracking branch 'origin/develop' into properly-expose-pe…

…nding-coinbase-aux

CONTRIBUTING.md

@@ -43,7 +43,13 @@ labels.
 
 ## Pull Requests
 
-Our branching workflow is [documented on the wiki](https://minaprotocol.notion.site/minaprotocol/Compatible-vs-Develop-branch-management-31c845b2924b4c518740eb9da4514dcc).
+### Branching workflow
+
+Make sure to read about the [branching workflow](README-branching.md) to
+understand on which branch (`compatible` or `develop`) you should be working,
+and how to manage simultaneous PRs to both branches.
+
+### Continuous integration
 
 All pull requests go through Buildkite CI, which makes sure the code doesn't need to
 be reformatted, builds Mina in its various configurations, and runs all the
@@ -67,6 +73,9 @@ If you are PRing from the main remote, add `ci-build-me` label when you want to
 Once a PR has been reviewed and approved, and all CI tests have passed, it can be merged
 by a maintainer (or by you, if you have this access).
 
+If you encounter problems with the CI, read [CI FAILURES](README-ci-failures.md)
+for common troubleshooting steps.
+
 ## Documentation
 
 There are three main pieces of Mina documentation:
@@ -76,8 +85,10 @@ There are three main pieces of Mina documentation:
 2. The `README.md` files in various directories. These explain the contents of that
    directory at a high level: the purpose of the library, design constraints, anything else
    specific to that directory.
-3. Inline code comments. There are very few of these, and we don't currently run ocamldoc
-   so they won't be exposed even as we write them. This should change eventually.
+3. Inline code comments. There are unfortunately very few of these currently,
+   but this will slowly change. We are now running `ocamldoc` and the generated
+   documentation is browsable
+   [online](https://mina-docs.storage.googleapis.com/index.html).
 
 Changes to the software should come with changes to the documentation.
 

Makefile

@@ -299,9 +299,6 @@ genesis-ledger-ocaml:
 test-ppx:
 	$(MAKE) -C src/lib/ppx_mina/tests
 
-web:
-	./scripts/web.sh
-
 ########################################
 ## Benchmarks
 

README-branching.md

@@ -0,0 +1,78 @@
+# Branch management: `compatible` vs `develop`
+
+The first section describes the reasoning behind the current state of branching,
+and the second section gives you a few tips on how to handle this on a day to
+day basis.
+
+## Rationale
+
+Instead of a single `main` or `master` branch, there are two active branches
+`compatible` and `develop` at all times, where you might want to make changes.
+
+- `compatible` contains all changes which are literally backwards compatible
+  with what people currently run on mainnet.  Any nodes running a version of
+  mina based off of compatible should connect to current mainnet just fine.
+- `develop` contains changes which break backwards compatibility, or changes
+  that depend on past compatibility-breaking changes.  “Not backwards
+  compatible” means that a daemon running this version of mina will not connect
+  to mainnet.
+
+Major changes to the daemon, protocol, or crypto sometimes will sometimes cause
+backwards-compatibility breaking changes, and of course such changes need to be
+done with deliberation and are not to be taken lightly.  Changes to
+infrastructure, auxiliary develop scripts, tests, CI, are usually not be
+backwards compatibility breaking, and thereby should go into compatible (unless
+you are doing something very special and you know what you’re doing).
+
+On a semi-regular basis, `compatible` gets manually merged into `develop` so
+that — generally speaking — `develop` contains all changes in `compatible.` As
+such, `develop` is a superset of `compatible` i.e. `develop` contains everything
+that `compatible` contains, and more.
+
+### Hard fork
+
+Whenever a hard fork happens, the code in  `develop` is released. When this
+happens, the current `compatible` is entirely discarded and a new `compatible`
+gets created based off of `develop`
+
+![Illustration of the branching strategy](docs/res/branching_flow.png)
+
+### Releases
+
+`release/1.X.X` branches are made off of `compatible` and tagged with alpha and
+beta tags until the code is deemed stable, then the `release/1.X.X` branch is
+merged into `master` and given a stable tag. Whenever code is tagged, if
+anything is missing in in the upstream branches (compatible, develop) then the
+tagged branch is also merged back for consistency.
+
+`release/2.0.0`is the branch where Berkeley QA Net releases are being cut,
+between `compatible`and `develop` in the tree. So far nothing has been tagged
+there but there will be `2.0.0alphaX` tags once the code is more stable and we
+are closer to the Incentivized testnet.
+
+Unless it is an emergency, code should flow from feature branches
+into `compatible`then in batches into the release branch for tagging and testing
+
+## Day to day
+
+When developing a feature, if it’s not something that breaks compatibility, then
+you should be developing a feature branch, called `foo` for example, based off
+of `compatible`.  If you’re not sure whether or not your changes are breaking,
+they probably are not and should build upon compatible.
+
+There is a CI job called “merges cleanly to develop” which runs whenever you
+have a PR off of `compatible`.  If that CI job passes, then you can simply merge
+`foo` into `compatible`.  If it does not pass, then when you’re done with your
+changes to `foo` and the PR is all approved, then make a new branch+PR based off
+of your original PR  called `foo_DEVELOP` (for example), and then merge
+`develop` into `foo_DEVELOP`.  Fix any merge errors that result, then once
+`foo_DEVELOP` is approved, you can merge it into `develop`.  Once that’s done,
+the “merges cleanly to develop” CI job in your original `foo` PR should
+automagically start passing when you manually re-run it in CI, in which case you
+can merge.
+
+If, after making `foo_DEVELOP`, you need to make changes to `foo`, then make
+sure to merge `foo` into `foo_DEVELOP`.  In order for the git magic to work,
+`foo_DEVELOP` needs to be a superset of the commits from `foo`, and it also
+needs to merge first.  You can make further changes post-merge in `foo_DEVELOP`
+as needed to ensure correctness.

README-ci-failures.md

@@ -1,29 +1,59 @@
+## Jobs not starting
+
 CI jobs are dispatched by a script which responds to both the `ci-build-me`
 label and comments by MinaProtocol organization members containing exactly
 `!ci-build-me`. If your CI job has not started after adding the `ci-build-me`
 label, please comment on the pull request with `!ci-build-me` to attempt to
 re-trigger the script.
 
-**Please note:**
-* If you encounter an error where jobs are not run, it should normally suffice
-  to retry the script with a `!ci-build-me` comment on your PR when the fix has
-  been deployed.
-* If your CI error is related to a timeout logged by one of the integration
-  test runnners, this is a known issue and re-running the test in the CircleCI
-  interface will usually succeed.
-
 If CI jobs are not running after applying both the `ci-build-me` label and
 comment, you may be able to find and fix the error in the script. The script
 lives in `frontend/ci-build-me/src/index.js`, and instructions for deploying
 the new version are in the readme at `frontend/ci-build-me/README.md`. You
 should still follow normal procedure: submit a pull request and await approval
 for the changes before attempting to deploy the fixed script.
 
+## Integration test failures
+
+If your CI error is related to a timeout logged by one of the integration test
+runnners, this is a known issue and re-running the test in the Buildkite 
+interface will usually succeed.
+
 If an issue arises, please post an update in both `development` on the Mina
 Protocol discord and `engineering-internal` on the O(1) Labs discord with the
 details and links to the failures.
 
-Where you have a bugfix for failing CI, or are seeing a CI failure across
+## CI environment mismatch
+
+The CI runs its jobs in multiple Docker images. The images that it is using are
+specified in `buildkite/src/Constants/COntainerImages.dhall`: the CI uses all
+Debian images prefixed by `minaToolchainBuster`.
+
+Theses images are generated by the CI itself, in particular based on the content of the
+`dockerfiles` directory and the `opam.export` file (which describes versions of
+OCaml packages). If you PR modifies how the images are generated (for example by
+changing a package version in `opam.export`), then the CI will not automatically
+use these new images and will potentially fail.
+
+In this case, you should:
+1. Let the CI run once with the old images. Jobs may fail, but ensure that jobs
+   of the form `Docker: toolchain-*` succeed. This is the step that creates new
+   Docker images from your PR.
+2. Look for the newly generated images on
+   [Google Cloud](https://console.cloud.google.com/gcr/images/o1labs-192920/global/mina-toolchain)
+   where they have been automatically uploaded. Look for your branch name and
+   commit hash in the second column. You should find several images suffixed
+   with `-buster`, `-bullseye`, `-stretch` and `-focal` (Debian versions).
+3. For each such image, retrieve its full name and hash by hovering its link and
+   clicking the `Copy full image name` tooltip that appears (or retrieve it on
+   the image's page).
+4. Edit `buildkite/src/Constants/ContainerImages.dhall` and replace the name of
+   each `minaToolchain-*` image by the ones you retrieved.
+5. Commit the change, the CI should now be using the new images.
+
+## Contact
+
+If you have a bugfix for failing CI, or are seeing a CI failure across
 multiple PRs, the best people to contact are:
 * @bkase (bkase#2492 on discord) (Europe - misc.)
 * @lk86 (linuskrom#2287 on discord) (US West Coast)

README-dev.md

@@ -106,23 +106,20 @@ let g:syntastic_ocaml_checkers=['merlin']
 Emacs has a built-in autocomplete, via `M-x completion-at-point`, or simply `M-tab`. There are other
 Emacs autocompletion packages; see [Emacs from scratch](https://github.com/ocaml/merlin/wiki/emacs-from-scratch).
 
-## Using the makefile
+## Using the Makefile
 
 The makefile contains phony targets for all the common tasks that need to be done.
 It also knows how to use Docker automatically. 
 
 These are the most important `make` targets:
 
 - `build`: build everything
-- `test`: run the tests
 - `libp2p_helper`: build the libp2p helper
-- `web`: build the website, including the state explorer
+- `reformat`: automatically use `ocamlformat` to reformat the source files (use
+    it if the hook fails during a commit)
 
 We use the [dune](https://github.com/ocaml/dune/) buildsystem for our OCaml code.
 
-NOTE: all of the `test-*` targets (including `test-all`) won't run in the container.
-`test` wraps them in the container.
-
 ## Steps for adding a new dependency
 
 Rarely, you may edit one of our forked opam-pinned packages, or add a new system

README.md

@@ -28,7 +28,7 @@ Mina is the first cryptocurrency with a lightweight, constant-sized blockchain.
 
 ## Contributing
 
-Read the [Contributing Guide](https://docs.minaprotocol.com/en/developers/contributing) for information on how to make both technical and non-technical contributions.
+Read the [Contributing Guide](https://docs.minaprotocol.com/en/developers/contributing) for high-level information on how to make both technical and non-technical contributions. To contribute to this repository, make sure to read its [CONTRIBUTING](CONTRIBUTING.md) document.
 
 ## Developers
 

automation/scripts/gen-keys-ledger.sh

@@ -70,7 +70,7 @@ done
 echo "generating seeds' libp2p keys"
 mkdir "${KEYSDIR}/libp2p-keys"
 for i in $(seq 1 $SEEDS); do
-    mina advanced generate-libp2p-keypair --privkey-path "${KEYSDIR}/libp2p-keys/seed-${i}" 2>/dev/null
+    mina libp2p generate-keypair --privkey-path "${KEYSDIR}/libp2p-keys/seed-${i}" 2>/dev/null
 done
 
 

automation/scripts/generate-keys-and-ledger.sh

@@ -110,7 +110,7 @@ function generate_key_files {
     docker run \
       --mount type=bind,source=${output_dir},target=/keys \
       --entrypoint /bin/bash $MINA_DAEMON_IMAGE \
-      -c "MINA_LIBP2P_PASS='${privkey_pass}' mina advanced generate-libp2p-keypair -privkey-path /keys/${name_prefix}_libp2p_${k}"
+      -c "MINA_LIBP2P_PASS='${privkey_pass}' mina libp2p generate-keypair -privkey-path /keys/${name_prefix}_libp2p_${k}"
   done
 
   # ensure proper r+w permissions for access to keys external to container

automation/terraform/modules/google-cloud/coda-seed-node/main.tf

@@ -1,5 +1,5 @@
 locals {
-  container_command = format("mina daemon -log-level Info -config-directory /root/.mina-config -client-port 8301 -rest-port 8304 -external-port 10001 -metrics-port 10000 -discovery-keypair %s -seed %s -config-file /root/daemon.json | tee log.txt", var.discovery_keypair, var.seed_peers)
+  container_command = format("mina daemon -log-level Info -config-directory /root/.mina-config -client-port 8301 -rest-port 8304 -external-port 10001 -metrics-port 10000 -libp2p-keypair %s -seed %s -config-file /root/daemon.json | tee log.txt", var.discovery_keypair, var.seed_peers)
 }
 
 resource "google_compute_address" "external_ip" {

automation/terraform/modules/kubernetes/testnet/helm.tf

@@ -26,7 +26,7 @@ resource "helm_release" "seeds" {
   name       = "${var.testnet_name}-seeds"
   repository = var.use_local_charts ? "" : local.mina_helm_repo
   chart      = var.use_local_charts ? "../../../../helm/seed-node" : "seed-node"
-  version    = "1.0.6"
+  version    = "1.0.8"
   namespace  = kubernetes_namespace.testnet_namespace.metadata[0].name
   values = [
     yamlencode(local.seed_vars)
@@ -47,7 +47,7 @@ resource "helm_release" "block_producers" {
   name       = "${var.testnet_name}-block-producers"
   repository = var.use_local_charts ? "" : local.mina_helm_repo
   chart      = var.use_local_charts ? "../../../../helm/block-producer" : "block-producer"
-  version    = "1.0.5"
+  version    = "1.0.7"
   namespace  = kubernetes_namespace.testnet_namespace.metadata[0].name
   values = [
     yamlencode(local.block_producer_vars)
@@ -83,7 +83,7 @@ resource "helm_release" "snark_workers" {
   name       = "${var.testnet_name}-snark-set-${count.index + 1}"
   repository = var.use_local_charts ? "" : local.mina_helm_repo
   chart      = var.use_local_charts ? "../../../../helm/snark-worker" : "snark-worker"
-  version    = "1.0.3"
+  version    = "1.0.5"
   namespace  = kubernetes_namespace.testnet_namespace.metadata[0].name
   values = [
     yamlencode(local.snark_vars[count.index])
@@ -102,7 +102,7 @@ resource "helm_release" "archive_node" {
   name       = "archive-${count.index + 1}"
   repository = var.use_local_charts ? "" : local.mina_helm_repo
   chart      = var.use_local_charts ? "../../../../helm/archive-node" : "archive-node"
-  version    = "1.0.5"
+  version    = "1.0.7"
   namespace  = kubernetes_namespace.testnet_namespace.metadata[0].name
   values = [
     yamlencode(local.archive_vars[count.index])

buildkite/scripts/test-snarkyjs-bindings.sh

@@ -12,6 +12,11 @@ make snarkyjs
 echo "Run SnarkyJS bindings unit tests..."
 node src/lib/snarky_js_bindings/tests/run-tests.mjs
 
+echo "Run SnarkyJS unit tests..."
+cd src/lib/snarky_js_bindings/snarkyjs
+npm run test:unit
+cd ../../../..
+
 echo "Build MinaSigner..."
 make mina_signer
 
@@ -27,4 +32,3 @@ echo "Run SnarkyJS + MinaSigner tests..."
 node src/lib/snarky_js_bindings/test_module/simple-zkapp-mina-signer.js
 node src/lib/snarky_js_bindings/test_module/simple-zkapp-mock-apply.js
 node src/lib/snarky_js_bindings/test_module/inductive-proofs.js
-node src/lib/snarky_js_bindings/test_module/to-hash-input.js

buildkite/scripts/unit-test.sh

@@ -12,8 +12,10 @@ path=$2
 
 source ~/.profile
 
+export MINA_LIBP2P_PASS="naughty blue worm"
+
 echo "--- Make build"
-export LIBP2P_NIXLESS=1 PATH=/usr/lib/go/bin:$PATH GO=/usr/lib/go/bin/go 
+export LIBP2P_NIXLESS=1 PATH=/usr/lib/go/bin:$PATH GO=/usr/lib/go/bin/go
 time make build
 
 # Note: By attempting a re-run on failure here, we can avoid rebuilding and

buildkite/src/Constants/ContainerEnvVars.dhall

@@ -45,5 +45,6 @@
 , "BUILDKITE_AGENT_META_DATA_QUEUE"
 , "BUILDKITE_TIMEOUT"
 , "BUILDKITE_STEP_IDENTIFIER"
+, "NIGHTLY"
 ]
 

dockerfiles/auxiliary_entrypoints/01-run-demo.sh

@@ -15,29 +15,23 @@ if [[ -n ${RUN_DEMO} ]]; then
 
     mkdir /root/keys && chmod go-rwx /root/keys
     mkdir -p --mode=700 ${MINA_CONFIG_DIR}/wallets/store/
-    echo "$PK" >  ${MINA_CONFIG_DIR}/wallets/store/$PK.pub
-    echo '{"box_primitive":"xsalsa20poly1305","pw_primitive":"argon2i","nonce":"6pcvpWSLkMi393dT5VSLR6ft56AWKkCYRqJoYia","pwsalt":"ASoBkV3NsY7ZRuxztyPJdmJCiz3R","pwdiff":[134217728,6],"ciphertext":"Dmq1Qd8uNbZRT1NT7zVbn3eubpn9Myx9Je9ZQGTKDxUv4BoPNmZAGox18qVfbbEUSuhT4ZGDt"}' > ${MINA_CONFIG_DIR}/wallets/store/${PK}
+    echo "$PK" >${MINA_CONFIG_DIR}/wallets/store/$PK.pub
+    echo '{"box_primitive":"xsalsa20poly1305","pw_primitive":"argon2i","nonce":"6pcvpWSLkMi393dT5VSLR6ft56AWKkCYRqJoYia","pwsalt":"ASoBkV3NsY7ZRuxztyPJdmJCiz3R","pwdiff":[134217728,6],"ciphertext":"Dmq1Qd8uNbZRT1NT7zVbn3eubpn9Myx9Je9ZQGTKDxUv4BoPNmZAGox18qVfbbEUSuhT4ZGDt"}' >${MINA_CONFIG_DIR}/wallets/store/${PK}
     chmod go-rwx ${MINA_CONFIG_DIR}/wallets/store/${PK}
-    echo '{"genesis": {"genesis_state_timestamp": "${GENESIS_STATE_TIMESTAMP}"},"ledger":{"name":"mina-demo","accounts":[{"pk":"'${PK}'","balance":"66000","sk":null,"delegate":null}]}}' > ${CONFIG_TEMPLATE}
+    echo '{"genesis": {"genesis_state_timestamp": "${GENESIS_STATE_TIMESTAMP}"},"ledger":{"name":"mina-demo","accounts":[{"pk":"'${PK}'","balance":"66000","sk":null,"delegate":null}]}}' >${CONFIG_TEMPLATE}
 
     if [ -z "$GENESIS_STATE_TIMESTAMP" ]; then
         export GENESIS_STATE_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
     fi
     echo "Genesis State Timestamp for this run is: ${GENESIS_STATE_TIMESTAMP}"
 
     echo "Rewriting config file from template ${CONFIG_TEMPLATE} to ${MINA_CONFIG_FILE}"
-    envsubst < ${CONFIG_TEMPLATE} > ${MINA_CONFIG_FILE}
+    envsubst <${CONFIG_TEMPLATE} >${MINA_CONFIG_FILE}
 
     echo "Contents of config file ${MINA_CONFIG_FILE}:"
     cat "${MINA_CONFIG_FILE}"
 
-    MINA_TIME_OFFSET=${MINA_TIME_OFFSET:-0}
-    MINA_TIME_OFFSET=${MINA_TIME_OFFSET:-0}
-
-    MINA_PRIVKEY_PASS=${MINA_PRIVKEY_PASS:-""}
-    MINA_PRIVKEY_PASS=${MINA_PRIVKEY_PASS:-""}
-
-    exec mina daemon --generate-genesis-proof true --seed --demo-mode --proof-level none --config-dir ${MINA_CONFIG_DIR} --block-producer-pubkey ${PK} --run-snark-worker ${SNARK_PK} -insecure-rest-server $@
+    exec MINA_PRIVKEY_PASS=${MINA_PRIVKEY_PASS:-""} MINA_TIME_OFFSET=${MINA_TIME_OFFSET:-0} mina daemon --generate-genesis-proof true --seed --demo-mode --proof-level none --config-dir ${MINA_CONFIG_DIR} --block-producer-pubkey ${PK} --run-snark-worker ${SNARK_PK} -insecure-rest-server $@
 
     rc=$?
     echo "Exiting Mina demo." && exit $rc