feat: Use yarns topological build to get rid of explicit sequential s…

…teps, and let it solve. (#4868) * Moves all code generation steps into package.json scripts consistently called "generate". * Use yarns topological build to get rid of explicit sequential steps, and let it solve for building the dependency tree in parallel. * Makes yarn-project bootstrap default to a fast incremental build. This is basically `yarn generate && yarn tsc -b`, which will do codegen in parallel where it can, and only do an incremental build of the TS. If for whatever reason this fails, it falls back to full build. * Stop running eslint as part of codegen step (slow). * Stop prettier runs (.prettierignore) on JSON artifacts, slow and unnecessary. * Fixes bug in the remove_old_images script. * Parallelises the build of noir-contracts and noir-protocol-circuits. A yarn project bootstrap can now be as fast as ~10s if nothings changed, or ~50s if it needs to rebuild all TS. Running `yarn build` takes 1m40s, as not only does it clean, but it also does webpack stuff. The `generate` job by itself can take 6 seconds if nothing todo. Not perfect, but probably fast enough that with incremental builds on `noir-projects` plus triggering `generate` via `inotifywait` specifically in the right workspaces we can have a decent `build:dev` experience. A clean repo `./bootstrap.sh fast` took 1m40s on mainframe. A repeat run straight after took 34s. A clean repo `./bootstrap.sh full` took 8m40s on mainframe. A repeat run straight after took 2m22s.
AztecProtocol · Mar 2, 2024 · c909966 · c909966
1 parent 7feb7de
commit c909966
Show file tree

Hide file tree

Showing 21 changed files with 70 additions and 81 deletions.
diff --git a/build-system/scripts/remove_old_images b/build-system/scripts/remove_old_images
@@ -7,7 +7,7 @@ REPOSITORY=$1
 shift
 
 IMAGE_COMMIT_URI=$(calculate_image_uri $REPOSITORY)
-for IMAGE in $(docker images --format "{{.ID}}" aztecprotocol/$REPOSITORY --filter "before=$IMAGE_COMMIT_URI"); do
+for IMAGE in $(docker images --format "{{.ID}}" $ECR_URL/$REPOSITORY --filter "before=$IMAGE_COMMIT_URI"); do
   echo "Removing $IMAGE..."
   docker rmi --force $IMAGE
 done
diff --git a/noir-projects/bootstrap.sh b/noir-projects/bootstrap.sh
@@ -18,11 +18,11 @@ fi
 # Attempt to just pull artefacts from CI and exit on success.
 [ -n "${USE_CACHE:-}" ] && ./bootstrap_cache.sh && exit
 
-PROJECTS=(
-  noir-contracts
-  noir-protocol-circuits
-)
+g="\033[32m"  # Green
+b="\033[34m"  # Blue
+r="\033[0m"   # Reset
 
-for PROJECT in "${PROJECTS[@]}"; do
-  (cd "./$PROJECT" && ./bootstrap.sh "$@")
-done
+((cd "./noir-contracts" && ./bootstrap.sh) > >(awk -v g="$g" -v r="$r" '$0=g"contracts: "r $0')) &
+((cd "./noir-protocol-circuits" && ./bootstrap.sh) > >(awk -v b="$b" -v r="$r" '$0=b"protocol-circuits: "r $0')) &
+
+wait
diff --git a/yarn-project/.prettierignore b/yarn-project/.prettierignore
@@ -5,4 +5,4 @@ noir-contracts.js/**/*.json
 boxes/*/src/artifacts/*.json
 boxes/*/src/artifacts/*.ts
 boxes/*/src/contracts/target/*.json
-*.md
+*.md
diff --git a/yarn-project/Dockerfile b/yarn-project/Dockerfile
@@ -1,9 +1,5 @@
-# This base dockerfile adds all the remaining source files, performs artifact generation, and builds the project.
-# See yarn-project-base/Dockerfile for deeper insight into why things are how they are.
-# This should *only* build what is necessary to:
-#   - Run the tests.
-#   - Run the formatter checks.
-# Any subsequent build steps needed to support downstream containers should be done in those containers build files.
+# This base dockerfile adds all the remaining source files and builds the project.
+# See yarn-project-base/Dockerfile for why we have separate base Dockerfile.
 FROM aztecprotocol/l1-contracts as contracts
 FROM aztecprotocol/noir-projects as noir-projects
 FROM aztecprotocol/boxes-files as boxes-files
@@ -13,15 +9,5 @@ COPY --from=contracts /usr/src/l1-contracts /usr/src/l1-contracts
 COPY --from=noir-projects /usr/src/noir-projects /usr/src/noir-projects
 COPY --from=boxes-files /usr/src/boxes /usr/src/boxes
 COPY . .
-# Generate L1 contract TypeScript artifacts.
-RUN cd l1-artifacts && ./scripts/generate-artifacts.sh && rm -rf /usr/src/l1-contracts
-# This is actually our code generation tool. Needed to build contract typescript wrappers.
-RUN yarn workspace @aztec/noir-compiler build
-# Generates typescript wrappers.
-RUN yarn workspace @aztec/noir-contracts.js build
-# We need to build accounts as it needs to copy in account contracts from noir-contracts.
-RUN yarn workspace @aztec/accounts build:copy-contracts
-RUN yarn workspace @aztec/protocol-contracts build:copy-contracts
-RUN yarn workspace @aztec/noir-protocol-circuits-types build
-RUN yarn tsc -b
+RUN ./bootstrap.sh
 ENTRYPOINT ["yarn"]
diff --git a/yarn-project/accounts/.prettierignore b/yarn-project/accounts/.prettierignore
@@ -0,0 +1 @@
+src/artifacts/*.json
diff --git a/yarn-project/accounts/package.json b/yarn-project/accounts/package.json
@@ -23,8 +23,8 @@
     "tsconfig": "./tsconfig.json"
   },
   "scripts": {
-    "build": "yarn clean && yarn build:copy-contracts && tsc -b",
-    "build:copy-contracts": "./scripts/copy-contracts.sh",
+    "build": "yarn clean && yarn generate && tsc -b",
+    "generate": "./scripts/copy-contracts.sh",
     "build:dev": "tsc -b --watch",
     "build:ts": "tsc -b",
     "clean": "rm -rf ./dest .tsbuildinfo ./src/artifacts",

diff --git a/yarn-project/accounts/package.local.json b/yarn-project/accounts/package.local.json
@@ -1,7 +1,7 @@
 {
   "scripts": {
-    "build": "yarn clean && yarn build:copy-contracts && tsc -b",
-    "build:copy-contracts": "./scripts/copy-contracts.sh",
+    "build": "yarn clean && yarn generate && tsc -b",
+    "generate": "./scripts/copy-contracts.sh",
     "build:dev": "tsc -b --watch",
     "build:ts": "tsc -b",
     "clean": "rm -rf ./dest .tsbuildinfo ./src/artifacts"

diff --git a/yarn-project/accounts/scripts/copy-contracts.sh b/yarn-project/accounts/scripts/copy-contracts.sh
@@ -5,7 +5,5 @@ mkdir -p ./src/artifacts
 contracts=(schnorr_account_contract-SchnorrAccount ecdsa_account_contract-EcdsaAccount schnorr_single_key_account_contract-SchnorrSingleKeyAccount)
 
 for contract in "${contracts[@]}"; do
-  cp "../noir-contracts.js/artifacts/$contract.json" ./src/artifacts/${contract#*-}.json
-done
-
-yarn run -T prettier -w ./src/artifacts
+  cp "../../noir-projects/noir-contracts/target/$contract.json" ./src/artifacts/${contract#*-}.json
+done
diff --git a/yarn-project/bootstrap.sh b/yarn-project/bootstrap.sh
@@ -8,6 +8,12 @@ major=${node_version%%.*}
 rest=${node_version#*.}
 minor=${rest%%.*}
 
+YELLOW="\033[93m"
+BLUE="\033[34m"
+GREEN="\033[32m"
+BOLD="\033[1m"
+RESET="\033[0m"
+
 if ((major < 18 || (major == 18 && minor < 19))); then
   echo "Node.js version is less than 18.19. Exiting."
   exit 1
@@ -27,32 +33,17 @@ if [ -n "$CMD" ]; then
   fi
 fi
 
+# Fast build does not delete everything first.
+# It regenerates all generated code, then performs an incremental tsc build.
+echo -e "${BLUE}${BOLD}Attempting fast incremental build...${RESET}"
+echo
 yarn install --immutable
 
-echo -e "\033[1mGenerating constants files...\033[0m"
-# Required to run remake-constants.
-yarn workspace @aztec/foundation build
-# Run remake constants before building Aztec.nr contracts or l1 contracts as they depend on files created by it.
-yarn workspace @aztec/circuits.js remake-constants
-
-echo -e "\033[1mSetting up compiler and building contracts...\033[0m"
-# This is actually our code generation tool. Needed to build contract typescript wrappers.
-echo "Building noir compiler..."
-yarn workspace @aztec/noir-compiler build
-# Builds noir contracts (TODO: move this stage pre yarn-project). Generates typescript wrappers.
-echo "Building contracts from noir-contracts..."
-yarn workspace @aztec/noir-contracts.js build
-# Bundle compiled contracts into other packages
-echo "Copying account contracts..."
-yarn workspace @aztec/accounts build:copy-contracts
-echo "Copying protocol contracts..."
-yarn workspace @aztec/protocol-contracts build:copy-contracts
-# Build protocol circuits. TODO: move pre yarn-project.
-echo "Building circuits from noir-protocol-circuits..."
-yarn workspace @aztec/noir-protocol-circuits-types build
-
-echo -e "\033[1mBuilding all packages...\033[0m"
-yarn build
+if ! yarn build:fast; then
+  echo -e "${YELLOW}${BOLD}Incremental build failed for some reason, attempting full build...${RESET}"
+  echo
+  yarn build
+fi
 
 echo
-echo "Yarn project successfully built."
+echo -e "${GREEN}Yarn project successfully built!${RESET}"
diff --git a/yarn-project/circuits.js/src/scripts/constants.in.ts b/yarn-project/circuits.js/src/scripts/constants.in.ts
@@ -1,7 +1,6 @@
-import { fileURLToPath } from '@aztec/foundation/url';
-
 import * as fs from 'fs';
 import { dirname, join } from 'path';
+import { fileURLToPath } from 'url';
 
 const NOIR_CONSTANTS_FILE = '../../../../noir-projects/noir-protocol-circuits/crates/types/src/constants.nr';
 const TS_CONSTANTS_FILE = '../constants.gen.ts';
@@ -157,6 +156,7 @@ function main(): void {
 
   // Solidity
   const solidityTargetPath = join(__dirname, SOLIDITY_CONSTANTS_FILE);
+  fs.mkdirSync(dirname(solidityTargetPath), { recursive: true });
   generateSolidityConstants(parsedContent, solidityTargetPath);
 }
 

diff --git a/yarn-project/noir-compiler/package.json b/yarn-project/noir-compiler/package.json
@@ -19,6 +19,7 @@
   "scripts": {
     "build": "yarn clean && tsc -b",
     "build:dev": "tsc -b --watch",
+    "generate": "tsc -b",
     "clean": "rm -rf ./dest .tsbuildinfo",
     "formatting": "run -T prettier --check ./src && run -T eslint ./src",
     "formatting:fix": "run -T eslint --fix ./src && run -T prettier -w ./src",

diff --git a/yarn-project/noir-contracts.js/package.json b/yarn-project/noir-contracts.js/package.json
@@ -8,13 +8,13 @@
     "./*": "./dest/src/*.js"
   },
   "scripts": {
-    "build": "yarn clean && yarn build:contracts && yarn formatting:fix",
+    "build": "yarn clean && yarn generate",
     "build:dev": "tsc -b --watch",
     "clean": "rm -rf .tsbuildinfo ./artifacts ./codegenCache.json",
     "formatting": "run -T prettier --check ./src && run -T eslint ./src",
     "formatting:fix": "run -T eslint --fix ./src && run -T prettier -w ./src",
     "test": "NODE_NO_WARNINGS=1 node --experimental-vm-modules $(yarn bin jest) --passWithNoTests",
-    "build:contracts": "./scripts/generate-types.sh"
+    "generate": "./scripts/generate-types.sh && run -T prettier -w ./src"
   },
   "inherits": [
     "../package.common.json",
@@ -33,6 +33,7 @@
     "tslib": "^2.4.0"
   },
   "devDependencies": {
+    "@aztec/noir-compiler": "workspace:^",
     "@jest/globals": "^29.5.0",
     "@types/jest": "^29.5.0",
     "jest": "^29.5.0",

diff --git a/yarn-project/noir-contracts.js/package.local.json b/yarn-project/noir-contracts.js/package.local.json
@@ -1,7 +1,7 @@
 {
   "scripts": {
-    "build": "yarn clean && yarn build:contracts && yarn formatting:fix",
-    "build:contracts": "./scripts/generate-types.sh",
+    "build": "yarn clean && yarn generate",
+    "generate": "./scripts/generate-types.sh && run -T prettier -w ./src",
     "clean": "rm -rf .tsbuildinfo ./artifacts ./codegenCache.json"
   }
-}
+}
diff --git a/yarn-project/noir-contracts.js/tsconfig.json b/yarn-project/noir-contracts.js/tsconfig.json
@@ -8,8 +8,17 @@
   "references": [
     {
       "path": "../aztec.js"
+    },
+    {
+      "path": "../noir-compiler"
     }
   ],
-  "include": ["src", "artifacts", "artifacts/*.json"],
-  "exclude": ["dest"]
+  "include": [
+    "src",
+    "artifacts",
+    "artifacts/*.json"
+  ],
+  "exclude": [
+    "dest"
+  ]
 }
diff --git a/yarn-project/noir-protocol-circuits-types/package.json b/yarn-project/noir-protocol-circuits-types/package.json
@@ -7,12 +7,12 @@
     "./types": "./dest/types/index.js"
   },
   "scripts": {
-    "build": "yarn clean && yarn noir:types && tsc -b",
+    "build": "yarn clean && yarn generate && tsc -b",
     "clean": "rm -rf ./dest .tsbuildinfo src/types src/target",
     "formatting": "run -T prettier --check ./src && run -T eslint ./src",
     "formatting:fix": "NODE_OPTIONS='--max-old-space-size=8096' run -T eslint --fix ./src && run -T prettier -w ./src",
     "formatting:fix:types": "NODE_OPTIONS='--max-old-space-size=8096' run -T eslint --fix ./src/types && run -T prettier -w ./src/types",
-    "noir:types": "cp -r ../../noir-projects/noir-protocol-circuits/target ./src/target && node --loader ts-node/esm src/scripts/generate_ts_from_abi.ts && yarn formatting:fix:types",
+    "generate": "mkdir -p ./src/target && cp ../../noir-projects/noir-protocol-circuits/target/* ./src/target && node --no-warnings --loader ts-node/esm src/scripts/generate_ts_from_abi.ts && run -T prettier -w ./src/types",
     "test": "NODE_NO_WARNINGS=1 node --experimental-vm-modules $(yarn bin jest) --passWithNoTests"
   },
   "jest": {

diff --git a/yarn-project/package.json b/yarn-project/package.json
@@ -10,9 +10,11 @@
     "formatting:fix": "FORCE_COLOR=true yarn workspaces foreach -p -v run formatting:fix",
     "lint": "yarn eslint --cache --ignore-pattern l1-artifacts .",
     "format": "yarn prettier --cache -w .",
-    "test": "FORCE_COLOR=true yarn workspaces foreach --exclude @aztec/aztec3-packages --exclude @aztec/end-to-end --exclude private-token -p -j ${JOBS:-unlimited} -v run test",
-    "build": "yarn workspace @aztec/l1-artifacts build && tsc -b tsconfig.json",
-    "build:dev": "yarn workspace @aztec/l1-artifacts build && tsc -b tsconfig.json --watch",
+    "test": "FORCE_COLOR=true yarn workspaces foreach --exclude @aztec/aztec3-packages --exclude @aztec/end-to-end -p -j ${JOBS:-unlimited} -v run test",
+    "build": "FORCE_COLOR=true yarn workspaces foreach --parallel --topological-dev --verbose --exclude @aztec/aztec3-packages --exclude @aztec/docs run build",
+    "build:fast": "yarn generate && tsc -b",
+    "build:dev": "tsc -b tsconfig.json --watch",
+    "generate": "FORCE_COLOR=true yarn workspaces foreach --parallel --topological-dev --verbose run generate",
     "clean": "yarn workspaces foreach -p -v run clean"
   },
   "workspaces": [

diff --git a/yarn-project/protocol-contracts/.prettierignore b/yarn-project/protocol-contracts/.prettierignore
@@ -0,0 +1 @@
+src/artifacts/*.json
diff --git a/yarn-project/protocol-contracts/package.json b/yarn-project/protocol-contracts/package.json
@@ -18,8 +18,8 @@
     "tsconfig": "./tsconfig.json"
   },
   "scripts": {
-    "build": "yarn clean && yarn build:copy-contracts && tsc -b",
-    "build:copy-contracts": "./scripts/copy-contracts.sh",
+    "build": "yarn clean && yarn generate && tsc -b",
+    "generate": "./scripts/copy-contracts.sh",
     "build:dev": "tsc -b --watch",
     "build:ts": "tsc -b",
     "clean": "rm -rf ./dest .tsbuildinfo ./src/artifacts",

diff --git a/yarn-project/protocol-contracts/package.local.json b/yarn-project/protocol-contracts/package.local.json
@@ -1,7 +1,7 @@
 {
   "scripts": {
-    "build": "yarn clean && yarn build:copy-contracts && tsc -b",
-    "build:copy-contracts": "./scripts/copy-contracts.sh",
+    "build": "yarn clean && yarn generate && tsc -b",
+    "generate": "./scripts/copy-contracts.sh",
     "build:dev": "tsc -b --watch",
     "build:ts": "tsc -b",
     "clean": "rm -rf ./dest .tsbuildinfo ./src/artifacts"

diff --git a/yarn-project/protocol-contracts/scripts/copy-contracts.sh b/yarn-project/protocol-contracts/scripts/copy-contracts.sh
@@ -9,7 +9,5 @@ contracts=(
 )
 
 for contract in "${contracts[@]}"; do
-  cp "../noir-contracts.js/artifacts/$contract.json" ./src/artifacts/${contract#*-}.json
-done
-
-yarn run -T prettier -w ./src/artifacts
+  cp "../../noir-projects/noir-contracts/target/$contract.json" ./src/artifacts/${contract#*-}.json
+done
diff --git a/yarn-project/yarn.lock b/yarn-project/yarn.lock
@@ -633,6 +633,7 @@ __metadata:
   resolution: "@aztec/noir-contracts.js@workspace:noir-contracts.js"
   dependencies:
     "@aztec/aztec.js": "workspace:^"
+    "@aztec/noir-compiler": "workspace:^"
     "@jest/globals": ^29.5.0
     "@types/jest": ^29.5.0
     jest: ^29.5.0