diff --git a/docs/README.md b/docs/README.md index 625823e2be1..d07cc7f3981 100644 --- a/docs/README.md +++ b/docs/README.md @@ -56,10 +56,12 @@ Most changes are reflected live without having to restart the server. $ yarn build ``` -This command generates static content into the `build` directory and can be served using any static contents hosting service. +This command generates static content into the `build` directory and can be served using any static contents hosting service. When run on Netlify, it will also build the typescript projects needed for extracting type information via typedoc. -## #include_code macro +## Macros + +### `#include_code` You can embed code snippets into a `.md`/`.mdx` file from code which lives elsewhere in the repo. - In your markdown file: @@ -119,7 +121,26 @@ You can embed code snippets into a `.md`/`.mdx` file from code which lives elsew - `#include_code hello path/from/repo/root/to/file.ts typescript noTitle,noLineNumbers,noSourceLink` - Ironically, we can't show you a rendering of these examples, because this README.md file doesn't support the `#include_code` macro! +> See [here](./src/components/GithubCode/index.js) for another way to include code, although this approach is flakier, so the above `#include_code` macro is preferred. + +### `#include_aztec_version` + +This macros will be replaced inline with the current aztec packages tag, which is `aztec-packages-v0.7.10` at the time of these writing. This value is sourced from `.release-please-manifest.json` on the project root. -### Another way to include code. +Alternatively, you can also use the `AztecPackagesVersion()` js function, which you need to import explicitly: + +``` +import { AztecPackagesVersion } from "@site/src/components/Version"; +<>{AztecPackagesVersion()}</> +``` -See [here](./src/components/GithubCode/index.js), although this approach is flakier, so the above `#include_code` macro is preferred. \ No newline at end of file + ### `#include_noir_version` + +This macros will be replaced inline with the required nargo version, which is `0.11.1-aztec.0` at the time of these writing. This value is sourced from `yarn-project/noir-compiler/src/noir-version.json`. + +Alternatively, you can also use the `NoirVersion()` js function, which you need to import explicitly: + +``` +import { NoirVersion } from "@site/src/components/Version"; +<>{NoirVersion()}</> +``` diff --git a/docs/docs/dev_docs/contracts/syntax/main.md b/docs/docs/dev_docs/contracts/syntax/main.md index c89e97a1a2d..124e271e7ee 100644 --- a/docs/docs/dev_docs/contracts/syntax/main.md +++ b/docs/docs/dev_docs/contracts/syntax/main.md @@ -18,22 +18,21 @@ Aztec.nr contains abstractions which remove the need to understand the low-level To import Aztec.nr into your Aztec contract project, simply include it as a dependency. For example: -import { AztecPackagesVersion } from "@site/src/components/Version"; - -<CodeBlock language="toml">{`[package] +```toml +[package] name = "token_contract" authors = [""] compiler_version = "0.1" type = "contract" - + [dependencies] # Framework import -aztec = { git="https://github.com/AztecProtocol/aztec-packages/", tag="${AztecPackagesVersion()}", directory="yarn-project/aztec-nr/aztec" } - +aztec = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/aztec" } + # Utility dependencies -value_note = { git="https://github.com/AztecProtocol/aztec-packages/", tag="${AztecPackagesVersion()}", directory="yarn-project/aztec-nr/value-note"} -safe_math = { git="https://github.com/AztecProtocol/aztec-packages/", tag="${AztecPackagesVersion()}", directory="yarn-project/aztec-nr/safe-math"} -`}</CodeBlock> +value_note = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/value-note"} +safe_math = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/safe-math"} +``` :::info Note: currently the dependency name ***MUST*** be `aztec`. The framework expects this namespace to be available when compiling into contracts. This limitation may be removed in the future. diff --git a/docs/docs/dev_docs/dapps/tutorials/contract_deployment.md b/docs/docs/dev_docs/dapps/tutorials/contract_deployment.md index d677b403f77..36972b19fc7 100644 --- a/docs/docs/dev_docs/dapps/tutorials/contract_deployment.md +++ b/docs/docs/dev_docs/dapps/tutorials/contract_deployment.md @@ -17,13 +17,12 @@ nargo new --contract token Then, open the `contracts/token/Nargo.toml` configuration file, and add the `aztec.nr` and `value_note` libraries as dependencies: -import { AztecPackagesVersion } from "@site/src/components/Version"; - -<CodeBlock language="toml">{`[dependencies] -aztec = { git="https://github.com/AztecProtocol/aztec-packages/", tag="${AztecPackagesVersion()}", directory="yarn-project/aztec-nr/aztec" } -value_note = { git="https://github.com/AztecProtocol/aztec-packages/", tag="${AztecPackagesVersion()}", directory="yarn-project/aztec-nr/value-note"} -safe_math = { git="https://github.com/AztecProtocol/aztec-packages/", tag="${AztecPackagesVersion()}", directory="yarn-project/aztec-nr/safe-math"} -`}</CodeBlock> +```toml +[dependencies] +aztec = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/aztec" } +value_note = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/value-note"} +safe_math = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/safe-math"} +``` Last, copy-paste the code from the `Token` contract into `contracts/token/main.nr`: diff --git a/docs/docs/dev_docs/getting_started/noir_contracts.md b/docs/docs/dev_docs/getting_started/noir_contracts.md index 65adc75609f..d05a4cafbb4 100644 --- a/docs/docs/dev_docs/getting_started/noir_contracts.md +++ b/docs/docs/dev_docs/getting_started/noir_contracts.md @@ -59,17 +59,16 @@ Before writing the contracts, we must add the aztec.nr library. This adds smart 3. Add aztec.nr library as a dependency to your noir project. Open Nargo.toml that is in the `contracts/example_contract` folder, and add the dependency section as follows: -import { AztecPackagesVersion } from "@site/src/components/Version"; - -<CodeBlock language="toml">{`[package] +```toml +[package] name = "example_contract" authors = [""] compiler_version = "0.1" type = "contract" [dependencies] -aztec = { git="https://github.com/AztecProtocol/aztec-packages/", tag="master", directory="yarn-project/aztec-nr/aztec" } -`}</CodeBlock> +aztec = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/aztec" } +``` :::note You may need to update your dependencies depending on the contract that you are writing. For example, the token contract [imports more](../getting_started/token_contract_tutorial#project-setup). diff --git a/docs/docs/dev_docs/getting_started/token_contract_tutorial.md b/docs/docs/dev_docs/getting_started/token_contract_tutorial.md index d53e63c161d..64a61a5faae 100644 --- a/docs/docs/dev_docs/getting_started/token_contract_tutorial.md +++ b/docs/docs/dev_docs/getting_started/token_contract_tutorial.md @@ -80,19 +80,18 @@ Your project should look like this: Add the following dependencies to your Nargo.toml file, below the package information: -import { AztecPackagesVersion } from "@site/src/components/Version"; - -<CodeBlock language="toml">{`[package] +```toml +[package] name = "token_contract" authors = [""] compiler_version = "0.1" type = "contract" - + [dependencies] -aztec = { git="https://github.com/AztecProtocol/aztec-packages/", tag="${AztecPackagesVersion()}", directory="yarn-project/aztec-nr/aztec" } -value_note = { git="https://github.com/AztecProtocol/aztec-packages/", tag="${AztecPackagesVersion()}", directory="yarn-project/aztec-nr/value-note"} -safe_math = { git="https://github.com/AztecProtocol/aztec-packages/", tag="${AztecPackagesVersion()}", directory="yarn-project/aztec-nr/safe-math"} -`}</CodeBlock> +aztec = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/aztec" } +value_note = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/value-note"} +safe_math = { git="https://github.com/AztecProtocol/aztec-packages/", tag="#include_aztec_version", directory="yarn-project/aztec-nr/safe-math"} +``` ## Contract Interface diff --git a/docs/src/preprocess/include_code.js b/docs/src/preprocess/include_code.js new file mode 100644 index 00000000000..1e87b9aa351 --- /dev/null +++ b/docs/src/preprocess/include_code.js @@ -0,0 +1,324 @@ +const fs = require("fs"); +const path = require("path"); +const childProcess = require("child_process"); + +const getLineNumberFromIndex = (fileContent, index) => { + return fileContent.substring(0, index).split("\n").length; +}; + +/** + * Search for lines of the form + */ +function processHighlighting(codeSnippet, identifier) { + const lines = codeSnippet.split("\n"); + /** + * For an identifier = bar: + * + * Matches of the form: `highlight-next-line:foo:bar:baz` will be replaced with "highlight-next-line". + * Matches of the form: `highlight-next-line:foo:baz` will be replaced with "". + */ + const regex1 = /highlight-next-line:([a-zA-Z0-9-._:]+)/; + const replacement1 = "highlight-next-line"; + const regex2 = /highlight-start:([a-zA-Z0-9-._:]+)/; + const replacement2 = "highlight-start"; + const regex3 = /highlight-end:([a-zA-Z0-9-._:]+)/; + const replacement3 = "highlight-end"; + const regex4 = /this-will-error:([a-zA-Z0-9-._:]+)/; + const replacement4 = "this-will-error"; + + let result = ""; + let mutated = false; + + const processLine = (line, regex, replacement) => { + const match = line.match(regex); + if (match) { + mutated = true; + + const identifiers = match[1].split(":"); + if (identifiers.includes(identifier)) { + line = line.replace(match[0], replacement); + } else { + // Remove matched text completely + line = line.replace(match[0], ""); + } + } else { + // No match: it's an ordinary line of code. + } + return line.trim() == "//" || line.trim() == "#" ? "" : line; + }; + + for (let line of lines) { + mutated = false; + line = processLine(line, regex1, replacement1); + line = processLine(line, regex2, replacement2); + line = processLine(line, regex3, replacement3); + line = processLine(line, regex4, replacement4); + result += line === "" && mutated ? "" : line + "\n"; + } + + return result.trim(); +} + +let lastReleasedVersion; + +/** Returns the last released tag */ +function getLatestTag() { + if (!lastReleasedVersion) { + const manifest = path.resolve( + __dirname, + "../../../.release-please-manifest.json" + ); + lastReleasedVersion = JSON.parse(fs.readFileSync(manifest).toString())["."]; + } + return lastReleasedVersion + ? `aztec-packages-v${lastReleasedVersion}` + : undefined; +} + +/** Returns whether to use the latest release or the current version of stuff. */ +function useLastRelease() { + return process.env.NETLIFY || process.env.INCLUDE_RELEASED_CODE; +} + +/** + * Returns the contents of a file. If the build is running for publishing, it will load the contents + * of the file in the last released version. + */ +function readFile(filePath, tag) { + if (tag && tag !== "master") { + try { + const tag = getLatestTag(); + const root = path.resolve(__dirname, "../../../"); + const relPath = path.relative(root, filePath); + return childProcess.execSync(`git show ${tag}:${relPath}`).toString(); + } catch (err) { + console.error( + `Error reading file ${relPath} from latest version. Falling back to current content.` + ); + } + } + return fs.readFileSync(filePath, "utf-8"); +} + +/** Extracts a code snippet, trying with the last release if applicable, and falling back to current content. */ +function extractCodeSnippet(filePath, identifier) { + if (useLastRelease()) { + try { + return doExtractCodeSnippet(filePath, identifier, false); + } catch (err) { + console.error( + `Error extracting code snippet ${identifier} for ${filePath}: ${err}. Falling back to current content.` + ); + } + } + + return doExtractCodeSnippet(filePath, identifier, true); +} + +/** + * Parse a code file, looking for identifiers of the form: + * `docs:start:${identifier}` and `docs:end:{identifier}`. + * Extract that section of code. + * + * It's complicated if code snippet identifiers overlap (i.e. the 'start' of one code snippet is in the + * middle of another code snippet). The extra logic in this function searches for all identifiers, and + * removes any which fall within the bounds of the code snippet for this particular `identifier` param. + * @returns the code snippet, and start and end line numbers which can later be used for creating a link to github source code. + */ +function doExtractCodeSnippet(filePath, identifier, useCurrent) { + const tag = useCurrent ? "master" : getLatestTag(); + let fileContent = readFile(filePath, tag); + let lineRemovalCount = 0; + let linesToRemove = []; + + const startRegex = /(?:\/\/|#)\s+docs:start:([a-zA-Z0-9-._:]+)/g; // `g` will iterate through the regex.exec loop + const endRegex = /(?:\/\/|#)\s+docs:end:([a-zA-Z0-9-._:]+)/g; + + /** + * Search for one of the regex statements in the code file. If it's found, return the line as a string and the line number. + */ + const lookForMatch = (regex) => { + let match; + let matchFound = false; + let matchedLineNum = null; + let actualMatch = null; + let lines = fileContent.split("\n"); + while ((match = regex.exec(fileContent))) { + if (match !== null) { + const identifiers = match[1].split(":"); + let tempMatch = identifiers.includes(identifier) ? match : null; + + if (tempMatch === null) { + // If it's not a match, we'll make a note that we should remove the matched text, because it's from some other identifier and should not appear in the snippet for this identifier. + for (let i = 0; i < lines.length; i++) { + let line = lines[i]; + if (line.trim() == match[0].trim()) { + linesToRemove.push(i + 1); // lines are indexed from 1 + ++lineRemovalCount; + } + } + } else { + if (matchFound === true) { + throw new Error( + `Duplicate for regex ${regex} and identifier ${identifier}` + ); + } + matchFound = true; + matchedLineNum = getLineNumberFromIndex(fileContent, tempMatch.index); + actualMatch = tempMatch; + } + } + } + + return [actualMatch, matchedLineNum]; + }; + + let [startMatch, startLineNum] = lookForMatch(startRegex); + let [endMatch, endLineNum] = lookForMatch(endRegex); + + // Double-check that the extracted line actually contains the required start and end identifier. + if (startMatch !== null) { + const startIdentifiers = startMatch[1].split(":"); + startMatch = startIdentifiers.includes(identifier) ? startMatch : null; + } + if (endMatch !== null) { + const endIdentifiers = endMatch[1].split(":"); + endMatch = endIdentifiers.includes(identifier) ? endMatch : null; + } + + if (startMatch === null || endMatch === null) { + if (startMatch === null && endMatch === null) { + throw new Error( + `Identifier "${identifier}" not found in file "${filePath}"` + ); + } else if (startMatch === null) { + throw new Error( + `Start line "docs:start:${identifier}" not found in file "${filePath}"` + ); + } else { + throw new Error( + `End line "docs:end:${identifier}" not found in file "${filePath}"` + ); + } + } + + let lines = fileContent.split("\n"); + + // We only want to remove lines which actually fall within the bounds of our code snippet, so narrow down the list of lines that we actually want to remove. + linesToRemove = linesToRemove.filter((lineNum) => { + const removal_in_bounds = lineNum >= startLineNum && lineNum <= endLineNum; + return removal_in_bounds; + }); + + // Remove lines which contain `docs:` comments for unrelated identifiers: + lines = lines.filter((l, i) => { + return !linesToRemove.includes(i + 1); // lines are indexed from 1 + }); + + // Remove lines from the snippet which fall outside the `docs:start` and `docs:end` values. + lines = lines.filter((l, i) => { + return i + 1 > startLineNum && i + 1 < endLineNum - linesToRemove.length; // lines are indexed from 1 + }); + + // We have our code snippet! + let codeSnippet = lines.join("\n"); + + // The code snippet might contain some docusaurus highlighting comments for other identifiers. We should remove those. + codeSnippet = processHighlighting(codeSnippet, identifier); + + return [codeSnippet, startLineNum, endLineNum, tag]; +} + +/** + * Explaining this regex: + * + * E.g. `#include_code snippet_identifier /circuits/my_code.cpp cpp` + * + * #include_code\s+(\S+)\s+(\S+)\s+(\S+) + * - This is the main regex to match the above format. + * - \s+: one or more whitespace characters (space or tab) after `include_code` command. + * - (\S+): one or more non-whitespaced characters. Captures this as the first argument, which is a human-readable identifier for the code block. + * - etc. + * + * Lookaheads are needed to allow us to ignore commented-out lines: + * + * ^(?!<!--.*) + * - ^: Asserts the beginning of the line. + * - (?!<!--.*): Negative lookahead assertion to ensure the line does not start with markdown comment syntax `<!--`. + * + * (?=.*STUFF) + * - Positive lookahead assertion to ensure the line contains the command (STUFF) we want to match. + * + * .*$ + * - .*: Matches any characters (except newline) in the line. + * - $: Asserts the end of the line. + * + * `/gm` + * - match globally (g) across the entire input text and consider multiple lines (m) when matching. This is necessary to handle multiple include tags throughout the markdown content. + */ +const regex = + /^(?!<!--.*)(?=.*#include_code\s+(\S+)\s+(\S+)\s+(\S+)(?:[ ]+(\S+))?).*$/gm; + +async function preprocessIncludeCode(markdownContent, filePath, rootDir) { + // Process each include tag in the current markdown file + let updatedContent = markdownContent; + let matchesFound = false; + let match; + while ((match = regex.exec(markdownContent))) { + matchesFound = true; + const fullMatch = match[0]; + const identifier = match[1]; + let codeFilePath = match[2]; + const language = match[3]; + const opts = match[4] || ""; + + if (codeFilePath.slice(0) != "/") { + // Absolute path to the code file from the root of the Docusaurus project + // Note: without prefixing with `/`, the later call to `path.resolve()` gives an incorrect path (absolute instead of relative) + codeFilePath = `/${codeFilePath}`; + } + + const noTitle = opts.includes("noTitle"); + const noLineNumbers = opts.includes("noLineNumbers"); + const noSourceLink = opts.includes("noSourceLink"); + + try { + const absCodeFilePath = path.join(rootDir, codeFilePath); + + // Extract the code snippet between the specified comments + const extracted = extractCodeSnippet(absCodeFilePath, identifier); + const [codeSnippet, startLine, endLine, tag] = extracted; + + const relativeCodeFilePath = path.resolve(rootDir, codeFilePath); + + let urlText = `${relativeCodeFilePath}#L${startLine}-L${endLine}`; + if (tag && tag !== "master") urlText += ` (${tag})`; + const url = `https://github.com/AztecProtocol/aztec-packages/blob/${tag}/${relativeCodeFilePath}#L${startLine}-L${endLine}`; + + const title = noTitle ? "" : `title="${identifier}"`; + const lineNumbers = noLineNumbers ? "" : "showLineNumbers"; + const source = noSourceLink + ? "" + : `\n> [<sup><sub>Source code: ${urlText}</sub></sup>](${url})`; + const replacement = + language === "raw" + ? codeSnippet + : `\`\`\`${language} ${title} ${lineNumbers} \n${codeSnippet}\n\`\`\`${source}\n`; + + // Replace the include tag with the code snippet + updatedContent = updatedContent.replace(fullMatch, replacement); + } catch (error) { + const lineNum = getLineNumberFromIndex(markdownContent, match.index); + // We were warning here, but code snippets were being broken. So making this throw an error instead: + throw new Error( + `Error processing "${filePath}:${lineNum}": ${error.message}.` + ); + } + } + + return { content: updatedContent, isUpdated: matchesFound }; +} + +module.exports = { + preprocessIncludeCode, +}; diff --git a/docs/src/preprocess/include_version.js b/docs/src/preprocess/include_version.js new file mode 100644 index 00000000000..296b06158c8 --- /dev/null +++ b/docs/src/preprocess/include_version.js @@ -0,0 +1,52 @@ +const path = require("path"); +const fs = require("fs"); + +const VERSION_IDENTIFIERS = ["noir", "aztec"]; + +let versions; +async function getVersions() { + if (!versions) { + try { + const noirVersionPath = path.resolve( + __dirname, + "../../../yarn-project/noir-compiler/src/noir-version.json" + ); + const noirVersion = JSON.parse( + fs.readFileSync(noirVersionPath).toString() + ).tag; + const aztecVersionPath = path.resolve( + __dirname, + "../../../.release-please-manifest.json" + ); + const aztecVersion = JSON.parse( + fs.readFileSync(aztecVersionPath).toString() + )["."]; + versions = { + noir: noirVersion, + aztec: `aztec-packages-v${aztecVersion}`, + }; + } catch (err) { + throw new Error( + `Error loading versions in docusaurus preprocess step.\n${err}` + ); + } + } + return versions; +} + +async function preprocessIncludeVersion(markdownContent) { + const originalContent = markdownContent; + for (const identifier of VERSION_IDENTIFIERS) { + const version = (await getVersions())[identifier]; + markdownContent = markdownContent.replaceAll( + `#include_${identifier}_version`, + version + ); + } + return { + content: markdownContent, + isUpdated: originalContent !== markdownContent, + }; +} + +module.exports = { preprocessIncludeVersion }; diff --git a/docs/src/preprocess/index.js b/docs/src/preprocess/index.js index c7199e29006..129321d4ecd 100644 --- a/docs/src/preprocess/index.js +++ b/docs/src/preprocess/index.js @@ -2,315 +2,42 @@ const fs = require("fs"); const path = require("path"); const childProcess = require("child_process"); -const getLineNumberFromIndex = (fileContent, index) => { - return fileContent.substring(0, index).split("\n").length; -}; - -/** - * Search for lines of the form - */ -function processHighlighting(codeSnippet, identifier) { - const lines = codeSnippet.split("\n"); - /** - * For an identifier = bar: - * - * Matches of the form: `highlight-next-line:foo:bar:baz` will be replaced with "highlight-next-line". - * Matches of the form: `highlight-next-line:foo:baz` will be replaced with "". - */ - const regex1 = /highlight-next-line:([a-zA-Z0-9-._:]+)/; - const replacement1 = "highlight-next-line"; - const regex2 = /highlight-start:([a-zA-Z0-9-._:]+)/; - const replacement2 = "highlight-start"; - const regex3 = /highlight-end:([a-zA-Z0-9-._:]+)/; - const replacement3 = "highlight-end"; - const regex4 = /this-will-error:([a-zA-Z0-9-._:]+)/; - const replacement4 = "this-will-error"; - - let result = ""; - let mutated = false; - - const processLine = (line, regex, replacement) => { - const match = line.match(regex); - if (match) { - mutated = true; - - const identifiers = match[1].split(":"); - if (identifiers.includes(identifier)) { - line = line.replace(match[0], replacement); - } else { - // Remove matched text completely - line = line.replace(match[0], ""); - } - } else { - // No match: it's an ordinary line of code. - } - return line.trim() == "//" || line.trim() == "#" ? "" : line; - }; - - for (let line of lines) { - mutated = false; - line = processLine(line, regex1, replacement1); - line = processLine(line, regex2, replacement2); - line = processLine(line, regex3, replacement3); - line = processLine(line, regex4, replacement4); - result += line === "" && mutated ? "" : line + "\n"; - } - - return result.trim(); -} - -let lastReleasedVersion; - -/** Returns the last released tag */ -function getLatestTag() { - if (!lastReleasedVersion) { - const manifest = path.resolve( - __dirname, - "../../../.release-please-manifest.json" - ); - lastReleasedVersion = JSON.parse(fs.readFileSync(manifest).toString())["."]; - } - return lastReleasedVersion - ? `aztec-packages-v${lastReleasedVersion}` - : undefined; -} - -/** Returns whether to use the latest release or the current version of stuff. */ -function useLastRelease() { - return process.env.NETLIFY || process.env.INCLUDE_RELEASED_CODE; -} - -/** - * Returns the contents of a file. If the build is running for publishing, it will load the contents - * of the file in the last released version. - */ -function readFile(filePath, tag) { - if (tag && tag !== "master") { - try { - const tag = getLatestTag(); - const root = path.resolve(__dirname, "../../../"); - const relPath = path.relative(root, filePath); - return childProcess.execSync(`git show ${tag}:${relPath}`).toString(); - } catch (err) { - console.error( - `Error reading file ${relPath} from latest version. Falling back to current content.` - ); - } - } - return fs.readFileSync(filePath, "utf-8"); -} - -/** Extracts a code snippet, trying with the last release if applicable, and falling back to current content. */ -function extractCodeSnippet(filePath, identifier) { - if (useLastRelease()) { - try { - return doExtractCodeSnippet(filePath, identifier, false); - } catch (err) { - console.error( - `Error extracting code snippet ${identifier} for ${filePath}: ${err}. Falling back to current content.` - ); - } - } - - return doExtractCodeSnippet(filePath, identifier, true); -} - -/** - * Parse a code file, looking for identifiers of the form: - * `docs:start:${identifier}` and `docs:end:{identifier}`. - * Extract that section of code. - * - * It's complicated if code snippet identifiers overlap (i.e. the 'start' of one code snippet is in the - * middle of another code snippet). The extra logic in this function searches for all identifiers, and - * removes any which fall within the bounds of the code snippet for this particular `identifier` param. - * @returns the code snippet, and start and end line numbers which can later be used for creating a link to github source code. - */ -function doExtractCodeSnippet(filePath, identifier, useCurrent) { - const tag = useCurrent ? "master" : getLatestTag(); - let fileContent = readFile(filePath, tag); - let lineRemovalCount = 0; - let linesToRemove = []; - - const startRegex = /(?:\/\/|#)\s+docs:start:([a-zA-Z0-9-._:]+)/g; // `g` will iterate through the regex.exec loop - const endRegex = /(?:\/\/|#)\s+docs:end:([a-zA-Z0-9-._:]+)/g; - - /** - * Search for one of the regex statements in the code file. If it's found, return the line as a string and the line number. - */ - const lookForMatch = (regex) => { - let match; - let matchFound = false; - let matchedLineNum = null; - let actualMatch = null; - let lines = fileContent.split("\n"); - while ((match = regex.exec(fileContent))) { - if (match !== null) { - const identifiers = match[1].split(":"); - let tempMatch = identifiers.includes(identifier) ? match : null; - - if (tempMatch === null) { - // If it's not a match, we'll make a note that we should remove the matched text, because it's from some other identifier and should not appear in the snippet for this identifier. - for (let i = 0; i < lines.length; i++) { - let line = lines[i]; - if (line.trim() == match[0].trim()) { - linesToRemove.push(i + 1); // lines are indexed from 1 - ++lineRemovalCount; - } - } - } else { - if (matchFound === true) { - throw new Error( - `Duplicate for regex ${regex} and identifier ${identifier}` - ); - } - matchFound = true; - matchedLineNum = getLineNumberFromIndex(fileContent, tempMatch.index); - actualMatch = tempMatch; - } - } - } - - return [actualMatch, matchedLineNum]; - }; - - let [startMatch, startLineNum] = lookForMatch(startRegex); - let [endMatch, endLineNum] = lookForMatch(endRegex); - - // Double-check that the extracted line actually contains the required start and end identifier. - if (startMatch !== null) { - const startIdentifiers = startMatch[1].split(":"); - startMatch = startIdentifiers.includes(identifier) ? startMatch : null; - } - if (endMatch !== null) { - const endIdentifiers = endMatch[1].split(":"); - endMatch = endIdentifiers.includes(identifier) ? endMatch : null; - } - - if (startMatch === null || endMatch === null) { - if (startMatch === null && endMatch === null) { - throw new Error( - `Identifier "${identifier}" not found in file "${filePath}"` - ); - } else if (startMatch === null) { - throw new Error( - `Start line "docs:start:${identifier}" not found in file "${filePath}"` - ); - } else { - throw new Error( - `End line "docs:end:${identifier}" not found in file "${filePath}"` - ); - } - } - - let lines = fileContent.split("\n"); - - // We only want to remove lines which actually fall within the bounds of our code snippet, so narrow down the list of lines that we actually want to remove. - linesToRemove = linesToRemove.filter((lineNum) => { - const removal_in_bounds = lineNum >= startLineNum && lineNum <= endLineNum; - return removal_in_bounds; - }); - - // Remove lines which contain `docs:` comments for unrelated identifiers: - lines = lines.filter((l, i) => { - return !linesToRemove.includes(i + 1); // lines are indexed from 1 - }); - - // Remove lines from the snippet which fall outside the `docs:start` and `docs:end` values. - lines = lines.filter((l, i) => { - return i + 1 > startLineNum && i + 1 < endLineNum - linesToRemove.length; // lines are indexed from 1 - }); - - // We have our code snippet! - let codeSnippet = lines.join("\n"); - - // The code snippet might contain some docusaurus highlighting comments for other identifiers. We should remove those. - codeSnippet = processHighlighting(codeSnippet, identifier); - - return [codeSnippet, startLineNum, endLineNum, tag]; -} +const { preprocessIncludeCode } = require("./include_code"); +const { preprocessIncludeVersion } = require("./include_version"); async function processMarkdownFilesInDir(rootDir, docsDir, regex) { const files = fs.readdirSync(docsDir); - const contentPromises = []; + const contentUpdates = []; for (const file of files) { - const filePath = path.join(docsDir, file); - const stat = fs.statSync(filePath); + const filepath = path.join(docsDir, file); + const stat = fs.statSync(filepath); if (stat.isDirectory()) { - contentPromises.push(processMarkdownFilesInDir(rootDir, filePath, regex)); + contentUpdates.push(processMarkdownFilesInDir(rootDir, filepath, regex)); } else if ( stat.isFile() && (file.endsWith(".md") || file.endsWith(".mdx")) ) { - const markdownContent = fs.readFileSync(filePath, "utf-8"); + const markdownContent = fs.readFileSync(filepath, "utf-8"); - // Process each include tag in the current markdown file let updatedContent = markdownContent; - let matchesFound = false; - let match; - while ((match = regex.exec(markdownContent))) { - matchesFound = true; - const fullMatch = match[0]; - const identifier = match[1]; - let codeFilePath = match[2]; - const language = match[3]; - const opts = match[4] || ""; - - if (codeFilePath.slice(0) != "/") { - // Absolute path to the code file from the root of the Docusaurus project - // Note: without prefixing with `/`, the later call to `path.resolve()` gives an incorrect path (absolute instead of relative) - codeFilePath = `/${codeFilePath}`; - } - - const noTitle = opts.includes("noTitle"); - const noLineNumbers = opts.includes("noLineNumbers"); - const noSourceLink = opts.includes("noSourceLink"); - - try { - const absCodeFilePath = path.join(rootDir, codeFilePath); - - // Extract the code snippet between the specified comments - const extracted = extractCodeSnippet(absCodeFilePath, identifier); - const [codeSnippet, startLine, endLine, tag] = extracted; - - const relativeCodeFilePath = path.resolve(rootDir, codeFilePath); - - let urlText = `${relativeCodeFilePath}#L${startLine}-L${endLine}`; - if (tag && tag !== "master") urlText += ` (${tag})`; - const url = `https://github.com/AztecProtocol/aztec-packages/blob/${tag}/${relativeCodeFilePath}#L${startLine}-L${endLine}`; - - const title = noTitle ? "" : `title="${identifier}"`; - const lineNumbers = noLineNumbers ? "" : "showLineNumbers"; - const source = noSourceLink - ? "" - : `\n> [<sup><sub>Source code: ${urlText}</sub></sup>](${url})`; - const replacement = (language === "raw") - ? codeSnippet - : `\`\`\`${language} ${title} ${lineNumbers} \n${codeSnippet}\n\`\`\`${source}\n`; - - // Replace the include tag with the code snippet - updatedContent = updatedContent.replace(fullMatch, replacement); - } catch (error) { - const lineNum = getLineNumberFromIndex(markdownContent, match.index); - // We were warning here, but code snippets were being broken. So making this throw an error instead: - throw new Error( - `Error processing "${filePath}:${lineNum}": ${error.message}.` - ); - } + let isUpdated = false; + for (preprocess of [preprocessIncludeCode, preprocessIncludeVersion]) { + const result = await preprocess(updatedContent, filepath, rootDir); + updatedContent = result.content; + isUpdated = isUpdated || result.isUpdated; } - contentPromises.push({ - filepath: filePath, + contentUpdates.push({ content: updatedContent, - isUpdated: matchesFound, + filepath, + isUpdated, }); } } - const contentArray = await Promise.all(contentPromises); - - return contentArray; + return Promise.all(contentUpdates); } async function writeProcessedFiles(docsDir, destDir, cachedDestDir, content) { @@ -386,7 +113,7 @@ async function writeProcessedFiles(docsDir, destDir, cachedDestDir, content) { ); } - return writePromises; + return Promise.all(writePromises); } async function run() { @@ -395,37 +122,7 @@ async function run() { const destDir = path.join(rootDir, "docs", "processed-docs"); const cachedDestDir = path.join(rootDir, "docs", "processed-docs-cache"); - /** - * Explaining this regex: - * - * E.g. `#include_code snippet_identifier /circuits/my_code.cpp cpp` - * - * #include_code\s+(\S+)\s+(\S+)\s+(\S+) - * - This is the main regex to match the above format. - * - \s+: one or more whitespace characters (space or tab) after `include_code` command. - * - (\S+): one or more non-whitespaced characters. Captures this as the first argument, which is a human-readable identifier for the code block. - * - etc. - * - * Lookaheads are needed to allow us to ignore commented-out lines: - * - * ^(?!<!--.*) - * - ^: Asserts the beginning of the line. - * - (?!<!--.*): Negative lookahead assertion to ensure the line does not start with markdown comment syntax `<!--`. - * - * (?=.*STUFF) - * - Positive lookahead assertion to ensure the line contains the command (STUFF) we want to match. - * - * .*$ - * - .*: Matches any characters (except newline) in the line. - * - $: Asserts the end of the line. - * - * `/gm` - * - match globally (g) across the entire input text and consider multiple lines (m) when matching. This is necessary to handle multiple include tags throughout the markdown content. - */ - const regex = - /^(?!<!--.*)(?=.*#include_code\s+(\S+)\s+(\S+)\s+(\S+)(?:[ ]+(\S+))?).*$/gm; - - const content = await processMarkdownFilesInDir(rootDir, docsDir, regex); + const content = await processMarkdownFilesInDir(rootDir, docsDir); await writeProcessedFiles(docsDir, destDir, cachedDestDir, content);