[feat] Query binary (#46)

* Add MVP bin * Add MVP bin * Update cache command * Refactor cmd line items * Mock subcommand + logical interface updates * Update interface for reading db * Change interface to read * Change interface to read * Add write command * Add output command for write * Feat/m1 execution air (#66) * feat: sorted_limbs chip checking each limb less than limb_bits bits * wip * feat: completed sorted_limbs chip with tests * wip * feat: SortedLimbsChip with LessThan subchip * feat: less_than subchip refactored * feat: rename SortedLimbsChip to AssertSortedChip and write LessThanChip tests * chore: change name of assert sorted chip * chore: fix names in tests for AssertSortedChip * chore: address comments * chore: cleanup * chore: change MAX from generic to instance field for LessThanChip and AssertSortedChip * wip: added extra bits to middle chip trace * wip * adding connection to IsEqualVec chip * feat: IsLessThanChip to compare two numbers * feat: IsLessThanTuple subchip for different limb_bits * test: added tests for partially and non-allocated pages * test: added negative tests * added constraints * adding comments * feat: IsLessThanTupleChip subchip in AssertSortedChip * renaming * removing TODO comments * fixing clippy * chore: renaming to idx and data * chore: address comments first pass * chore: moving page_controller inside page_read * chore: refactor AssertSorted, IsEqual, IsLessThan, and IsLessThanTuple chips * chore: address comments * chore: eliminate high dim poly from IsLessThanTupleChip * chore: fix tests * chore: address comments for AssertSortedChip * chore: cleanup AssertSorted * chore: cleanup * chore: include roundtrip flatten and from_slice tests * feat: flatten and from_slice for IO and Aux columns * wip: final page chip * remove txt file * feat: final_page_chip * feat: integrating all chips for page_read_write * fixing lib * using field bits() * adding extra communication between checker and final chip * muting clippy * renaming to AIR * adding comments * wip * wip * feat: execution air finished --------- Co-authored-by: bfan <[email protected]> Co-authored-by: Osama Alkhodairy <[email protected]> Co-authored-by: Jonathan Wang <[email protected]> * Add print flag to afi command, update readmes * Update warnings * wip keygen * wip: start prove * Predicate * wip: prove and verify * change branches * Add to_page for Table * Use btreemap * Update functions * Update page to/from * Update tests * Add predicate binary * wip: proof not serializable * chore: patch stuff * add serde bounds * feat: zk commands (not debugged yet) * update mockdb * wip: mold into new format * chore: remove extraneous stuff * [WIP: query-binary] Merge r/w changes (#69) * Add is_less_than_bits chip (#60) * Add is_less_than_bits chip * Address zlangely's comments * Remove chip and use from_bool * Use row_mut * Add comments explaining comparison_check * Add alt version of is_less_than_bits chip * Replace is_less_than_bits with alt version, address comments * Fix lint issues * Feat/is less than bits tuple chip (#63) * Add is_less_than_tuple_bits chip * Incorporate is_less_than_bits feedback into is_less_than_tuple_bits * Add is_less_than_tuple_bits chip * Use match and cmp instead of else if * Adjust for new is_less_than_bits chip * chore: remove range_max field from IsLessThanAir and IsLessThanTupleAir (#65) * chore: derive serialize for proof (#67) * feat: page read write checker (#45) * feat: sorted_limbs chip checking each limb less than limb_bits bits * wip * feat: completed sorted_limbs chip with tests * wip * feat: SortedLimbsChip with LessThan subchip * feat: less_than subchip refactored * feat: rename SortedLimbsChip to AssertSortedChip and write LessThanChip tests * chore: change name of assert sorted chip * chore: fix names in tests for AssertSortedChip * chore: address comments * chore: cleanup * chore: change MAX from generic to instance field for LessThanChip and AssertSortedChip * wip: added extra bits to middle chip trace * wip * adding connection to IsEqualVec chip * feat: IsLessThanChip to compare two numbers * feat: IsLessThanTuple subchip for different limb_bits * test: added tests for partially and non-allocated pages * test: added negative tests * added constraints * adding comments * feat: IsLessThanTupleChip subchip in AssertSortedChip * renaming * removing TODO comments * fixing clippy * chore: renaming to idx and data * chore: address comments first pass * chore: moving page_controller inside page_read * chore: refactor AssertSorted, IsEqual, IsLessThan, and IsLessThanTuple chips * chore: address comments * chore: eliminate high dim poly from IsLessThanTupleChip * chore: fix tests * chore: address comments for AssertSortedChip * chore: cleanup AssertSorted * chore: cleanup * chore: include roundtrip flatten and from_slice tests * feat: flatten and from_slice for IO and Aux columns * wip: final page chip * remove txt file * feat: final_page_chip * feat: integrating all chips for page_read_write * fixing lib * using field bits() * adding extra communication between checker and final chip * muting clippy * renaming to AIR * adding comments * comment fix * optimization to one less bus * comments * moving old page_controller inside page_read * updateing comment * feat: more general FinalPageAir * making IsEqualVecAuxCols clonable * enforcing unallocated rows to be zero * aligning with new less than chip * using Air::eval instead of SubAir::eval * fix comment * removed arguments from load_page for readability * addressing comments and adding common/ directory with Page struct --------- Co-authored-by: bfan <[email protected]> Co-authored-by: Jonathan Wang <[email protected]> * Update to/from page and table tests * Update tests * chore: satisfy clippy (#68) * Fix execution_air test * Update --------- Co-authored-by: TlatoaniHJ <[email protected]> Co-authored-by: bfan05 <[email protected]> Co-authored-by: Jonathan Wang <[email protected]> Co-authored-by: Osama Alkhodairy <[email protected]> Co-authored-by: Zach Langley <[email protected]> * Tmp: remove predicate * Update readme * Update commands for newest rows * feat: add tracing and switch default fri params to log_blowup_factor = 3 * chore: update logging * Fix readme typo * Fix clippy warnings * Update to handle input byte lengths * Remove extraneous code * Revert 9f02efc^..9bad23e * Update logical interface to add byte lengths * Remove generics from logical-interface * merging new Offline Checker changes * Revert "change branches" This reverts commit ec8639f. * Update mock read/write commands * chore: add `mock describe` command * chore: do not overwrite db if exists * chore: remove schema config * chore: switch config back to data_bytes=1024 --------- Co-authored-by: MonkeyKing-1 <[email protected]> Co-authored-by: bfan <[email protected]> Co-authored-by: Osama Alkhodairy <[email protected]> Co-authored-by: Jonathan Wang <[email protected]> Co-authored-by: Zach Langley <[email protected]> Co-authored-by: TlatoaniHJ <[email protected]>
openvm-org · Jun 21, 2024 · 04514d0 · 04514d0
1 parent 3aeea45
commit 04514d0
Show file tree

Hide file tree

Showing 82 changed files with 171,453 additions and 505 deletions.
diff --git a/Cargo.toml b/Cargo.toml
@@ -4,6 +4,7 @@ members = [
     "stark-backend",
     "test-utils",
     "derive",
+    "bin/afs",
     "logical-interface",
 ]
 resolver = "2"

diff --git a/bin/afs/Cargo.toml b/bin/afs/Cargo.toml
@@ -0,0 +1,29 @@
+[package]
+name = "afs"
+version.workspace = true
+authors.workspace = true
+edition.workspace = true
+
+[dependencies]
+alloy-primitives = "0.7.6"
+bincode = "1.3.3"
+clap = { version = "4.5.4", features = ["derive"] }
+color-eyre = "0.6.3"
+logical-interface = { path = "../../logical-interface" }
+rand = "0.8.5"
+serde = "1.0.203"
+serde_derive = "1.0.203"
+serde_json = "1.0.117"
+tokio = { version = "1.38.0", features = ["rt", "macros"] }
+toml = "0.8.14"
+p3-baby-bear = { workspace = true }
+p3-util = { workspace = true }
+afs-stark-backend = { path = "../../stark-backend" }
+afs-derive = { path = "../../derive" }
+afs-test-utils = { path = "../../test-utils" }
+afs-chips = { path = "../../chips" }
+p3-uni-stark = { workspace = true }
+
+tracing = "0.1.37"
+tracing-subscriber = { version = "0.3.17", features = ["std", "env-filter"] }
+tracing-forest = { version = "0.1.6", features = ["ansi", "smallvec"] }
diff --git a/bin/afs/README.md b/bin/afs/README.md
@@ -0,0 +1,99 @@
+# AFS Query Binary
+
+## Instructions
+
+Display help:
+
+```bash
+cargo run --release --bin afs -- --help
+```
+
+## Configuration
+
+All configuration is handled via the `config.toml` file at the root of the repository.
+
+## Commands
+
+Commands to generate keys, cache traces, prove, and verify. Run these commands from the root of the repository.
+
+### keygen
+
+Generate partial proving and verifying keys.
+
+```bash
+cargo run --release --bin afs -- keygen --output-folder bin/afs/tests/data
+```
+
+### cache
+
+Cache a trace of a table.
+
+```bash
+cargo run --release --bin afs -- cache -t 0x155687649d5789a399211641b38bb93139f8ceca042466aa98e500a904657711 --db-file bin/afs/tests/data/input_file_32_1024.mockdb --output-file bin/afs/tests/data
+```
+
+### prove
+
+Prove a set of instructions.
+
+```bash
+cargo run --release --bin afs -- prove --afi-file bin/afs/tests/data/test_input_file_32_1024.afi --db-file bin/afs/tests/data/input_file_32_1024.mockdb --cache-folder bin/afs/tests/data --keys-folder bin/afs/tests/data
+```
+
+### verify
+
+Verify the proof
+
+```bash
+cargo run --release --bin afs -- verify --proof-file bin/afs/tests/data/input_file_32_1024.mockdb.prove.bin --db-file bin/afs/tests/data/input_file_32_1024.mockdb --keys-folder bin/afs/tests/data
+```
+
+## Mock commands
+
+Useful for reading/writing the .mockdb files. Run these commands from the root of the repository.
+
+### Describe
+
+List all tables and table metadata in a given mock database file. Set the --db-file (-d) flag.
+
+```bash
+cargo run --release --bin afs -- mock describe -d bin/afs/tests/data/afs_db.mockdb
+```
+
+### Read
+
+Read from a local mock database file. Set the --db-file (-d), --table-id (-t), and print to stdout with the --print (-p) flag.
+
+```bash
+cargo run --release --bin afs -- mock read -d bin/afs/tests/data/afs_db.mockdb -t 5
+```
+
+### Write
+
+Write to a local mock database file using an AFS Instruction file. Set the --afi-file (-f), --db-file (-d) to write the AFI file into the mock database. Optionally set --print (-p) to print to stdout and --output-db-file (-o) to save the new mock database to file.
+
+```bash
+cargo run --release --bin afs -- mock write -f bin/afs/tests/data/test_input_file_32_1024.afi -d bin/afs/tests/data/afs_db.mockdb -o bin/afs/tests/data/afs_db1.mockdb
+```
+
+### AFI
+
+Print the afs instruction set to file.
+
+```bash
+cargo run --release --bin afs -- mock afi -f bin/afs/tests/data/test_input_file_32_1024.afi
+```
+
+## Full test
+
+```bash
+cargo run --release --bin afs -- mock write -f bin/afs/tests/data/test_input_file_32_1024.afi -o bin/afs/tests/data/input_file_32_1024.mockdb
+
+cargo run --release --bin afs -- keygen --output-folder bin/afs/tests/data
+
+cargo run --release --bin afs -- cache -t 0x155687649d5789a399211641b38bb93139f8ceca042466aa98e500a904657711 --db-file bin/afs/tests/data/input_file_32_1024.mockdb --output-file bin/afs/tests/data
+
+cargo run --release --bin afs -- prove --afi-file bin/afs/tests/data/test_input_file_32_1024.afi --db-file bin/afs/tests/data/input_file_32_1024.mockdb --cache-folder bin/afs/tests/data --keys-folder bin/afs/tests/data
+
+cargo run --release --bin afs -- verify --proof-file bin/afs/tests/data/input_file_32_1024.mockdb.prove.bin --db-file bin/afs/tests/data/input_file_32_1024.mockdb --keys-folder bin/afs/tests/data
+```
diff --git a/bin/afs/src/cli/mod.rs b/bin/afs/src/cli/mod.rs
@@ -0,0 +1,66 @@
+use crate::commands::keygen::KeygenCommand;
+use crate::commands::{cache, keygen, mock, prove, verify};
+use afs_test_utils::page_config::PageConfig;
+use clap::Parser;
+use clap::Subcommand;
+
+#[derive(Debug, Parser)]
+#[command(author, version, about = "AFS CLI")]
+#[command(propagate_version = true)]
+pub struct Cli {
+    #[command(subcommand)]
+    pub command: CliCommand,
+}
+
+#[derive(Debug, Subcommand)]
+pub enum CliCommand {
+    #[command(name = "mock", about = "Mock functions")]
+    /// Mock functions
+    Mock(mock::MockCommand),
+
+    #[command(name = "keygen", about = "Generate partial proving and verifying keys")]
+    /// Generate partial proving and verifying keys
+    Keygen(keygen::KeygenCommand),
+
+    #[command(
+        name = "cache",
+        about = "Create the cached trace of a page from a page file"
+    )]
+    /// Create cached trace of a page from a page file
+    Cache(cache::CacheCommand),
+
+    #[command(name = "prove", about = "Generates a multi-STARK proof")]
+    /// Generates a multi-STARK proof
+    Prove(prove::ProveCommand),
+
+    #[command(name = "verify", about = "Verifies a multi-STARK proof")]
+    /// Verifies a multi-STARK proof
+    Verify(verify::VerifyCommand),
+}
+
+impl Cli {
+    pub fn run(config: &PageConfig) -> Self {
+        let cli = Self::parse();
+        match &cli.command {
+            CliCommand::Mock(mock) => {
+                mock.execute(config).unwrap();
+            }
+            CliCommand::Keygen(keygen) => {
+                let cmd = KeygenCommand {
+                    output_folder: keygen.output_folder.clone(),
+                };
+                cmd.execute(config).unwrap();
+            }
+            CliCommand::Cache(cache) => {
+                cache.execute(config).unwrap();
+            }
+            CliCommand::Prove(prove) => {
+                prove.execute(config).unwrap();
+            }
+            CliCommand::Verify(verify) => {
+                verify.execute(config).unwrap();
+            }
+        }
+        cli
+    }
+}
diff --git a/bin/afs/src/commands/cache/mod.rs b/bin/afs/src/commands/cache/mod.rs
@@ -0,0 +1,95 @@
+use std::time::Instant;
+
+use afs_chips::common::page::Page;
+use afs_stark_backend::prover::{trace::TraceCommitmentBuilder, MultiTraceStarkProver};
+use afs_test_utils::{
+    config::{self, baby_bear_poseidon2::BabyBearPoseidon2Config},
+    page_config::PageConfig,
+};
+use clap::Parser;
+use color_eyre::eyre::Result;
+use logical_interface::{afs_interface::AfsInterface, mock_db::MockDb};
+use p3_baby_bear::BabyBear;
+use p3_util::log2_strict_usize;
+
+use crate::commands::write_bytes;
+
+#[cfg(test)]
+pub mod tests;
+
+/// `afs cache` command
+#[derive(Debug, Parser)]
+pub struct CacheCommand {
+    #[arg(long = "table-id", short = 't', help = "The table ID", required = true)]
+    pub table_id: String,
+
+    #[arg(
+        long = "db-file",
+        short = 'd',
+        help = "Mock DB file input",
+        required = true
+    )]
+    pub db_file_path: String,
+
+    #[arg(
+        long = "output-file",
+        short = 'o',
+        help = "The folder to output the cached traces to",
+        required = false,
+        default_value = "cache"
+    )]
+    pub output_folder: String,
+}
+
+impl CacheCommand {
+    /// Execute the `cache` command
+    pub fn execute(&self, config: &PageConfig) -> Result<()> {
+        println!("Caching table {} from {}", self.table_id, self.db_file_path);
+
+        let start = Instant::now();
+        let mut db = MockDb::from_file(&self.db_file_path);
+        let height = config.page.height;
+        let mut interface =
+            AfsInterface::new(config.page.index_bytes, config.page.data_bytes, &mut db);
+        let page = interface.get_table(self.table_id.clone()).unwrap().to_page(
+            config.page.index_bytes,
+            config.page.data_bytes,
+            height,
+        );
+
+        assert!(height > 0);
+
+        let checker_trace_degree = config.page.max_rw_ops * 4;
+
+        let max_log_degree = log2_strict_usize(checker_trace_degree)
+            .max(log2_strict_usize(height))
+            .max(8);
+
+        let trace = page.gen_trace::<BabyBear>();
+        let engine = config::baby_bear_poseidon2::default_engine(max_log_degree);
+        let prover = MultiTraceStarkProver::new(&engine.config);
+        let trace_builder = TraceCommitmentBuilder::<BabyBearPoseidon2Config>::new(prover.pcs());
+        let trace_prover_data = trace_builder.committer.commit(vec![trace]);
+        let encoded_data = bincode::serialize(&trace_prover_data).unwrap();
+        let path = self.output_folder.clone() + "/" + &self.table_id + ".cache.bin";
+        write_bytes(&encoded_data, path).unwrap();
+
+        let duration = start.elapsed();
+        println!("Cached table {} in {:?}", self.table_id, duration);
+
+        Ok(())
+    }
+
+    pub fn read_page_file(&self) -> Result<Page> {
+        let path = self.output_folder.clone() + "/" + &self.table_id + ".cache.bin";
+        let page_file = std::fs::read(path)?;
+        let page_file: Page = serde_json::from_slice(&page_file)?;
+        Ok(page_file)
+    }
+
+    pub fn write_output_file(&self, output: Vec<u8>) -> Result<()> {
+        let path = self.output_folder.clone() + "/" + &self.table_id + ".cache.bin";
+        std::fs::write(path, output)?;
+        Ok(())
+    }
+}
diff --git a/bin/afs/src/commands/cache/tests.rs b/bin/afs/src/commands/cache/tests.rs
@@ -0,0 +1,11 @@
+// use super::CacheCommand;
+
+// #[test]
+// fn test_read_page_file() {
+//     let cache = CacheCommand {
+//         page_file: "tests/cache_page_file.json".to_string(),
+//         output_file: "tests/cache_output_cache.bin".to_string(),
+//     };
+//     let page_file = cache.read_page_file().unwrap();
+//     println!("{:?}", page_file);
+// }