Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node from stream backrefs optimisation #532

Merged
merged 110 commits into from
Feb 11, 2025
Merged
Changes from 1 commit
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
438e2df
initial commit
matt-o-how Jan 13, 2025
fdeef9f
basic structure
matt-o-how Jan 13, 2025
bdeea87
comment for clarity
matt-o-how Jan 13, 2025
a9b3351
Improve clarity in error messages and comments
matt-o-how Jan 13, 2025
47e41b8
passing tests!
matt-o-how Jan 13, 2025
5d38c06
clippy cleanups
matt-o-how Jan 13, 2025
99b0c1d
clarify comment
matt-o-how Jan 13, 2025
ddceeba
remove Cost and move into de_br
matt-o-how Jan 14, 2025
8060a47
use an index to simulate stack rather than cloning vec
matt-o-how Jan 14, 2025
d394e0b
add catch for underflow
matt-o-how Jan 14, 2025
f8f1e08
catch another underflow
matt-o-how Jan 14, 2025
de2b748
add fuzz against old
matt-o-how Jan 15, 2025
3d35f09
fmt
matt-o-how Jan 15, 2025
5231929
add pair count fuzz
matt-o-how Jan 15, 2025
791030b
fmt fuzz
matt-o-how Jan 15, 2025
675ab2c
try adding features to fuzzer cargo.toml
matt-o-how Jan 15, 2025
982498f
forward the counters feature
matt-o-how Jan 15, 2025
3b86c79
fix name in fuzz cargo.toml
matt-o-how Jan 15, 2025
423dd73
add all features to cargo fuzz github action
matt-o-how Jan 15, 2025
64340f1
return error when traversing empty values stack
matt-o-how Jan 15, 2025
bd93d2a
special case 0xfe 0x01
matt-o-how Jan 15, 2025
49e26e2
special case 0xfe 0x00
matt-o-how Jan 15, 2025
25cf07c
treat empty stack as sexp
matt-o-how Jan 15, 2025
d322960
prevent underflow in empty case
matt-o-how Jan 15, 2025
fcc634e
parse as empty list when we empty the list
matt-o-how Jan 15, 2025
8b31b55
add benchmarking for node_from_bytes_backrefs_old
matt-o-how Jan 16, 2025
55085d0
add stackpointer on empty stack test case
matt-o-how Jan 29, 2025
e7bb047
initial commit
matt-o-how Jan 13, 2025
9e96280
basic structure
matt-o-how Jan 13, 2025
880d7f6
comment for clarity
matt-o-how Jan 13, 2025
97f6f0f
Improve clarity in error messages and comments
matt-o-how Jan 13, 2025
0ce9f50
passing tests!
matt-o-how Jan 13, 2025
31e4fb4
clippy cleanups
matt-o-how Jan 13, 2025
b2946db
clarify comment
matt-o-how Jan 13, 2025
1ef2bc7
remove Cost and move into de_br
matt-o-how Jan 14, 2025
14d1de5
use
matt-o-how Jan 29, 2025
c469103
use checkpoint in fuzzer
matt-o-how Jan 29, 2025
d3986f9
pub(crate) msb_mask
matt-o-how Jan 29, 2025
e7a8b5c
add unit test for traverse_path_with_vec - not working
matt-o-how Jan 29, 2025
864ef16
fix rebase weirdness
matt-o-how Jan 30, 2025
d6d8cd4
fix the unit test
matt-o-how Jan 30, 2025
0a00f77
fmt unit test
matt-o-how Jan 30, 2025
85a9785
Update src/serde/de_br.rs
matt-o-how Jan 30, 2025
e8f3f2a
add cache
matt-o-how Jan 30, 2025
1a30940
clippy fix
matt-o-how Jan 30, 2025
1f790fe
Update fuzz/Cargo.toml
matt-o-how Jan 31, 2025
e2fed6c
remove useless line
matt-o-how Jan 31, 2025
eb9f678
underscore unused vars
matt-o-how Jan 31, 2025
62c4a82
merge the fuzzers into one
matt-o-how Jan 31, 2025
c6a4071
check original version of fuzzer
matt-o-how Jan 31, 2025
55cdb7f
cache when arg_index == 0
matt-o-how Jan 31, 2025
437f493
test fuzzing without checkpoint
matt-o-how Jan 31, 2025
08dd203
try 2 allocators
matt-o-how Jan 31, 2025
948712f
try checkpointing differently
matt-o-how Jan 31, 2025
11ad810
fix typo
matt-o-how Jan 31, 2025
90bb3de
test if assertion was the real problem
matt-o-how Jan 31, 2025
bf09050
clippy fix for check
matt-o-how Jan 31, 2025
24abb5a
check old first
matt-o-how Jan 31, 2025
11fb7f6
fmt for check
matt-o-how Jan 31, 2025
a82e47f
disable checkpoint
matt-o-how Jan 31, 2025
e1d0b32
add pair count fuzz
matt-o-how Jan 31, 2025
cbdca85
Update src/serde/de_br.rs
matt-o-how Feb 3, 2025
271a3e7
Revert "Update src/serde/de_br.rs"
matt-o-how Feb 3, 2025
9c39cbf
use and fill cache in every pass
matt-o-how Feb 3, 2025
9f3f5d2
add break to loop
matt-o-how Feb 3, 2025
319ef6b
optimise cache usage to one loop
matt-o-how Feb 3, 2025
509b966
remove round-trip from fuzzer and other cleanups
matt-o-how Feb 3, 2025
afe636f
remove commented out code
matt-o-how Feb 3, 2025
185d6b8
add assertions to validate program deserialization
matt-o-how Feb 3, 2025
27cc61a
assert XOR is_err()
matt-o-how Feb 3, 2025
d0f6dde
remove error comparison
matt-o-how Feb 3, 2025
3e6317c
refactor comments for clarity
matt-o-how Feb 4, 2025
29fe058
change allocator to have adjustable internal limits for pairs and ato…
matt-o-how Feb 4, 2025
5d7b742
fmt
matt-o-how Feb 4, 2025
6b06165
increase max when we actually allocate pairs
matt-o-how Feb 4, 2025
359e5e1
add better error message
matt-o-how Feb 4, 2025
1cdc750
cache node 0 and count it as a pair
matt-o-how Feb 4, 2025
138d75c
reduce pair amount for cons
matt-o-how Feb 4, 2025
d6a5deb
more max reduction
matt-o-how Feb 4, 2025
7dfdf1d
use saturating add in increase_max_pair
matt-o-how Feb 5, 2025
3685eca
more accurate comparisons for max pairs
matt-o-how Feb 5, 2025
a9eb01b
MAX_NUM_ATOMS not usize::MAX
matt-o-how Feb 5, 2025
59986d3
condense to one line
matt-o-how Feb 5, 2025
1e0e831
check for failure parity in fuzzer and reuse data instead of generati…
matt-o-how Feb 6, 2025
50f37ae
switch to using ghost pairs in allocator rather than internal max
matt-o-how Feb 6, 2025
cdd627e
further simplify fuzz for de_brs
matt-o-how Feb 6, 2025
258f92b
remove special case for arg_index 0 from traverse_path_with_vec
matt-o-how Feb 6, 2025
5d24e54
remove some redundant code
matt-o-how Feb 6, 2025
250cc15
Add comments to ghost pair funcs
matt-o-how Feb 6, 2025
9f81c62
counter includes ghost pairs in pairs, fuzzer checks for equality
matt-o-how Feb 6, 2025
573b071
rename ghost_pair functions for clarity
matt-o-how Feb 6, 2025
a9afbc6
add debug output message if fuzzer fails
matt-o-how Feb 6, 2025
da02ba0
add test which triggers counters discrepancy
matt-o-how Feb 6, 2025
1fb0a47
fix checkpointing for ghost_pair counting
matt-o-how Feb 6, 2025
fa4523b
remove commented out optimisation
matt-o-how Feb 7, 2025
9a43216
address arvids comments about allocator
matt-o-how Feb 7, 2025
cbd3d01
fmt
matt-o-how Feb 7, 2025
824644f
fix allocation amounts
matt-o-how Feb 7, 2025
2065df1
add unit test for ghost pair limit
matt-o-how Feb 7, 2025
140e908
remove allocator.max_num_atoms and add comment for num_ghost_pairs
matt-o-how Feb 10, 2025
ad09f81
add additional test case
matt-o-how Feb 10, 2025
f2571b3
reformat fuzzer
matt-o-how Feb 10, 2025
494f536
fmt fuzz
matt-o-how Feb 10, 2025
b56cb0f
fix fuzz
matt-o-how Feb 10, 2025
647ee8a
don't compare errors as they don't support ==
matt-o-how Feb 10, 2025
a4818c4
fmt fuzz again
matt-o-how Feb 10, 2025
c197b2b
underscore unused vars
matt-o-how Feb 10, 2025
e896a8c
remove fuzz target for deserialization pair count
matt-o-how Feb 10, 2025
144b4dd
add cache related tests for traverse_path_with_vec
matt-o-how Feb 11, 2025
76bc1b4
address arvid's comments
matt-o-how Feb 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test if assertion was the real problem
  • Loading branch information
matt-o-how committed Jan 31, 2025
commit 90bb3deccffaed39e465158c7aa19fc7620e9335
2 changes: 1 addition & 1 deletion fuzz/fuzz_targets/deserialize_br.rs
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@ fuzz_target!(|data: &[u8]| {
let new_pair_count = allocator.pair_count();
allocator.restore_checkpoint(&cp);
let program_old = node_from_bytes_backrefs_old(&mut allocator, &b1).unwrap();
assert!(new_pair_count <= allocator.pair_count());
// assert!(new_pair_count <= allocator.pair_count());
let b2 = node_to_bytes_backrefs(&allocator, program).unwrap();
assert_eq!(b1, b2);
let b3 = node_to_bytes_backrefs(&allocator, program_old).unwrap();