Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage/engine: add MVCC metamorphic test #43762

Closed
petermattis opened this issue Jan 6, 2020 · 0 comments · Fixed by #44630
Closed

storage/engine: add MVCC metamorphic test #43762

petermattis opened this issue Jan 6, 2020 · 0 comments · Fixed by #44630
Assignees
@itsbilal

Comments

@petermattis
Copy link
Collaborator

The Pebble metamorphic test has uncovered numerous bugs. We should introduce a similar test at the MVCC layer which can be used to compare the RocksDB and Pebble MVCC implementations, as well as providing bidirectional compatibility testing.

The high-level idea of the Pebble metamorphic test is to generate a series of random Pebble operations, serialize these operations to disk, run the operations against differing Pebble configurations, and compare the resulting output for differences. The same general idea would be done for an MVCC metamorphic test, except we'd replace Pebble operations with MVCC operations. The random operation generation would need to be know about transactions, and meta operations on transactions such as creating, committing, aborting, and bumping the epoch and sequence numbers.

In addition to running the operations against RocksDB and Pebble, bidirectional compatibility testing could be performed by introducing a "restart" operation alongside the MVCC operations. A "restart" operation would cause all open snapshots, iterators, and batches to be orphaned. Upon "restart", the underlying engine could be randomly switched.

The MVCC metamorphic test would run a set of operations against RocksDB, and against Pebble, and against a config which randomly switches the backend at "restart". Any discrepancies in the output would be indications of likely bugs.

Cc @itsbilal and @sumeerbhola as I discussed some of these ideas with both of you separately today. Does the above description make sense? We'd probably also want a roachtest for RocksDB/Pebble compatibility, but doing something in the metamorphic-style would be easier to cover the full MVCC surface area.
@itsbilal itsbilal mentioned this issue Jan 28, 2020
Merged
4 tasks
itsbilal added a commit to itsbilal/cockroach that referenced this issue Jan 28, 2020
@itsbilal
storage/engine: MVCC Metamorphic test suite, first phase
bb8bf97 
This PR adds a new test-only sub-package to engine, metamorphic,
which has one test, TestMeta, that generates and runs random MVCC
operations on rocksdb and pebble instances with default settings.

Future additions to this test suite could include:

 - A "check" mode that takes an output file as input, parses it,
   runs the operations in that sequence, and compares output
   strings.
 - Diffing test output between rocksdb and pebble and failing if
   there's a difference
 - Adding support for more operations
 - Adding a "restart" operation that closes the engine and
   restarts a different kind of engine in the store directory,
   then confirming operations after that point generate the
   same output.

First-but-biggest part of cockroachdb#43762 .

Release note: None
itsbilal added a commit to itsbilal/cockroach that referenced this issue Jan 29, 2020
@itsbilal
storage/engine: MVCC Metamorphic test suite, first phase
d9f7baf 
This PR adds a new test-only sub-package to engine, metamorphic,
which has one test, TestMeta, that generates and runs random MVCC
operations on rocksdb and pebble instances with default settings.

Future additions to this test suite could include:

 - A "check" mode that takes an output file as input, parses it,
   runs the operations in that sequence, and compares output
   strings.
 - Diffing test output between rocksdb and pebble and failing if
   there's a difference
 - Adding support for more operations
 - Adding a "restart" operation that closes the engine and
   restarts a different kind of engine in the store directory,
   then confirming operations after that point generate the
   same output.

First-but-biggest part of cockroachdb#43762 .

Release note: None
@itsbilal itsbilal mentioned this issue Feb 2, 2020
Merged
itsbilal added a commit to itsbilal/cockroach that referenced this issue Feb 5, 2020
@itsbilal
storage/engine: MVCC Metamorphic test suite, first phase
aee94ca 
This PR adds a new test-only sub-package to engine, metamorphic,
which has one test, TestMeta, that generates and runs random MVCC
operations on rocksdb and pebble instances with default settings.

Future additions to this test suite could include:

 - A "check" mode that takes an output file as input, parses it,
   runs the operations in that sequence, and compares output
   strings.
 - Diffing test output between rocksdb and pebble and failing if
   there's a difference
 - Adding support for more operations
 - Adding a "restart" operation that closes the engine and
   restarts a different kind of engine in the store directory,
   then confirming operations after that point generate the
   same output.

First-but-biggest part of cockroachdb#43762 .

Release note: None
itsbilal added a commit to itsbilal/cockroach that referenced this issue Feb 6, 2020
@itsbilal
engine/metamorphic: [WIP] Add engine restarts, compare across runs`
ebeaaea 
This PR builds on top of cockroachdb#44458 (all commits before the last one
are from that PR). It adds two things: one, it ensures that whenever
successive engine types are tested, it does a comparison for matching
outputs. This will ensure CI will fail if any change down the line
causes an observed difference in behaviour between rocksdb and pebble.

Furthermore, it adds a new kind of operation: restart. Restarts are
special in that they're added after the specified n number of runs
have happened; so a test run with 3 specified engine types
(so 2 restarts), and n = 10000 will have 3 * 10000 = 30000 operations.
A restart closes all open objects, then closes the engine, and starts
up the next engine in the specified engine sequence. This, combined
with the aforementioned checking functionality across different
engine runs, lets us test for bidirectional compatibility across
different engines.

Fixes cockroachdb#43762 .

Release note: None.
craig bot pushed a commit that referenced this issue Feb 6, 2020
@itsbilal @andreimatei @dt
Merge #44458 #44730 #44786
ece0b94 
44458: storage/engine: MVCC Metamorphic test suite, first phase r=itsbilal a=itsbilal

This PR adds a new test-only sub-package to engine, metamorphic,
which has one test, TestMeta, that generates and runs random MVCC
operations on rocksdb and pebble instances with default settings.

Future additions to this test suite could include:

- [x]  A "check" mode that takes an output file as input, parses it, runs the operations in that sequence, and compares output strings.
- [ ]  Diffing test output between rocksdb and pebble and failing if there's a difference
- [ ]  Adding support for more operations
- [ ]  Adding a "restart" operation that closes the engine and restarts a different kind of engine in the store directory, then confirming operations after that point generate the same output.

First-but-biggest part of #43762 .

Release note: None

44730: storage: fix some tests that were fooling themselves r=andreimatei a=andreimatei

A couple of tests wanted multiple write too old errors, but they were
running at the wrong timestamp and so they were really getting a single
one.
Also add a test showing a funky scenario where 1PC batches behave
differently from non-1PC.

Release note: None

44786: jobs: allow blocking job adoption via sentinel file r=dt a=dt

This change adds a check for a sentinel file, DISABLE_STARTING_BACKGROUND_JOBS,
in the first on-disk store directory to the job adoption loop. If it is found,
jobs will not be adopted by that node and current job executions are cancelled.

Operators can thus touch this file if a misbehaving job is causing problems
and otherwise preventing traditional job control that requires being able to
write to the database.

Release note (general change): background job execution is disabled on nodes where the file DISABLE_STARTING_BACKGROUND_JOBS exists in the first store directory providing an emergency tool for disabling job execution.

Co-authored-by: Bilal Akhtar <[email protected]>
Co-authored-by: Andrei Matei <[email protected]>
Co-authored-by: David Taylor <[email protected]>
itsbilal added a commit to itsbilal/cockroach that referenced this issue Feb 7, 2020
@itsbilal
engine/metamorphic: Add engine restarts, compare across runs`
07e243b 
This PR builds on top of cockroachdb#44458 (all commits before the last one
are from that PR). It adds two things: one, it ensures that whenever
successive engine types are tested, it does a comparison for matching
outputs. This will ensure CI will fail if any change down the line
causes an observed difference in behaviour between rocksdb and pebble.

It also adds more MVCC operations that would be useful to test, such
as MVCCFindSplitKey, MVCCDeleteRange, MVCCClearTimeRange,
MVCCConditionalPut, etc.

Furthermore, it adds a new kind of operation: restart. Restarts are
special in that they're added after the specified n number of runs
have happened; so a test run with 3 specified engine types
(so 2 restarts), and n = 10000 will have 3 * 10000 = 30000 operations.
A restart closes all open objects, then closes the engine, and starts
up the next engine in the specified engine sequence. This, combined
with the aforementioned checking functionality across different
engine runs, lets us test for bidirectional compatibility across
different engines.

Fixes cockroachdb#43762 .

Release note: None.
itsbilal added a commit to itsbilal/cockroach that referenced this issue Feb 10, 2020
@itsbilal
engine/metamorphic: Add engine restarts, compare across runs`
cf4b65d 
This PR builds on top of cockroachdb#44458 (all commits before the last one
are from that PR). It adds two things: one, it ensures that whenever
successive engine types are tested, it does a comparison for matching
outputs. This will ensure CI will fail if any change down the line
causes an observed difference in behaviour between rocksdb and pebble.

It also adds more MVCC operations that would be useful to test, such
as MVCCFindSplitKey, MVCCDeleteRange, MVCCClearTimeRange,
MVCCConditionalPut, etc.

Furthermore, it adds a new kind of operation: restart. Restarts are
special in that they're added after the specified n number of runs
have happened; so a test run with 3 specified engine types
(so 2 restarts), and n = 10000 will have 3 * 10000 = 30000 operations.
A restart closes all open objects, then closes the engine, and starts
up the next engine in the specified engine sequence. This, combined
with the aforementioned checking functionality across different
engine runs, lets us test for bidirectional compatibility across
different engines.

Fixes cockroachdb#43762 .

Release note: None.
craig bot pushed a commit that referenced this issue Feb 12, 2020
@itsbilal
Merge #44630
0cdbfd3 
44630:  engine/metamorphic: Add engine restarts, compare across runs r=itsbilal a=itsbilal

This PR builds on top of #44458 (all commits before the last one
are from that PR). It adds two things: one, it ensures that whenever
successive engine types are tested, it does a comparison for matching
outputs. This will ensure CI will fail if any change down the line
causes an observed difference in behaviour between rocksdb and pebble.

Furthermore, it adds a new kind of operation: restart. Restarts are
special in that they're added after the specified n number of runs
have happened; so a test run with 3 specified engine types
(so 2 restarts), and n = 10000 will have 3 * 10000 = 30000 operations.
A restart closes all open objects, then closes the engine, and starts
up the next engine in the specified engine sequence. This, combined
with the aforementioned checking functionality across different
engine runs, lets us test for bidirectional compatibility across
different engines.

Fixes #43762 .

Release note: None.

Co-authored-by: Bilal Akhtar <[email protected]>
@craig craig bot closed this as completed in 4c72ea5 Feb 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@itsbilal itsbilal

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

engine/metamorphic: Add engine restarts, compare across runs itsbilal/cockroach
2 participants
@itsbilal @petermattis