Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Epic 2: Replace tm-db database interface with new cosmos-db #2

Closed
i-norden opened this issue Apr 19, 2021 · 3 comments · Fixed by cosmos/cosmos-sdk#9573 or cosmos/cosmos-sdk#9952
Closed

Epic 2: Replace tm-db database interface with new cosmos-db #2

i-norden opened this issue Apr 19, 2021 · 3 comments · Fixed by cosmos/cosmos-sdk#9573 or cosmos/cosmos-sdk#9952

Comments

@i-norden
Copy link
Collaborator

i-norden commented Apr 19, 2021 •
edited
Loading

Currently the SDK uses tm-db as its low-level database interface. Here we create a new interface to replace tm-db and support ADR-040. This interface will be packaged as part of the SDK. This interface will only support databases which support efficient DB Snapshot. At the time of writing we are going to support RocksDB and BadgerDB.

Our new interface will

  • use batched transactions
  • support versioning
  • support direct querying and abci.Querying
  • needs to support the MapStore interface underpinning the SMT
  • be generic enough to work on top of our two selected databases: RocksDB and BadgerDB

After creating the interface we need to integrate it into the SDK, replacing the dependency on tm-db.
@i-norden i-norden mentioned this issue Apr 27, 2021
Closed
@i-norden i-norden changed the title Update state database to the new interface tm-dbv2 Epic 2: Update state database to the new interface tm-dbv2 May 7, 2021
@roysc roysc closed this as completed May 20, 2021
@i-norden i-norden reopened this May 20, 2021
@i-norden i-norden changed the title Epic 2: Update state database to the new interface tm-dbv2 Epic 2: Replace tm-db database interface with new cosmos-db Jun 9, 2021
@i-norden i-norden mentioned this issue Jun 10, 2021
Closed
@i-norden
Copy link
Collaborator Author

i-norden commented Jun 10, 2021 •
edited
Loading

MapStore updates

We will need to update the MapStore interface used by the LazyLedger SMT to expose needed features from the backend DB.

Batch support

If we want to implement efficient batched writes to the SMT, that could be supported in the following way. The tree will need to be able to read as well as write to the batch itself while building it, using something closer to a transaction object.

  • RocksDB has a Write Batch With Index utility which allows this.
  • BadgerDB's WriteBatch does not support Gets, but, as it is a relatively simple wrapper around a transaction, we could just use Txn.

These writable batch types would be wrapped in the MapStore interface. The SMT would need a new interface type to represent the batch (e.g. WriteBatch). This would wrap a derived SparseMerkleTree object using the transaction object as its MapStore. All normal SMT operations would then be available on the batch.

Versioning and snapshots

Creating state sync snapshots is not required for SMT data.

Versioning may not be needed either; however, if it is desired, it could be implemented with a MapStore.GetAt([]byte, uint64) method, which forwards to a DB.GetAt() method as described above.

@i-norden i-norden mentioned this issue Jun 23, 2021
Closed
3 tasks
@roysc
Copy link
Collaborator

roysc commented Aug 3, 2021

Feature PR on Cosmos SDK: cosmos#9573

@roysc roysc mentioned this issue Aug 5, 2021
Merged
19 tasks
@roysc roysc mentioned this issue Aug 17, 2021
Merged
19 tasks
mergify bot pushed a commit to cosmos/cosmos-sdk that referenced this issue Aug 19, 2021
@roysc
feat: ADR 040: New DB interface (#9573)
e3aec18 
<!--
The default pull request template is for types feat, fix, or refactor.
For other templates, add one of the following parameters to the url:
- template=docs.md
- template=other.md
-->

## New DB interface to replace `tm-db`

This introduces a new interface to wrap the backend KV store DB while supporting versioning and ACID transactions.
Part of [ADR-040](https://github.com/cosmos/cosmos-sdk/blob/eb7d939f86c6cd7b4218492364cdda3f649f06b5/docs/architecture/adr-040-storage-and-smt-state-commitments.md) changes.

This has been revised to consist of only the interface types. All implementations and utilities will be in follow-up PRs.

Partially resolves: vulcanize#2

---

### Author Checklist

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - N/A
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [x] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [x] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed

### Reviewers Checklist

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed 
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
mergify bot pushed a commit to cosmos/cosmos-sdk that referenced this issue Aug 31, 2021
@roysc
feat: ADR 040: Implement in-memory DB backend (#9952)
2c31451 
## Description

Implements an in-memory backend for the DB interface introduced by #9573 and specified by [ADR-040](https://github.com/cosmos/cosmos-sdk/blob/eb7d939f86c6cd7b4218492364cdda3f649f06b5/docs/architecture/adr-040-storage-and-smt-state-commitments.md). This expands on the [btree](https://pkg.go.dev/github.com/google/btree)-based [`MemDB`](https://github.com/tendermint/tm-db/tree/master/memdb) from `tm-db` by using copy-on-write clones to implement versioning.

Resolves: vulcanize#2

Will move out of draft once #9573 is merged and rebased on.

### Author Checklist

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [x] provided a link to the relevant issue or specification
- [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) n/a
- [x] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [x] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [x] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed

### Reviewers Checklist

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed 
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
@i-norden
Copy link
Collaborator Author

Closing, see cosmos#9816 and child issues, tracking on zenhub

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@i-norden @roysc