Remove RocksDB #1058
Remove RocksDB #1058
Conversation
ogenev
force-pushed
the
remove-rocksdb
branch
5 times, most recently
from
a854cec to
2103ffa
Compare
ogenev
force-pushed
the
remove-rocksdb
branch
from
2103ffa to
28585ea
Compare
portalnet/src/storage.rs
Outdated
|
|
||
| const INSERT_QUERY: &str = | ||
| "INSERT OR IGNORE INTO content_metadata (content_id_long, content_id_short, content_key, content_size) | ||
| VALUES (?1, ?2, ?3, ?4)"; | ||
| "INSERT OR IGNORE INTO content_data (content_id_long, content_id_short, content_key, content_value, content_size) |
There was a problem hiding this comment.
Before we deploy this change we should add a field to tag which network the content is for. We can probably extract it from the content_key, but the properties letting this hold true won't work for custom networks built on trin. I also want to avoid byte parsing when calculating what distribution of storage is taken by each network.
So if we can have a network tag field that would be super nice to avoid the 2 cases above. This doesn't have to be done in this PR. But it should be done before we deploy these breaking changes so we don't have to deploy breaking changes twice.
The reason for this change is so we can properly report how much of storage is being taken by each network. Right now we are incorrectly reporting this metric
There was a problem hiding this comment.
Yeah, I agree, it also makes sense to use a separate db table per network, we will discuss this.
| @@ -1,181 +0,0 @@ | |||
| use anyhow::Result; | |||
There was a problem hiding this comment.
It's not obvious to me that this is worth removing rather than transforming it to work with sqlite? I guess ultimately, we're not currently using this script in devops. But at some point I imagine we will? But maybe that's a big effort, and not really worth tackling at this point.
There was a problem hiding this comment.
I thought about this but we still don't know what exactly our database would look like when we refactor it to support beacon and state. I think it is more productive to remove it now and rewrite it if needed at a later stage.
ogenev
force-pushed
the
remove-rocksdb
branch
2 times, most recently
from
b851560 to
59e6642
Compare
ogenev
force-pushed
the
remove-rocksdb
branch
from
59e6642 to
b996b3c
Compare
|
Added a "network" column to the main db table as discussed in Trin's sync. |
ogenev
force-pushed
the
remove-rocksdb
branch
2 times, most recently
from
835a748 to
2ac6f3e
Compare
ogenev
force-pushed
the
remove-rocksdb
branch
from
2ac6f3e to
319c983
Compare
|
@njgheorghita Going to merge this to the upstream |
* refactor: remove RocksDB * refactor: remove `purge_db`d * fix: address remove RocksDB PR comments * feat(db): add "network" column to content table
* refactor: remove RocksDB * refactor: remove `purge_db`d * fix: address remove RocksDB PR comments * feat(db): add "network" column to content table
* refactor: remove RocksDB * refactor: remove `purge_db`d * fix: address remove RocksDB PR comments * feat(db): add "network" column to content table
* refactor: remove RocksDB * refactor: remove `purge_db`d * fix: address remove RocksDB PR comments * feat(db): add "network" column to content table
* refactor: remove RocksDB * refactor: remove `purge_db`d * fix: address remove RocksDB PR comments * feat(db): add "network" column to content table
* refactor: remove RocksDB * refactor: remove `purge_db`d * fix: address remove RocksDB PR comments * feat(db): add "network" column to content table
* refactor: remove RocksDB * refactor: remove `purge_db`d * fix: address remove RocksDB PR comments * feat(db): add "network" column to content table
What was wrong?
Remove RocksDB and switch to SQLite for storing content values.
How was it fixed?
content_valuecolumn to SQLite main db table and use this column to store and lookup data.purge_dbbinary.TrieDBbecause it is based on RocksDB.Tested this PR locally with
portal-hive.To-Do