Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop KeyIndexType.SEQUENTIAL_INCREMENTING_LONGS support #14309

Closed
artemananiev opened this issue Jul 18, 2024 · 0 comments · Fixed by #14323
Closed

Drop KeyIndexType.SEQUENTIAL_INCREMENTING_LONGS support #14309

artemananiev opened this issue Jul 18, 2024 · 0 comments · Fixed by #14323
Assignees
Labels
Platform Data Structures Platform Virtual Map Platform Tickets pertaining to the platform Tech Debt Reduced Issues which reduce technical debt.
Milestone

Comments

@artemananiev
Copy link
Member

MerkleDb supports two key modes: long keys and generic keys. The corresponding values are captured in the KeyIndexType enum. Long mode provides significant performance boost, but unfortunately its use cases are very limited. Today this mode is only used in tests and benchmarks, while all Hedera virtual maps use the generic mode.

Here is a related comment in OnDiskKeySerializer:

    public int getSerializedSize() {
        // We're going to use variable size keys, always. MerkleDB was designed with
        // fast paths if you knew you were using a Long as the key -- but we really
        // cannot use that. The problem manifests itself with state proofs. We wanted
        // to be able to say that we only store a long for the account ID, and implicitly
        // know the shard and realm as they could be system properties when the JVM
        // is started up. The problem is that the shard and realm MUST be part of the
        // serialized bytes, because they MUST be part of the state proof. Otherwise, you
        // would only have a proof of the accountNum, but not the shard and realm (well,
        // you may be able to reconstruct the shard and realm via the signature on the
        // state proof, if each shard has its own public key / ledger id, but that is
        // very cumbersome!).
        //
        // With protobuf serialization, we can encode shard/realm/num as a single long
        // in many cases, since "0" is the default for the fields, and therefore can
        // be skipped entirely. And anyway they are varint encoded, so for low values
        // of shard and realm (likely to be true for a very long time), they will
        // at most consume 7 bits each of the value.
        //
        // While that may be true, the KeySerializer requires either fixed size or variable
        // size, and protobuf would either be fixed > 8 bytes, or variable sized, and being
        // fixed but greater than 8 bytes doesn't help us in performance, so we will
        // have to use variable size always.
        return VARIABLE_DATA_SIZE;
    }

This ticket is to drop the long key mode support and leave the generic mode only.

@artemananiev artemananiev added Tech Debt Reduced Issues which reduce technical debt. Platform Virtual Map Platform Data Structures Platform Tickets pertaining to the platform labels Jul 18, 2024
@artemananiev artemananiev self-assigned this Jul 18, 2024
@artemananiev artemananiev added this to the v0.53 milestone Jul 19, 2024
artemananiev added a commit that referenced this issue Jul 19, 2024
…14323)

Fixes: #14309
Reviewed-by: Anthony Petrov <[email protected]>, Austin Littley <[email protected]>, Ivan Malygin <[email protected]>, Oleg Mazurov <[email protected]>
Signed-off-by: Artem Ananev <[email protected]>
mxtartaglia-sl pushed a commit that referenced this issue Jul 23, 2024
…14323)

Fixes: #14309
Reviewed-by: Anthony Petrov <[email protected]>, Austin Littley <[email protected]>, Ivan Malygin <[email protected]>, Oleg Mazurov <[email protected]>
Signed-off-by: Artem Ananev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Platform Data Structures Platform Virtual Map Platform Tickets pertaining to the platform Tech Debt Reduced Issues which reduce technical debt.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant