Support using secondary indices with write-committed transactions #13180
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request was exported from Phabricator. Differential Revision: D66672931 |
ltamasi
force-pushed
the
export-D66672931
branch
from
88895dc to
3cef50b
Compare
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 3, 2024
…cebook#13180) Summary: The patch adds initial support for secondary indices using write-committed transactions. Currently, only the `PutEntity` API is supported; other APIs like `Put` and `Delete` will be added separately. Applications can set up secondary indices using the new configuration option `TransactionDBOptions::secondary_indices`. When secondary indices are enabled, calling `PutEntity` via a (n explicit or implicit) transaction performs the following steps: 1) Retrieves the current value (if any) of the primary key using `GetEntityForUpdate`. 2) If there is an existing primary key-value, it removes any existing secondary index entries using `SingleDelete`. (Note: as a later optimization, we can avoid removing and recreating secondary index entries when neither the secondary key nor the value changes during an update.) 3) It invokes `UpdatePrimaryColumnValue` for all applicable `SecondaryIndex` objects, that is, those for which the primary column family matches the column family from the `PutEntity` call and for which the primary column appears in the new wide-column structure. 4) It writes the new primary key-value. Note that the values of the indexing columns might have been changed in step facebook#3 above. 5) It builds the secondary key-value for each applicable secondary index using `GetSecondaryKeyPrefix` and `GetSecondaryValue`, and writes it to the appropriate secondary column family. All the above operations are performed as part of the same transaction. The logic uses `SavePoint`s to roll back any earlier operations related to a primary key if a subsequent step fails. Implementation-wise, the code uses a mixin template `SecondaryIndexMixin` that can inherit from any kind of transaction, and use the write APIs of the base class (including its concurrency control mechanisms) to implement the index maintenance logic. The mixin will enable to later extend secondary indices to optimistic or write-prepared/write-unprepared pessimistic transactions as well. Differential Revision: D66672931
|
This pull request was exported from Phabricator. Differential Revision: D66672931 |
ltamasi
force-pushed
the
export-D66672931
branch
from
3cef50b to
cf55f1a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66672931 |
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 4, 2024
…cebook#13180) Summary: The patch adds initial support for secondary indices using write-committed transactions. Currently, only the `PutEntity` API is supported; other APIs like `Put` and `Delete` will be added separately. Applications can set up secondary indices using the new configuration option `TransactionDBOptions::secondary_indices`. When secondary indices are enabled, calling `PutEntity` via a (n explicit or implicit) transaction performs the following steps: 1) Retrieves the current value (if any) of the primary key using `GetEntityForUpdate`. 2) If there is an existing primary key-value, it removes any existing secondary index entries using `SingleDelete`. (Note: as a later optimization, we can avoid removing and recreating secondary index entries when neither the secondary key nor the value changes during an update.) 3) It invokes `UpdatePrimaryColumnValue` for all applicable `SecondaryIndex` objects, that is, those for which the primary column family matches the column family from the `PutEntity` call and for which the primary column appears in the new wide-column structure. 4) It writes the new primary key-value. Note that the values of the indexing columns might have been changed in step 3 above. 5) It builds the secondary key-value for each applicable secondary index using `GetSecondaryKeyPrefix` and `GetSecondaryValue`, and writes it to the appropriate secondary column family. All the above operations are performed as part of the same transaction. The logic uses `SavePoint`s to roll back any earlier operations related to a primary key if a subsequent step fails. Implementation-wise, the code uses a mixin template `SecondaryIndexMixin` that can inherit from any kind of transaction, and use the write APIs of the base class (including its concurrency control mechanisms) to implement the index maintenance logic. The mixin will enable to later extend secondary indices to optimistic or write-prepared/write-unprepared pessimistic transactions as well. Differential Revision: D66672931
ltamasi
force-pushed
the
export-D66672931
branch
from
cf55f1a to
25e7856
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66672931 |
ltamasi
force-pushed
the
export-D66672931
branch
from
25e7856 to
8ab7ecb
Compare
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 4, 2024
…cebook#13180) Summary: The patch adds initial support for secondary indices using write-committed transactions. Currently, only the `PutEntity` API is supported; other APIs like `Put` and `Delete` will be added separately. Applications can set up secondary indices using the new configuration option `TransactionDBOptions::secondary_indices`. When secondary indices are enabled, calling `PutEntity` via a (n explicit or implicit) transaction performs the following steps: 1) Retrieves the current value (if any) of the primary key using `GetEntityForUpdate`. 2) If there is an existing primary key-value, it removes any existing secondary index entries using `SingleDelete`. (Note: as a later optimization, we can avoid removing and recreating secondary index entries when neither the secondary key nor the value changes during an update.) 3) It invokes `UpdatePrimaryColumnValue` for all applicable `SecondaryIndex` objects, that is, those for which the primary column family matches the column family from the `PutEntity` call and for which the primary column appears in the new wide-column structure. 4) It writes the new primary key-value. Note that the values of the indexing columns might have been changed in step 3 above. 5) It builds the secondary key-value for each applicable secondary index using `GetSecondaryKeyPrefix` and `GetSecondaryValue`, and writes it to the appropriate secondary column family. All the above operations are performed as part of the same transaction. The logic uses `SavePoint`s to roll back any earlier operations related to a primary key if a subsequent step fails. Implementation-wise, the code uses a mixin template `SecondaryIndexMixin` that can inherit from any kind of transaction, and use the write APIs of the base class (including its concurrency control mechanisms) to implement the index maintenance logic. The mixin will enable to later extend secondary indices to optimistic or write-prepared/write-unprepared pessimistic transactions as well. Differential Revision: D66672931
|
This pull request was exported from Phabricator. Differential Revision: D66672931 |
jowlyzhang
reviewed
Dec 5, 2024
|
Thanks so much for the detailed review @jowlyzhang ! |
…cebook#13180) Summary: The patch adds initial support for secondary indices using write-committed transactions. Currently, only the `PutEntity` API is supported; other APIs like `Put` and `Delete` will be added separately. Applications can set up secondary indices using the new configuration option `TransactionDBOptions::secondary_indices`. When secondary indices are enabled, calling `PutEntity` via a (n explicit or implicit) transaction performs the following steps: 1) It retrieves the current value (if any) of the primary key using `GetEntityForUpdate`. 2) If there is an existing primary key-value, it removes any existing secondary index entries using `SingleDelete`. (Note: as a later optimization, we can avoid removing and recreating secondary index entries when neither the secondary key nor the value changes during an update.) 3) It invokes `UpdatePrimaryColumnValue` for all applicable `SecondaryIndex` objects, that is, those for which the primary column family matches the column family from the `PutEntity` call and for which the primary column appears in the new wide-column structure. 4) It writes the new primary key-value. Note that the values of the indexing columns might have been changed in step 3 above. 5) It builds the secondary key-value for each applicable secondary index using `GetSecondaryKeyPrefix` and `GetSecondaryValue`, and writes it to the appropriate secondary column family. All the above operations are performed as part of the same transaction. The logic uses `SavePoint`s to roll back any earlier operations related to a primary key if a subsequent step fails. Implementation-wise, the code uses a mixin template `SecondaryIndexMixin` that can inherit from any kind of transaction and use the write APIs and concurrency control mechanisms of the base class to implement the index maintenance logic. The mixin will enable us to later extend secondary indices to optimistic or write-prepared/write-unprepared pessimistic transactions as well. Differential Revision: D66672931
ltamasi
force-pushed
the
export-D66672931
branch
from
8ab7ecb to
c006c0e
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66672931 |
|
This pull request has been merged in 1f96e65. |
|
Is there a plan for the requested optimization of WAL size (only include in WAL writes that aren't implied by others)? @ltamasi |
I wasn't aware of this request but it sounds like a Good Idea (TM) |
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 10, 2025
Summary: The patch adds support for `Put` / `PutUntracked` to the secondary indexing logic. Similarly to `PutEntity` (see facebook#13180), calling these APIs automatically add or remove secondary index entries as needed in an atomic and transparent fashion. Differential Revision: D68035089
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 10, 2025
Summary: The patch adds support for `Put` / `PutUntracked` to the secondary indexing logic. Similarly to `PutEntity` (see facebook#13180), calling these APIs automatically add or remove secondary index entries as needed in an atomic and transparent fashion. Differential Revision: D68035089
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 10, 2025
Summary: The patch adds support for `Put` / `PutUntracked` to the secondary indexing logic. Similarly to `PutEntity` (see facebook#13180), calling these APIs automatically add or remove secondary index entries as needed in an atomic and transparent fashion. Differential Revision: D68035089
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 10, 2025
Summary: The patch adds support for `Put` / `PutUntracked` to the secondary indexing logic. Similarly to `PutEntity` (see facebook#13180), calling these APIs automatically add or remove secondary index entries as needed in an atomic and transparent fashion. Differential Revision: D68035089
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 14, 2025
Summary: The patch adds support for `Put` / `PutUntracked` to the secondary indexing logic. Similarly to `PutEntity` (see facebook#13180), calling these APIs automatically add or remove secondary index entries as needed in an atomic and transparent fashion. Differential Revision: D68035089
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 14, 2025
Summary: The patch adds support for `Put` / `PutUntracked` to the secondary indexing logic. Similarly to `PutEntity` (see facebook#13180), calling these APIs automatically add or remove secondary index entries as needed in an atomic and transparent fashion. Reviewed By: jaykorean Differential Revision: D68035089
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 14, 2025
Summary: The patch adds support for `Put` / `PutUntracked` to the secondary indexing logic. Similarly to `PutEntity` (see facebook#13180), calling these APIs automatically add or remove secondary index entries as needed in an atomic and transparent fashion. Reviewed By: jaykorean Differential Revision: D68035089
facebook-github-bot
pushed a commit
that referenced
this pull request
Jan 14, 2025
Summary: Pull Request resolved: #13289 The patch adds support for `Put` / `PutUntracked` to the secondary indexing logic. Similarly to `PutEntity` (see #13180), calling these APIs automatically add or remove secondary index entries as needed in an atomic and transparent fashion. Reviewed By: jaykorean Differential Revision: D68035089 fbshipit-source-id: db37bce62151ae1909b46b1020592c8348156653
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
The patch adds initial support for secondary indices using write-committed transactions. Currently, only the
PutEntityAPI is supported; other APIs likePutandDeletewill be added separately. Applications can set up secondary indices using the new configuration optionTransactionDBOptions::secondary_indices. When secondary indices are enabled, callingPutEntityvia a (n explicit or implicit) transaction performs the following steps:GetEntityForUpdate.SingleDelete. (Note: as a later optimization, we can avoid removing and recreating secondary index entries when neither the secondary key nor the value changes during an update.)UpdatePrimaryColumnValuefor all applicableSecondaryIndexobjects, that is, those for which the primary column family matches the column family from thePutEntitycall and for which the primary column appears in the new wide-column structure.GetSecondaryKeyPrefixandGetSecondaryValue, and writes it to the appropriate secondary column family.All the above operations are performed as part of the same transaction. The logic uses
SavePoints to roll back any earlier operations related to a primary key if a subsequent step fails.Implementation-wise, the code uses a mixin template
SecondaryIndexMixinthat can inherit from any kind of transaction, and use the write APIs of the base class (including its concurrency control mechanisms) to implement the index maintenance logic. The mixin will enable to later extend secondary indices to optimistic or write-prepared/write-unprepared pessimistic transactions as well.Differential Revision: D66672931