Intra-row secondary indexing

[my-pal-jason]

Use cases that leverage nested child rows may be able to benefit from a simpler index implementation that has better atomicity guarantees and better performance. If the row is identified through another index (whether in Phoenix or via another mechanism, such as a lucene-based search), the data stored in the columns of the row can be indexed by other columns in the row.

Below is a simple example implementation. Given row:

{salt}person_id:123

You could have columns:

{leading byte for index 1}attribute_1:value{delimiter}event:123 -> (no value)
{leading byte for index 2}attribute_2:value{delimiter}event:123 -> (no value)
{leading byte for data}event:123 -> {nested data with attributes 1 & 2}

Given the structure above, as long as we know the person we are working with we benefit from a few things.

1. We can leverage row-level atomicity for index updates.
2. When data is written to the row, the index can be updated very efficiently, as its in the same region.
3. When reading from the row, the index can be used very efficiently. A filter determines which index(es) to hit based on the query conditions, then skips "down" to the data for the read. Though the common case would only require a single scan, multiple scans would be possible in the event of a query that joins back to itself or must use multiple indexes in an order different than they are written to the row.

[saintstack]

FB Messges does something similar keeping all indices and metadata for a particular user all in the one 'row': http://www.facebook.com/notes/facebook-engineering/inside-facebook-messages-application-server/10150162742108920