Update Row-Level TTL docs to note full support #15358

rmloveland · 2022-10-13T17:23:34Z

Fixes DOC-4756

Summary of changes:

Update 'Batch Delete Data with Row-Level TTL' page to note:
- You can now do per-row TTL using ttl_expiration_expression, so we mention that and give examples, describe how it's validated
- Remove at least the resolved limitations from the TTL docs and the 'Known Limitations' page
- Remove uses of ttl_automatic_column which is no longer used in v22.2+
Update 'WITH storage_parameter' page to use new output / TTL params
Update 'RESET storage_parameter' page to use new output / TTL params
Update included table of storage parameters to remove duplication of TTL params, instead linking to the canonical location on the TTL docs page

cockroach-teamcity · 2022-10-13T17:23:41Z

This change is

github-actions · 2022-10-13T17:24:00Z

Files changed:

_includes/v22.2/known-limitations/row-level-ttl-limitations.md:

_includes/v22.2/misc/table-storage-parameters.md:

_includes/v22.2/sidebar-data/reference.json
_includes/v22.2/sql/row-level-ttl-prefer-ttl-expiration-expressions.md:

v22.2/row-level-ttl.md

_includes/v22.2/sql/row-level-ttl.md:

images/v22.2/ttl-estimated-rows.png:

v22.2/ui-ttl-dashboard.md

images/v22.2/ttl-job-latency.png:

v22.2/ui-ttl-dashboard.md

images/v22.2/ttl-processing-rate.png:

v22.2/ui-ttl-dashboard.md

images/v22.2/ttl-ranges-in-progress.png:

v22.2/ui-ttl-dashboard.md

v22.2/reset-storage-parameter.md
v22.2/row-level-ttl.md
v22.2/ui-overview.md
v22.2/ui-ttl-dashboard.md
v22.2/with-storage-parameter.md

netlify · 2022-10-13T17:28:29Z

✅ Netlify Preview

Name	Link
🔨 Latest commit	`5529bc1`
🔍 Latest deploy log	https://app.netlify.com/sites/cockroachdb-docs/deploys/637276c6450deb00084c37b7
😎 Deploy Preview	https://deploy-preview-15358--cockroachdb-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

rmloveland · 2022-10-14T16:17:35Z

@rafiss added you as reviewer alongside @ecwall b/c of cockroachdb/cockroach#86269

let me know if you would rather not be added tho!

Pretty sure I caught most of the changes from reading PRs and playing with the DB but ... entirely possible I missed something. Please let me know if so. Also happy to make any other updates re: clarity, etc. Thanks!

rafiss · 2022-10-14T20:08:56Z

thanks @rmloveland ! two points before doing a deeper review:

@ecwall : are you planning to separate out the metrics name change in ttl: Remove ttl_range_concurrency config cockroach#89392 and backport that to v22.2.0? (I think we should.) If so, then we can document that here too.
I noticed that we don't document the TTL metrics dashboard in the DB Console currently. I think we should add docs for that now.

ecwall · 2022-10-14T20:27:45Z

thanks @rmloveland ! two points before doing a deeper review:

@ecwall : are you planning to separate out the metrics name change in ttl: Remove ttl_range_concurrency config cockroach#89392 and backport that to v22.2.0? (I think we should.) If so, then we can document that here too.

I noticed that we don't document the TTL metrics dashboard in the DB Console currently. I think we should add docs for that now.

Yeah it looks like some of the metrics are not working properly in the db console so I’m fixing that also.

rmloveland · 2022-10-18T20:08:26Z

Rafi Shamim ***@***.***> writes:

thanks @rmloveland ! two points before doing a deeper review:

thanks Rafi! Took a shot at addressing these points

* @ecwall : are you planning to separate out the metrics name change in cockroachdb/cockroach#89392 and backport that to v22.2.0? (I think we should.) If so, then we can document that here too.

Updated in latest push to remove the TTL range concurrency thing

* I noticed that we don't document the TTL metrics dashboard in the DB Console currently. I think we should add docs for that now.

Ah ok, missed that. Added a DB Console page on TTL dashboard!

rmloveland · 2022-10-18T20:14:50Z

Evan Wall ***@***.***> writes:

Yeah it looks like some of the metrics are not working properly in the db console so I’m fixing that also.

OK that explains why some of the charts are blank in this docs PR despite me doing a bunch of TTL stuff to the DB. (re: the blank charts, i'd like to update the screenshots in another PR - I'm under some time pressure to get a bunch of other things done before the v22.2.0 release next month) but with the other updates I just made this is RFAL now

ecwall · 2022-10-26T16:56:07Z

_includes/v22.2/known-limitations/row-level-ttl-limitations.md

- <a name="ttl-cannot-be-customized"></a> The TTL cannot be customized based on the values of other columns in the row. [cockroachdb/cockroach#76916](https://github.com/cockroachdb/cockroach/issues/76916)
-  - Because of the above limitation, adding TTL to large existing tables [can negatively affect performance](row-level-ttl.html#ttl-existing-table-performance-note), since a new column must be created and backfilled for every row. Creating a new table with a TTL is not affected by this limitation.
- The queries executed by Row-Level TTL are not yet optimized for performance:
-  - They do not use any indexes that may be available on the [`crdb_internal_expiration` column](row-level-ttl.html#crdb-internal-expiration).


This is still a limitation, but I would expand it to say that it does not use any indexes (not just indexes on crdb_internal_expiration).

Updated - PTAL!

ecwall · 2022-10-26T16:57:11Z

_includes/v22.2/misc/table-storage-parameters.md

-|---------------------+----------------------+-----+------|
-| `exclude_data_from_backup` | Excludes the data in this table from any future backups. | Boolean | `false` |
-| `ttl` | Signifies if a TTL is active. Automatically set and controls the reset of all TTL-related storage parameters. | N/A | N/A |
-| `ttl_automatic_column` | If set, use the value of the `crdb_internal_expiration` hidden column. Always set to `true` and cannot be reset. | Boolean | `true` |


All of these are still used except for ttl_automatic_column.

Yes definitely! All of the TTL params are now moved to row-level-ttl.md (line 128, the TTL Storage Parameters section) so all the TTL things are documented in one place

ecwall · 2022-10-26T16:57:31Z

_includes/v22.2/misc/table-storage-parameters.md

-| `ttl_job_cron` | The frequency at which the TTL job runs. | [CRON syntax](https://cron.help) | `'@hourly'` |
-| `ttl_label_metrics` | Whether or not [TTL metrics](row-level-ttl.html#ttl-metrics) are labelled by table name (at the risk of added cardinality). | Boolean | `false` |
-| `ttl_pause` | If set, stops the TTL job from executing. | Boolean | `false` |
-| `ttl_range_concurrency` | The Row-Level TTL queries split up scans by ranges, and this determines how many concurrent ranges are processed at a time. Minimum: `1`. | Integer | `1` |


This is used in 22.2.0, but not 22.2.1+.

I looked and ttl_range_concurrency is not in the list of params on row-level-ttl.md (line 128, the TTL Storage Parameters section)

ecwall · 2022-10-26T17:02:11Z

v22.2/row-level-ttl.md

+- [Per-table TTL](#per-table-ttl)
+- [Per-row TTL](#per-row-ttl)
+
+### Per-table TTL


The difference between ttl_expire_after and ttl_expiration_expression isn't per-table vs per-row. Both of them set TTL per-row in the table.

ttl_expire_after is a convenient way of setting rows to expire a fixed amount of time after they are created or updated.

ttl_expiration_expression lets you customize the expiration logic further by providing an expression. (You could get the same behavior as ttl_expire_after by creating a column with a default value and having the ttl_expiration_expression reference that column.)

Additionally, the 2 storage params can co-exist where the ttl_expire_after creates the crdb_internal_expiration column and ttl_expiration_expression overrides the default value of crdb_internal_expiration.

Thanks for clarifying this! I have updated these sections of the doc to add this info - PTAL

ecwall · 2022-10-26T17:06:36Z

v22.2/row-level-ttl.md

---------------------------------------+-------------+----------------------------+--------------------------------
-  a9404386-c4da-415f-b0b0-0dfad0f13c80 | a thing     | 2022-04-19 19:53:00.037216 | 2023-04-19 19:57:27.592898+00
-(1 row)
+ALTER TABLE events RESET (ttl);


You can also do ALTER TABLE events RESET (ttl_expire_after); or ALTER TABLE events RESET (ttl_expiration_expression); if both are set and you want to remove one or the other.

Added this info - PTAL

rmloveland · 2022-11-10T18:19:10Z

per convo with @dikshant : I will update this PR to state that ttl_expiration_expression is preferred over ttl_expire_after starting with v22.2 for the following reasons:

if you add ttl_expire_after it will cause a full table rewrite. you can't use this with an existing timestamp column
whereas ttl_expiration_expression can say "please expire based on this updated_at col that already exists" etc

Fixes: - DOC-4664 - DOC-4672 - DOC-4756 - DOC-4881 - DOC-5270 - DOC-5983 - DOC-5995 Summary of changes: - Update 'Batch Delete Data with Row-Level TTL' page to note: - You can now do per-row TTL using `ttl_expiration_expression`, so we mention that and give examples, describe how it's validated - Remove at least the resolved limitations from the TTL docs and the 'Known Limitations' page - Remove uses of `ttl_automatic_column' which is no longer used in v22.2+ - Remove `ttl_range_concurrency`, which is no longer available in v22.2+ per cockroachdb/cockroach#89392 - Add note on migrating from versions < v22.2 - Add recommendation that most users use `ttl_expiration_expression` over `ttl_expire_after` - Including moving sections of the page to put the expression info first - Add page for TTL Dashboard in DB Console docs - Update 'WITH storage_parameter' page to use new output / TTL params - Update 'RESET storage_parameter' page to use new output / TTL params - Update included table of storage parameters to remove duplication of TTL params, instead linking to the canonical location on the TTL docs page

rmloveland · 2022-11-14T18:59:16Z

@rafiss and @ecwall this has been updated with your feedback and the comment from @dikshant and should be RFAL

rafiss

nice, lgtm!

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ecwall)

rmloveland force-pushed the 20221011-full-support-row-level-ttl branch from 8cfdd63 to 3efa2c8 Compare October 13, 2022 19:50

rmloveland requested review from ecwall and rafiss October 14, 2022 16:15

rmloveland force-pushed the 20221011-full-support-row-level-ttl branch from 3efa2c8 to 1931aa8 Compare October 18, 2022 20:06

ecwall reviewed Oct 26, 2022

View reviewed changes

rafiss requested a review from ecwall November 7, 2022 21:17

rmloveland force-pushed the 20221011-full-support-row-level-ttl branch from 1931aa8 to d295707 Compare November 8, 2022 20:16

rmloveland force-pushed the 20221011-full-support-row-level-ttl branch from d295707 to 5529bc1 Compare November 14, 2022 17:11

rafiss approved these changes Nov 21, 2022

View reviewed changes

rmloveland requested a review from ianjevans November 21, 2022 18:44

ianjevans approved these changes Nov 28, 2022

View reviewed changes

rmloveland merged commit fd0b52c into master Nov 28, 2022

rmloveland deleted the 20221011-full-support-row-level-ttl branch November 28, 2022 20:59

This was referenced Nov 28, 2022

sql: support ttl_expiration_expression for row-level TTL #14406

Closed

sql: remove ttl_automatic_column storage parameter #14465

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Row-Level TTL docs to note full support #15358

Update Row-Level TTL docs to note full support #15358

rmloveland commented Oct 13, 2022

cockroach-teamcity commented Oct 13, 2022

github-actions bot commented Oct 13, 2022 •

edited

Loading

netlify bot commented Oct 13, 2022 •

edited

Loading

rmloveland commented Oct 14, 2022

rafiss commented Oct 14, 2022

ecwall commented Oct 14, 2022

rmloveland commented Oct 18, 2022 via email

rmloveland commented Oct 18, 2022 via email

ecwall Oct 26, 2022

rmloveland Nov 8, 2022

ecwall Oct 26, 2022

rmloveland Nov 8, 2022

ecwall Oct 26, 2022

rmloveland Nov 8, 2022

ecwall Oct 26, 2022

rmloveland Nov 8, 2022

ecwall Oct 26, 2022

rmloveland Nov 8, 2022

rmloveland commented Nov 10, 2022

rmloveland commented Nov 14, 2022

rafiss left a comment

Update Row-Level TTL docs to note full support #15358

Update Row-Level TTL docs to note full support #15358

Conversation

rmloveland commented Oct 13, 2022

cockroach-teamcity commented Oct 13, 2022

github-actions bot commented Oct 13, 2022 • edited Loading

Files changed:

netlify bot commented Oct 13, 2022 • edited Loading

✅ Netlify Preview

rmloveland commented Oct 14, 2022

rafiss commented Oct 14, 2022

ecwall commented Oct 14, 2022

rmloveland commented Oct 18, 2022 via email

rmloveland commented Oct 18, 2022 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmloveland commented Nov 10, 2022

rmloveland commented Nov 14, 2022

rafiss left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 13, 2022 •

edited

Loading

netlify bot commented Oct 13, 2022 •

edited

Loading