Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a cagg config parameter to override the instance-level timescaledb.enable_tiered_reads GUC #7587

Merged
merged 1 commit into from
Jan 15, 2025
Merged

Add a cagg config parameter to override the instance-level timescaledb.enable_tiered_reads GUC #7587

merged 1 commit into from
Jan 15, 2025

Conversation

zilder
Copy link
Contributor

@zilder zilder commented Jan 10, 2025 •
edited
Loading

This PR doesn't contain any API changes yet (to be discussed).

Refresh policy now includes or excludes tiered data based on policy's include_tiered_data config parameter regardless of the timescaledb.enable_tiered_reads GUC:

{
    "end_offset": "1 day",
    "start_offset": "6 months",
    "mat_hypertable_id": 43,
    "enable_tiered_reads": true
}

EDIT:

Added the API change too:

SELECT add_continuous_aggregate_policy(
	'ht_try_weekly',
	start_offset => '6 months'::interval,
	end_offset => '1 hour'::interval,
	schedule_interval => '1 hour'::interval,
	include_tiered_data => true
);

@zilder zilder force-pushed the zilder/cagg_policy_tiered_data branch 8 times, most recently from e507251 to 97f4dd5 Compare January 14, 2025 16:47
@zilder zilder marked this pull request as ready for review January 14, 2025 17:39
gayyappan
gayyappan reviewed Jan 14, 2025
View reviewed changes
tsl/test/sql/chunk_utils_internal.sql Outdated
@@ -440,6 +440,57 @@ CALL refresh_continuous_aggregate('ht_try_weekly', '2019-12-29', '2020-01-10', f
SELECT * FROM ht_try_weekly;
DROP MATERIALIZED VIEW ht_try_weekly;

\c :TEST_DBNAME :ROLE_SUPERUSER
Copy link
Contributor

@gayyappan gayyappan Jan 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do need this mapping? It is already setup for one of the test ROLES.
Does anything fail without connecting as superuser?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was a leftover from the previous iteration. I removed it.

tsl/test/sql/chunk_utils_internal.sql Outdated Show resolved Hide resolved
tsl/test/sql/chunk_utils_internal.sql Outdated Show resolved Hide resolved
gayyappan
gayyappan reviewed Jan 14, 2025
View reviewed changes
.unreleased/pr_7587 Outdated Show resolved Hide resolved
@philkra philkra added this to the TimescaleDB 2.18.0 milestone Jan 15, 2025
@zilder zilder force-pushed the zilder/cagg_policy_tiered_data branch from 97f4dd5 to 5eeb4f8 Compare January 15, 2025 10:34
@zilder
Add include_tiered_data option to cagg policy
94deb0a 
The `include_tiered_data` option allows user to override the value of
`timescaledb.enable_tiered_reads` defined on instance level for a
particular continuous aggregate policy.
@zilder zilder force-pushed the zilder/cagg_policy_tiered_data branch from 5eeb4f8 to 94deb0a Compare January 15, 2025 10:38
@zilder zilder requested review from gayyappan and mkindahl January 15, 2025 10:40
mkindahl
mkindahl approved these changes Jan 15, 2025
View reviewed changes
Copy link
Contributor

@mkindahl mkindahl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

tsl/src/bgw_policy/continuous_aggregate_api.c Show resolved Hide resolved
@zilder zilder merged commit 60ecd46 into main Jan 15, 2025
51 checks passed
@zilder zilder deleted the zilder/cagg_policy_tiered_data branch January 15, 2025 14:29
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Jan 16, 2025
@pallavisontakke
## 2.18.0
33a6e48 
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901: Add hypertable support for transition tables.
* timescale#7104: Hypercore table access method.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7412: Add GUC for `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7433 Add support for merging chunks
* timescale#7436 Add index creation on orderby columns
* timescale#7443: Add hypercore function and view aliases.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7458: Support vecorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7486: Prevent building against postgres versions with broken ABI.
* timescale#7521 Add optional `force` argument to `refresh_continuous_aggregate`
* timescale#7528 Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565 Add hint when hypertable creation fails
* timescale#7587 Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fixes the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517 Fixes performance regression on `cagg_migrate` procedure
* timescale#7527 Restart scheduler on error
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566 Improve transaction check in CAgg refresh
* timescale#7584 Fix NaN-handling for vectorized aggregation

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @k-rus for suggesting the improvement
* @pgloader for reporting the issue in an internal background job.
* @staticlibs for sending PR to improve transaction check in CAgg refresh
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
@pallavisontakke pallavisontakke mentioned this pull request Jan 16, 2025
Open
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Jan 17, 2025
@pallavisontakke
## 2.18.0
ed7c653 
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901: Add hypertable support for transition tables.
* timescale#7104: Hypercore table access method.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7412: Add GUC for `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7433 Add support for merging chunks
* timescale#7436 Add index creation on orderby columns
* timescale#7443: Add hypercore function and view aliases.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7458: Support vecorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7486: Prevent building against postgres versions with broken ABI.
* timescale#7521 Add optional `force` argument to `refresh_continuous_aggregate`
* timescale#7528 Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565 Add hint when hypertable creation fails
* timescale#7587 Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fixes the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517 Fixes performance regression on `cagg_migrate` procedure
* timescale#7527 Restart scheduler on error
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566 Improve transaction check in CAgg refresh
* timescale#7584 Fix NaN-handling for vectorized aggregation

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @k-rus for suggesting the improvement
* @pgloader for reporting the issue in an internal background job.
* @staticlibs for sending PR to improve transaction check in CAgg refresh
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
@pallavisontakke pallavisontakke mentioned this pull request Jan 17, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mkindahl mkindahl mkindahl approved these changes

@fabriziomello fabriziomello fabriziomello approved these changes

@gayyappan gayyappan Awaiting requested review from gayyappan

Assignees

@zilder zilder

Labels
None yet
Projects
None yet
Milestone
TimescaleDB 2.18.0
Development

Successfully merging this pull request may close these issues.
5 participants
@zilder @fabriziomello @mkindahl @gayyappan @philkra