Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

alter_table: Support adding columns to tables #30470

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

ParkMyCar
Copy link
Member

@ParkMyCar ParkMyCar commented Nov 13, 2024

This PR implements the SQL feature ALTER TABLE ... ADD COLUMN ....

Note: There are a lot of lines changed but the majority are new tests!

Specifically it:

  1. Uses VersionedRelationDesc on Tables to track new columns
  2. Adds a CatalogCollectionEntry which adds some typing around getting the current RelationDesc for an entry.
  3. Updates the storage-controller to create new Persist WriteHandles and pass them to the TxnsTableWorker
  4. Updates the storage controller's user of Persist's CriticalSinceHandle to open one per-version of a table. This proved necessary to get the proper read handles for Mat Views on top of tables.

Otherwise it also adds several tests:

  1. alter-table.slt which exercises a number of different scenarios
  2. A new Check for the platform-checks test framework
  3. A new Action for the parallel-workload test framework.
    • This new action is currently disabled because off a race condition in Persist's schema registry
  4. A legacy upgrade test to make sure we have coverage on the restart of MZ test case.

Motivation

Fixes https://github.com/MaterializeInc/database-issues/issues/8233

Tips for reviewer

I split the PR up into separate commits to ideally make it easier to review, most the changes here are new tests!

  1. Changes to the Catalog APIs and name resolution to support versions.
  2. Changes to sequencing and the storage controller. @bkirwi I would appreciate your eyes on this one!
  3. Tests
  4. Formatting and clippy

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

@ParkMyCar ParkMyCar force-pushed the alter_table2/support-adding-columns-to-tables branch 6 times, most recently from b1ba847 to 6edd439 Compare November 19, 2024 18:13
@ParkMyCar ParkMyCar marked this pull request as ready for review November 19, 2024 18:47
@ParkMyCar ParkMyCar requested review from a team as code owners November 19, 2024 18:47
@ParkMyCar ParkMyCar requested review from jkosh44 and bkirwi November 19, 2024 18:47
Copy link

shepherdlybot bot commented Nov 19, 2024

Risk Score:80 / 100 Bug Hotspots:9 Resilience Coverage:66%

Mitigations

Completing required mitigations increases Resilience Coverage.

  • (Required) Code Review 🔍 Detected
  • (Required) Feature Flag
  • (Required) Integration Test 🔍 Detected
  • (Required) Observability 🔍 Detected
  • (Required) QA Review 🔍 Detected
  • (Required) Run Nightly Tests
  • Unit Test
Risk Summary:

The risk score for this pull request is high at 80, driven by predictors such as the sum of bug reports of files and the delta of executable lines. Historically, pull requests with these predictors are 114% more likely to cause a bug compared to the repository baseline. Additionally, the repository's observed and predicted bug trends are both decreasing, which is a positive sign.

Note: The risk score is not based on semantic analysis but on historical predictors of bug occurrence in the repository. The attributes above were deemed the strongest predictors based on that history. Predictors and the score may change as the PR evolves in code, time, and review activity.

Bug Hotspots:
What's This?

File Percentile
../catalog/apply.rs 98
../src/main.rs 97
../catalog/state.rs 91
../src/coord.rs 100
../src/catalog.rs 98
../src/names.rs 92
../catalog/open.rs 99
../src/lib.rs 95
../catalog/transact.rs 93

Copy link
Contributor

@def- def- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I very much appreciate all the tests!

What happens if you call ALTER TABLE ADD COLUMN on a continual task/table from source/mv/...? What if you try to add a column with the name of the table? Can you keep adding columns or do things get slower with number of columns? Could add some tests checking for correct errors in the SLT.

Copy link
Contributor

@jkosh44 jkosh44 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adapter code LGTM

Comment on lines +1381 to +1451
.map(|id| self.get_entry_by_global_id(id))
.filter_map(|entry| entry.index().map(|index| index.on));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I might be missing something, what was the change here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was just some Rust lifetime shenanigans. .index() returns an Option<&Index> but .get_entry_by_global_id(...) returns an owned type that only lives for the duration of the .map(...) call

Comment on lines -178 to -183
storage_collection_metadata: TableTransaction::new_with_uniqueness_fn(
storage_collection_metadata,
|a: &StorageCollectionMetadataValue, b| a.shard == b.shard,
)?,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just confirming, now we can have multiple global IDs for the same object that all point to the same shard?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly

let item_id = self
.entry_by_global_id
.get(id)
.unwrap_or_else(|| panic!("catalog out of sync, missing id {id:?}"));
self.get_entry(item_id)

let entry = self.get_entry(item_id).clone();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels a little bad to clone the entry in this function. This used to be pretty cheap but now involves cloning potentially large expressions and create sql statements.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally agree, it's a bit tricky with Rust lifetimes and the trait CatalogItem, I'll circle back and see if I can improve this though. There might be a Cow<...> like thing we can do

Comment on lines +419 to +423
impl From<RelationVersion> for SchemaId {
fn from(value: RelationVersion) -> Self {
SchemaId(usize::cast_from(value.0))
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I understand this, what's the correlation b/w a relation version and a schema version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now RelationVersions are 1:1 with SchemaIds. At some point we can break this relationship and store the mapping somewhere in the Catalog, but it's not necessary at the moment.

Comment on lines +806 to +807
fn latest_version(&self) -> Option<RelationVersion> {
self.entry.latest_version()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised that this returns an Option, when would an entry ever not have a version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An entry only has a version if it's version-able, i.e. only Tables will return Some here.

src/catalog/src/memory/objects.rs Outdated Show resolved Hide resolved
Comment on lines +276 to +284
let is_versioned = c
.options
.iter()
.any(|o| matches!(o.option, ColumnOption::Versioned { .. }));
!is_versioned
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we filtering here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment, but it's because of how the names collection is used, I took a note for myself to refactor this entire block

@ParkMyCar ParkMyCar force-pushed the alter_table2/support-adding-columns-to-tables branch from 6edd439 to 02b0137 Compare December 5, 2024 15:13
@ParkMyCar ParkMyCar requested a review from petrosagg December 10, 2024 15:10
@@ -642,14 +649,16 @@ where
// Construct the handle in a separate block to ensure all error paths
// are diverging
let since_handle = {
// If the collection we're openning is versioned, be sure to use a
// different CriticalId so the SinceHandles don't conflict.
let reader_id = match version {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's still not clear to me that we're managing the lifecycle of this handle properly... for example, I think the finalization task will only force-downgrade the handle of the controller-global handle, not these per-version handles. (And at a first pass it's not clear to me how all N critical handles get updated when the write frontier advances...)

@ParkMyCar
Copy link
Member Author

Still iterating a bit, but the most recent commit removes the need for multiple CriticalSinceHandles and implements an approach @petrosagg and I talked about where earlier versions of a table track later versions as dependencies, and through initial testing things seems to workout!

At a high level the implementation is there, but pushed up the commits to run against CI

@ParkMyCar ParkMyCar force-pushed the alter_table2/support-adding-columns-to-tables branch 8 times, most recently from 1b30ffa to 7c3e244 Compare December 18, 2024 20:35
* changes Table to use a VersionedRelationDesc
* adds CatalogCollectionEntry and move desc(...) method to it
* renamed CatalogEntry::desc to CatalogEntry::desc_latest
* implement sequencing in the adapter
* updates to the storage controller
* change CriticalSinceHandles so we can have multiple per-shard
* add a good number of test cases to alter-table.slt
* delete duplicate table_alter.slt
* add a platform-check for Alter Table
* add a (disable) parallel-workload case for Alter Table
* add a legacy upgrade test for Alter Table
* refactors how alter_table_desc is implemented so the storage-controller tracks a dependency between different versions of the tables
* removes the API for creating a CriticalId from a 'seed'
* update bootstraping storage collections to properly order Tables
* fix upgrade tests by removing reference to old persist dyncfg
* change AstDisplay for ResolvedItemName to not print the version
* remove commented our impl of previous sorting
* update legacy upgrade test
* a few of the conditions we were testing fail when --auto-index-selects was enabled
* when creating collections for bootstrap, re-order them like we do in storage-collections
* in the storage-controller report the correct dependencies for tables
* in the Coordinator register ReadPolicies
* in the storage collections install read holds with the existing collections read frontier, not the implied capability
* in storage collections set the write frontier of the new collection to the write frontier of the existing collection
@ParkMyCar ParkMyCar force-pushed the alter_table2/support-adding-columns-to-tables branch from 8e3af36 to 219f937 Compare January 2, 2025 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants