PROD-2734 - initial partitioning support (user-defined windows) #5325

adamsachs · 2024-09-24T23:10:24Z

Closes PROD-2734

Description Of Changes

Depends on Initial support for DatasetCollection partitioning specifications fideslang#21
Accepts array of strings in a fides_meta.partitioning.where_clauses (maybe there's a better name?) field on the DatasetCollection config, which are (for BQ DSR execution) injected into access and erasure queries
if configured on a BQ dataset collection, DSR execution now executes one query per specified where_clauses string per node per DSR request, rather that just one query per node per DSR request.
validation of where_clauses input to prevent SQL injection

Configuration is within fides_meta of a DatasetCollection, e.g.:

dataset:
  - fides_key: bigquery_example_test_dataset_with_partitioning
    name: BigQuery Example Test Dataset Demonstrating Partitioning
    description: Example of a BigQuery dataset containing a variety of related tables like customers, products, addresses, etc.
    collections:
      - name: customer_partitioned
        fides_meta:
          partitioning:
            where_clauses: [
                "`created` > DATETIME_SUB(CURRENT_DATETIME(), INTERVAL 1000 DAY) AND `created` <= CURRENT_DATETIME()",
                "`created` > TIMESTAMP_SUB(CURRENT_DATETIME(), INTERVAL 2000 DAY) AND `created` <= TIMESTAMP_SUB(CURRENT_DATETIME(), INTERVAL 1000 DAY)"
            ]
        fields:
       ...

note that where_clauses values must strictly match an expected format/structure:

"`column_name` >(=) [value expression] (AND `column_1` <(=) [value expression])"

or

"`column_name` <(=) [value expression] (AND `column_1` >(=) [value expression])"

Code Changes

update dataset <--> request graph (de)serialization to correctly extract partitioning metadata from dataset config
perform rigid validation of where_clauses values in the partitioning metadata to prevent malicious string injection
ensure partitioning can only be used with BigQuery DSR execution for now, as this is still an experimental feature
provide a hook in generic retrieve_data to call a partitioned_retrieval method if the collection has partitioning defined (and the connector supports partitioning, i.e. only BigQuery for now)
implement the partitioned_retrieval hook in the BigQueryConnector to support executing multiple partitioned queries for DSR access requests against BQ
also update existing BQ-specific methods to support executing multiple partitioned queries for DSR erasure (update/delete) requests against BQ

Steps to Confirm

testing steps here: https://ethyca.atlassian.net/wiki/spaces/EN/pages/3316875290/Testing+BQ+DSR+Partitioning

Pre-Merge Checklist

vercel · 2024-09-24T23:10:29Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		Oct 4, 2024 4:17am

adamsachs

some self review -

adamsachs · 2024-09-24T23:11:05Z

data/dataset/postgres_example_test_dataset.yml

+          partitioning:
+            field: billing_address_id
+            windows:
+              - start: "0"


might be nice to allow proper ints here, but we do also need to support timestamps, and easiest for now is just to accept generic strings...

Not sure if this is where the issue is or downstream from this but I ran into the same thing for the SaaS config. You can do this:

class UserDefinedPartitionWindow(BaseModel): """Defines a user-defined partition window""" start: Union[int, str] end: Union[int, str] start_inclusive: bool = True end_inclusive: bool = True

Pydantic will try to parse as an int first, and if that doesn't work it will treat it as a string

adamsachs · 2024-09-24T23:11:17Z

requirements.txt

@@ -16,7 +16,7 @@ types-defusedxml==0.7.0.20240218
 expandvars==0.9.0
 fastapi[all]==0.111.0
 fastapi-pagination[sqlalchemy]==0.12.25
-fideslang==3.0.4
+fideslang[all] @ git+https://github.com/ethyca/fideslang@64c15da6e742e5c7cecd3b2484d2b66dbb94fce8


TODO: update once ethyca/fideslang#21 is finalized and merged

adamsachs · 2024-09-24T23:12:15Z

src/fides/api/service/connectors/query_config.py

@@ -458,7 +466,7 @@ def get_update_stmt(
        pk_clauses: List[str],
    ) -> str:
        """Returns a SQL UPDATE statement to fit SQL syntax."""
-        return f"UPDATE {self.node.address.collection} SET {', '.join(update_clauses)} WHERE {' AND '.join(pk_clauses)}"
+        return f"UPDATE {self.node.address.collection} SET {', '.join(update_clauses)} WHERE {' AND '.join(pk_clauses + ([PARTITION_CLAUSE_TEMPLATE] if self.partitioning else []))}"


TODO: could be cleaner

adamsachs · 2024-09-24T23:12:23Z

src/fides/api/service/connectors/query_config.py

@@ -476,7 +484,7 @@ def format_key_map_for_update_stmt(self, fields: List[str]) -> List[str]:

    def generate_update_stmt(
        self, row: Row, policy: Policy, request: PrivacyRequest
-    ) -> Optional[T]:
+    ) -> Optional[T] | List[T]:


TODO: probably could be cleaner

adamsachs · 2024-09-24T23:16:19Z

src/fides/api/service/connectors/query_config.py

+            logger.info(
+                "query = {}, params = {}", Pii(query_str), Pii(update_value_map)
+            )
+            return self.format_query_stmt(query_str, update_value_map)


same as before, just moved down here into the else.

adamsachs · 2024-09-24T23:16:31Z

src/fides/api/service/connectors/query_config.py

+            partitioned_queries = []
+            logger.info(
+                f"Generating {len(partition_var_sets)} partition queries for node '{self.node.address}' in DSR execution"
+            )
+            for partition_var_set in partition_var_sets:
+
+                partitioned_update_value_map = update_value_map.copy()
+                partitioned_update_value_map.update(partition_var_set)
+                logger.info(
+                    "query = {}, params = {}",
+                    Pii(query_str),
+                    Pii(partitioned_update_value_map),
+                )
+                partitioned_queries.append(
+                    self.format_query_stmt(query_str, partitioned_update_value_map)
+                )
+            return partitioned_queries


injection of the partitioning vars, a new piece

adamsachs · 2024-09-24T23:16:46Z

src/fides/api/service/connectors/sql_connector.py

+                results = connection.execute(stmt)
+                rows = self.cursor_result_to_rows(results)


same as before

adamsachs · 2024-09-24T23:17:00Z

src/fides/api/service/connectors/sql_connector.py

+                logger.info(
+                    f"Executing {len(partition_var_sets)} partition queries for node '{node.address}' in DSR execution"
+                )
+                for partition_var_set in partition_var_sets:
+                    logger.debug(
+                        f"Executing partition query with start '{partition_var_set['partition_start']}' and end ''{partition_var_set['partition_end']}'"
+                    )
+                    results = connection.execute(stmt, partition_var_set)
+                    rows.extend(self.cursor_result_to_rows(results))


injection of partitioning vars, the new piece

adamsachs · 2024-09-24T23:17:30Z

src/fides/api/service/connectors/sql_connector.py

-        return create_engine(
-            uri,
-            credentials_info=credentials_info,
-            hide_parameters=self.hide_parameters,
-            echo=not self.hide_parameters,
+        return BigQueryClient(
+            credentials=service_account.Credentials.from_service_account_info(
+                credentials_info  # pylint: disable=no-member
+            )


whoops, need to clean up. mistake.

cypress · 2024-09-24T23:27:25Z

fides Run #10306

Run Properties: Passed #10306 • 2c672a21ae ℹ️: Merge 445def52ba7bb05fb87210978ede924c1a72f4cc into eb9ba5ee372b244db58c108ead82...

Project	`fides`
Branch Review	`refs/pull/5325/merge`
Run status	`Passed #10306`
Run duration	`00m 39s`
Commit	`2c672a21ae ℹ️: Merge 445def52ba7bb05fb87210978ede924c1a72f4cc into eb9ba5ee372b244db58c108ead82...`
Committer	`Adam Sachs`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`4`
View all changes introduced in this branch ↗︎

galvana · 2024-09-25T03:39:34Z

data/dataset/postgres_example_test_dataset.yml

+          partitioning:
+            field: billing_address_id
+            windows:
+              - start: "0"


Not sure if this is where the issue is or downstream from this but I ran into the same thing for the SaaS config. You can do this:

class UserDefinedPartitionWindow(BaseModel): """Defines a user-defined partition window""" start: Union[int, str] end: Union[int, str] start_inclusive: bool = True end_inclusive: bool = True

Pydantic will try to parse as an int first, and if that doesn't work it will treat it as a string

galvana · 2024-09-25T03:41:50Z

src/fides/api/service/connectors/query_config.py

@@ -458,7 +466,7 @@ def get_update_stmt(
        pk_clauses: List[str],
    ) -> str:
        """Returns a SQL UPDATE statement to fit SQL syntax."""
-        return f"UPDATE {self.node.address.collection} SET {', '.join(update_clauses)} WHERE {' AND '.join(pk_clauses)}"
+        return f"UPDATE {self.node.address.collection} SET {', '.join(update_clauses)} WHERE {' AND '.join(pk_clauses + ([PARTITION_CLAUSE_TEMPLATE] if self.partitioning else []))}"


Make sure to change self.node.address.collection to self._generate_table_name() once you pull in my change

Suggested change

return f"UPDATE {self.node.address.collection} SET {', '.join(update_clauses)} WHERE {' AND '.join(pk_clauses + ([PARTITION_CLAUSE_TEMPLATE] if self.partitioning else []))}"

return f"UPDATE {self._generate_table_name()} SET {', '.join(update_clauses)} WHERE {' AND '.join(pk_clauses + ([PARTITION_CLAUSE_TEMPLATE] if self.partitioning else []))}"

ah ok, well i've done this on the abstract class, but looks like i missed updating the subclasses that override this method - most importantly, bigquery.

thanks for calling that out!

oh i see, i think this also changed on the delete override PR 👍

probably was a bit more overlap here than anticipated...

galvana · 2024-09-25T03:45:43Z

tests/ops/service/connectors/test_queryconfig.py

+        text_clause = text_clauses[0]
+        assert (
+            text_clause.text
+            == """UPDATE customer SET name = :name WHERE id = :id AND (:partition_field :greater_than_operand :partition_start AND :partition_field :less_than_operand :partition_end)"""


Same as above, let's add another test to make sure we can generate a namespaced version of the table name

adamsachs · 2024-10-01T19:56:38Z

src/fides/api/service/connectors/query_config.py

+
+            return partitioned_queries
+
+        return [table.delete().where(*pk_clauses)]


fall back to existing behavior if no partitioning

adamsachs · 2024-10-01T19:57:04Z

src/fides/api/service/connectors/query_config.py

+
+            return partitioned_queries
+
+        return [table.update().where(*pk_clauses).values(**update_value_map)]


fall back to existing behavior if no partitioning

…ion, for bq only. no validation (yet)

pattisdr

generally this makes sense - we're trying to support querying on a partitioned database with a very narrow scope, bigquery only to start, and the where clause is defined with bigquery syntax in the yaml that is added directly -

Given all that, I'd lean to keeping everything defined on the bigquery connector/bigquery queryconfig for now, as this is a super-specific implementation

pattisdr · 2024-10-02T00:32:54Z

src/fides/api/models/datasetconfig.py

@@ -339,6 +343,7 @@ def convert_dataset_to_graph(
            data_categories=(
                set(collection.data_categories) if collection.data_categories else set()
            ),
+            partitioning=collection_partitioning,


This reminds me, if we end up removing support for the where clauses since it's experimental, we'll have to do it in a way that still allows DSR's in process to run:

DSR 3.0 (which our customers should all be running by now) stores a representation of this collection on the requesttask backing the node in the database, so we don't have to rebuild the graph. If details change about the partitioning, there could still be requesttasks that haven't run in the database with the old partitioning configuration which would fail when they are queued by the worker and the worker goes to rehydrate the collection off of the request task.

thanks for calling that out, great edge case to consider. i don't think there's a whole lot to do right now to anticipate that future state, beyond some code comments to warn the future dev that they'll need to handle that backward compatible case, right? if you have other ideas, would love to hear them to make this as seamless as possible a transition in the (hopefully near) future!

just a callout would be fine!

pattisdr · 2024-10-02T00:33:40Z

src/fides/api/service/connectors/query_config.py

+            logger.warning(
+                "Partitioning is only supported on BigQuery connectors at this time!"
+            )


this seems useful, I could have added that for bigquery row-level deletes as well

pattisdr · 2024-10-02T00:42:40Z

src/fides/api/service/connectors/query_config.py

+    def get_partition_clauses(
+        self,
+    ) -> List[str]:
+        """


Since we for sure only support bigquery at the moment, it felt too early to move this to the generic location, but that's just my opinion

pattisdr · 2024-10-02T00:47:16Z

src/fides/api/service/connectors/query_config.py

+        if self.partitioning:
+            partition_clauses = self.get_partition_clauses()
+            partitioned_queries = []
+            logger.info(
+                f"Generating {len(partition_clauses)} partition queries for node '{self.node.address}' in DSR execution"
+            )
+
+            for partition_clause in partition_clauses:
+                partitioned_queries.append(
+                    table.delete().where(*(pk_clauses + [text(partition_clause)]))
+                )
+
+            return partitioned_queries


might be able to dry some of this up with the update/deletes, just a thought

i hear you and agree, but the codepaths were already a bit duplicated to begin with, and it'll take a bit more effort to pull apart at the moment. i'll mark as a TODO for us to come back to generally DRYing up the two methods in a future iteration 👍

pattisdr · 2024-10-02T00:48:47Z

src/fides/api/service/connectors/sql_connector.py

+            if query_config.partitioning:
+                partition_clauses = query_config.get_partition_clauses()
+                logger.info(
+                    f"Executing {len(partition_clauses)} partition queries for node '{node.address}' in DSR execution"
+                )
+                for partition_clause in partition_clauses:
+                    logger.debug(
+                        f"Executing partition query with partition clause '{partition_clause}'"
+                    )
+                    existing_bind_params = stmt.compile().params
+                    partitioned_stmt = text(
+                        f"{stmt} AND ({text(partition_clause)})"
+                    ).params(existing_bind_params)
+                    results = connection.execute(partitioned_stmt)
+                    rows.extend(self.cursor_result_to_rows(results))


This feels like it shouldn't be here yet while it's only supported for bigquery -

that's fair, but i'd generally prefer not to create parallel implementations for retrieve_data (for maintenance purposes) and i'm not seeing a simple way to refactor this otherwise - but maybe i'm not being creative enough!

pattisdr · 2024-10-02T17:32:30Z

src/fides/api/graph/config.py

+                raise ValueError("`where_clauses` must be a list of strings!")
+            for partition_clause in where_clauses:
+                if matching := match(PARTITION_CLAUSE_PATTERN, partition_clause):
+                    if matching["field_1"] != matching["field_2"]:


~~what is field_1 and field_2? can you add an example?~~

EDIT okay got it, parsing your PARTITION_CLAUSE_PATTERN now

~~what is there is no match then, something didn't fit the pattern and the match was None?~~

EDIT: nvm, that would be picked up in the if matching:

are all partitioning clauses expected to be on the same field as well, or do we just care about the pairs?

~~what is field_1 and field_2? can you add an example?~~

EDIT okay got it, parsing your PARTITION_CLAUSE_PATTERN now

let me still clarify things with some comments so this is easier to understand from a quick glance 👍

are all partitioning clauses expected to be on the same field as well, or do we just care about the pairs?

yeah, i do think we'd expect that, but i'm not sure that checking for it really helps in preventing malicious input. granted, i don't have a super concrete example of the type of malicious input that the current check prevents, but it does at least put some more bounds on each of the clause strings. i wasn't think that checking across strings provides a whole lot of value - but i guess it could help prevent/clarify user error?

just typing out my thoughts...

pattisdr · 2024-10-02T19:00:16Z

src/fides/api/graph/config.py

@@ -443,6 +444,9 @@ class MaskingOverride:
    length: Optional[int]


+PARTITION_CLAUSE_PATTERN = r"`(?P<field_1>[a-zA-Z0-9_]*)` ([<|>][=]?) ([a-zA-Z0-9_\s(),.]*) AND `(?P<field_2>[a-zA-Z0-9_]*)` ([<|>][=]?) ([a-zA-Z0-9_\s(),.]*)$"


is any of this bigquery specific? if so we might note that

yeah good call, i think the back ticks may be BQ-specific, and we're already scoped down to BQ in other respects so let me just make that more explicit 👍

src/fides/api/graph/config.py

requirements.txt

pattisdr · 2024-10-03T19:45:43Z

Reviewing/testing -

pattisdr

test_bigquery_example_data looks like a new test failure? that should be quick to resolve, I think it's just looking at table names

pattisdr · 2024-10-03T20:45:37Z

src/fides/api/graph/config.py

+PROHIBITED_KEYWORDS = [
+    "UNION",
+    "INSERT",
+    "UPDATE",
+    "CREATE",
+    "DROP",
+    "SELECT",
+    "CHAR",
+    "HAVING",
+    "EXEC",
+]


great this helps add confidence

pattisdr · 2024-10-03T20:47:08Z

src/fides/api/graph/config.py

+        To be clear, some notable constraints on the input:
+        - the clause string must begin by referencing a column name wrapped by backticks (`)
+        - the clause string must compare that first column with a `<>(=)` operator, and may
+        include at most one other conditional with a `<>(=)` operator that's joined to the first
+        conditional via an AND operator
+        - if the clause string contains a second conditional, it must reference the same column name
+        as the first conditional, also wrapped by backticks
+        - column names (wrapped by backticks) must always be on the _left_ side of the `<>(=)`operator
+        in its conditional


pattisdr · 2024-10-03T20:49:40Z

src/fides/api/graph/config.py

+        if where_clauses := partitioning.get("where_clauses"):
+            if not isinstance(where_clauses, List) or not all(
+                isinstance(where_clause, str) for where_clause in where_clauses
+            ):
+                raise ValueError("`where_clauses` must be a list of strings!")
+            for partition_clause in where_clauses:
+                if matching := match(
+                    BIGQUERY_PARTITION_CLAUSE_PATTERN, partition_clause
+                ):
+                    # check that if there are two field comparison sub-clauses, they reference the same field, e.g.:
+                    # "`my_field_1` > 5 AND `my_field_1` <= 10", not "`my_field_1` > 5 AND `my_field_1` <= 10"
+                    if matching["field_2"] is not None and (
+                        matching["field_1"] != matching["field_2"]
+                    ):
+                        raise ValueError(
+                            f"Partition clause must have matching fields. Identified non-matching field references '{matching['field_1']}' and '{matching['field_2']}"
+                        )
+
+                    for prohibited_keyword in PROHIBITED_KEYWORDS:
+                        search_str = prohibited_keyword.lower() + r"\s"
+                        if search(search_str, partition_clause.lower()):
+                            raise ValueError(
+                                "Prohibited keyword referenced in partition clause"
+                            )


I think you've done what you can given that we're allowing a where clause to be entered directly here! Lots of good checks.

pattisdr · 2024-10-03T20:52:37Z

src/fides/api/service/connectors/sql_connector.py

+            if (
+                query_config.partitioning
+            ):  # only BigQuery supports partitioning, for now
+                return self.partitioned_retrieval(query_config, connection, stmt)
+


I do like this, thanks

codecov · 2024-10-04T00:06:19Z

Codecov Report

Attention: Patch coverage is 38.09524% with 52 lines in your changes missing coverage. Please review.

Project coverage is 85.31%. Comparing base (eb9ba5e) to head (445def5).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/fides/api/service/connectors/query_config.py	20.00%	27 Missing and 1 partial ⚠️
src/fides/api/service/connectors/sql_connector.py	9.09%	18 Missing and 2 partials ⚠️
src/fides/api/graph/config.py	83.33%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5325      +/-   ##
==========================================
- Coverage   85.44%   85.31%   -0.14%     
==========================================
  Files         378      378              
  Lines       23896    23971      +75     
  Branches     3189     3209      +20     
==========================================
+ Hits        20419    20450      +31     
- Misses       2892     2931      +39     
- Partials      585      590       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

adamsachs · 2024-10-04T04:16:25Z

merging as all CI failures already exist in main 👍

cypress · 2024-10-04T04:32:47Z

fides Run #10305

Run Properties: Passed #10305 • c8f5b1f365: PROD-2734 - initial partitioning support (user-defined windows) (#5325)

Project	`fides`
Branch Review	`main`
Run status	`Passed #10305`
Run duration	`00m 39s`
Commit	`c8f5b1f365: PROD-2734 - initial partitioning support (user-defined windows) (#5325)`
Committer	`Adam Sachs`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`4`
View all changes introduced in this branch ↗︎

adamsachs commented Sep 24, 2024

View reviewed changes

galvana reviewed Sep 25, 2024

View reviewed changes

adamsachs force-pushed the asachs/PROD-2734 branch 2 times, most recently from 2e19433 to 774fd99 Compare September 25, 2024 16:31

adamsachs commented Oct 1, 2024

View reviewed changes

adamsachs self-assigned this Oct 1, 2024

vercel bot deployed to Preview October 1, 2024 20:26 View deployment

adamsachs force-pushed the asachs/PROD-2734 branch from 1441ec7 to 086484c Compare October 1, 2024 20:26

adamsachs added 8 commits October 1, 2024 16:27

initial poc of some partitioning stubbed out

a5bdae5

initial implementation of window based partitioning

8e37ed2

some cleanup

57f6c82

bump fideslang req

abba931

WIP - checkin commit of most recent work

975ca54

where-clause based partitioning with straightforward string substitut…

9079261

…ion, for bq only. no validation (yet)

cleanup

3d17e07

cleanup tests

e314302

adamsachs force-pushed the asachs/PROD-2734 branch from 086484c to e314302 Compare October 1, 2024 20:27

test cleanup

72a3da7

pattisdr reviewed Oct 2, 2024

View reviewed changes

adamsachs added 3 commits October 2, 2024 11:54

move partition functionality to live in bigquery-specific classes

084611d

initial partition clause validation

d8ab88d

update changelog

233bd69

pattisdr reviewed Oct 2, 2024

View reviewed changes

src/fides/api/graph/config.py Show resolved Hide resolved

adamsachs mentioned this pull request Oct 2, 2024

Initial support for DatasetCollection partitioning specifications ethyca/fideslang#21

Merged

7 tasks

improve partition clause validation and some misc. cleanup

d6c04fa

formatting fix

a3cfeb0

adamsachs commented Oct 2, 2024

View reviewed changes

requirements.txt Outdated Show resolved Hide resolved

adamsachs added 2 commits October 2, 2024 16:45

remove outdated code comment

f830164

Merge branch 'main' into asachs/PROD-2734

121d376

vercel bot deployed to Preview October 2, 2024 20:49 View deployment

adamsachs added 2 commits October 2, 2024 16:54

bump fideslang req

0e4dce0

Merge branch 'main' into asachs/PROD-2734

1fde421

vercel bot deployed to Preview October 3, 2024 16:39 View deployment

adamsachs marked this pull request as ready for review October 3, 2024 17:11

add integration test support for partitioned table access request

7a97f42

adamsachs added the run unsafe ci checks Runs fides-related CI checks that require sensitive credentials label Oct 3, 2024

Merge branch 'main' into asachs/PROD-2734

16bcad7

pattisdr approved these changes Oct 3, 2024

View reviewed changes

adjust tests

b8d2c2c

Merge branch 'main' into asachs/PROD-2734

445def5

adamsachs merged commit c8f5b1f into main Oct 4, 2024
14 checks passed

adamsachs deleted the asachs/PROD-2734 branch October 4, 2024 04:17

Kelsey-Ethyca pushed a commit that referenced this pull request Oct 7, 2024

PROD-2734 - initial partitioning support (user-defined windows) (#5325)

8321cd8

		results = connection.execute(stmt)
		rows = self.cursor_result_to_rows(results)

	return f"UPDATE {self.node.address.collection} SET {', '.join(update_clauses)} WHERE {' AND '.join(pk_clauses + ([PARTITION_CLAUSE_TEMPLATE] if self.partitioning else []))}"
	return f"UPDATE {self._generate_table_name()} SET {', '.join(update_clauses)} WHERE {' AND '.join(pk_clauses + ([PARTITION_CLAUSE_TEMPLATE] if self.partitioning else []))}"


		return partitioned_queries

		return [table.delete().where(*pk_clauses)]


		return partitioned_queries

		return [table.update().where(pk_clauses).values(*update_value_map)]

		@@ -443,6 +444,9 @@ class MaskingOverride:
		length: Optional[int]


		PARTITION_CLAUSE_PATTERN = r"`(?P<field_1>[a-zA-Z0-9_])` ([<\|>][=]?) ([a-zA-Z0-9_\s(),.]) AND `(?P<field_2>[a-zA-Z0-9_])` ([<\|>][=]?) ([a-zA-Z0-9_\s(),.])$"

PROD-2734 - initial partitioning support (user-defined windows) #5325

PROD-2734 - initial partitioning support (user-defined windows) #5325

Conversation

adamsachs commented Sep 24, 2024 • edited Loading

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

vercel bot commented Sep 24, 2024 • edited Loading

adamsachs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cypress bot commented Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pattisdr left a comment

Choose a reason for hiding this comment

pattisdr Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pattisdr Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

pattisdr Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pattisdr commented Oct 3, 2024

pattisdr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 4, 2024 • edited Loading

Codecov Report

adamsachs commented Oct 4, 2024

cypress bot commented Oct 4, 2024

adamsachs commented Sep 24, 2024 •

edited

Loading

vercel bot commented Sep 24, 2024 •

edited

Loading

cypress bot commented Sep 24, 2024 •

edited

Loading

pattisdr Oct 2, 2024 •

edited

Loading

pattisdr Oct 2, 2024 •

edited

Loading

pattisdr Oct 2, 2024 •

edited

Loading

codecov bot commented Oct 4, 2024 •

edited

Loading