Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not encode keys for transforms by default #127

Merged
merged 2 commits into from
Jun 16, 2023
Merged

Do not encode keys for transforms by default #127

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4071fba
Do not encode keys for transforms by default
mckornfield Jun 16, 2023
2cbce02
John's better docs :)
mckornfield Jun 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions src/gretel_trainer/relational/multi_table.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -653,16 +653,21 @@ def run_transforms(
identifier: Optional[str] = None,
in_place: bool = False,
data: Optional[dict[str, pd.DataFrame]] = None,
encode_keys: bool = False,
) -> None:
"""
identifier: (str, optional): Unique string identifying a specific call to this method. Defaults to `transforms_` + current timestamp
Run pre-trained Gretel Transform models on Relational table data:

If `in_place` set to True, overwrites source data in all locations
(internal Python state, local working directory, project artifact archive).
Used for transforms->synthetics workflows.

If `data` is supplied, runs only the supplied data through the corresponding transforms models.
Otherwise runs source data through all existing transforms models.
Args:
identifier: Unique string identifying a specific call to this method. Defaults to `transforms_` + current timestamp
in_place: If True, overwrites source data in all locations
(internal Python state, local working directory, project artifact archive).
Used for transforms->synthetics workflows.
data: If supplied, runs only the supplied data through the corresponding transforms models.
Otherwise runs source data through all existing transforms models.
encode_keys: If set, primary and foreign keys will be replaced with label encoded variants. This can add
an additional level of privacy at the cost of referential integrity between transformed and
original data.
"""
if data is not None:
unrunnable_tables = [
Expand Down Expand Up @@ -706,9 +711,11 @@ def run_transforms(
)
run_task(task, self._extended_sdk)

output_tables = self._strategy.label_encode_keys(
self.relational_data, task.output_tables
)
output_tables = task.output_tables
if encode_keys:
output_tables = self._strategy.label_encode_keys(
self.relational_data, task.output_tables
)

if in_place:
for table_name, transformed_table in output_tables.items():
Expand Down