Synthetic data generation using SDV and KFP #159

tarekabouzeid · 2025-02-16T17:27:07Z

AI-Generated Synthetic Data, using Kubeflow pipelines blog.
Authored by @akeed and @tarekabouzeid .

Signed-off-by: Tarek Abouzeid <[email protected]>

google-oss-prow · 2025-02-16T17:27:15Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johnugeorge for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tarekabouzeid · 2025-02-16T17:33:33Z

@kubeflow/wg-pipeline-leads can someone please review ?
cc @akgraner

juliusvonkohout · 2025-02-17T10:58:02Z

Maybe post it in the kubeflow pipelines slack channel as well. @hbelmiro @HumairAK @rimolive

tarekabouzeid · 2025-02-17T12:11:45Z

Maybe post it in the kubeflow pipelines slack channel as well. @hbelmiro @HumairAK @rimolive

Thanks Julius, will do so also

_posts/2025-02-16-synthetic-data-using-kfp.md

franciscojavierarceo · 2025-02-24T16:34:05Z

_posts/2025-02-16-synthetic-data-using-kfp.md

+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+


First I'd outline what Synthetic Data Generation (SDG) is, what KFP is, and how they work together at a high level.

franciscojavierarceo · 2025-02-24T16:35:09Z

_posts/2025-02-16-synthetic-data-using-kfp.md

+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.


The best results come from real data, but accessing it often requires lengthy security and legal processes.
I think it'd be good to outline what we're try ing to do here and providing context on the problem we're trying to solve.

franciscojavierarceo · 2025-02-24T16:35:55Z

_posts/2025-02-16-synthetic-data-using-kfp.md

+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:


I like this sentence but I think it's missing something about what we're trying to improve. The speed of development for building models?

Thanks, updated.

_posts/2025-02-16-synthetic-data-using-kfp.md

franciscojavierarceo · 2025-02-24T16:42:23Z

_posts/2025-02-16-synthetic-data-using-kfp.md

+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.


We should define synthesizer first somewhere

Thanks, added.

_posts/2025-02-16-synthetic-data-using-kfp.md

franciscojavierarceo · 2025-02-24T16:50:04Z

_posts/2025-02-16-synthetic-data-using-kfp.md

+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 


Could you link to some of those syntheizer resources?

They appear directly on the landing page of (link above, the https://sdv.dev), and not sure if they (SDV) might move around their sub-links (not the sdv.dev, I hope).

franciscojavierarceo · 2025-02-24T16:50:26Z

_posts/2025-02-16-synthetic-data-using-kfp.md

+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:


_posts/2025-02-16-synthetic-data-using-kfp.md

Co-authored-by: Francisco Arceo <[email protected]> Signed-off-by: Ake Edlund <[email protected]>

Updated, thanks to input from franciscojavierarceo Signed-off-by: Ake Edlund <[email protected]>

init - KFP SDV

8f43a5b

Signed-off-by: Tarek Abouzeid <[email protected]>

google-oss-prow bot requested review from franciscojavierarceo and juliusvonkohout February 16, 2025 17:27

google-oss-prow bot added the size/L label Feb 16, 2025

franciscojavierarceo reviewed Feb 24, 2025

View reviewed changes