Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sdk): Implement Pipeline Configuration with TTL #11269

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat(sdk): Implement Pipeline Configuration with TTL #11269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions api/v2alpha1/go/pipelinespec/pipeline_spec.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions api/v2alpha1/pipeline_spec.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1104,4 +1104,5 @@ message PlatformDeploymentConfig {
// Spec for pipeline-level config options. See PipelineConfig DSL class.
message PipelineConfig {
// TODO add pipeline-level configs
int32 pipelineTtl = 1;
}
5 changes: 5 additions & 0 deletions sdk/python/kfp/compiler/pipeline_spec_builder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2086,6 +2086,11 @@ def _merge_pipeline_config(pipelineConfig: pipeline_config.PipelineConfig,
# '<some pipeline config option>': pipelineConfig.<get that value>,
# }}, platformSpec.platforms['kubernetes'])

pipeline_config_json = json_format.ParseDict(
{'pipelineConfig': {
'pipelineTtl': pipelineConfig.get_ttl(),
}}, platformSpec.platforms['kubernetes'])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chensun the SDK code part of this PR correctly compiles and generates the Yaml IR as per the requirements we decided upon here.

Code: https://gist.github.com/rimolive/e0c31ccad1f6c963ec010c455b424afe
resulting IR yaml: https://gist.github.com/rimolive/fc900c8172e661a470a6890555159f97

Could you confirm that the hard coded specification of kubernetes platform here would work? Does the platform key in the IR always stay the same, i.e., kubernetes, or is it customized (to have it set as vertex, for example)? If yes, then we can look into having this key dynamically specified and inserted here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I fully understand the question. The key is hard coded for individual platforms, and the platform backend is expected to look after the key they recognize and ignore the ones they don't. For instance, the KFP backend extracts only kubernetes key:

pipelines/backend/src/apiserver/template/v2_template.go

Lines 72 to 78 in 6a35ee5

// Pick out Kubernetes platform configs
var kubernetesSpec *pipelinespec.SinglePlatformSpec
if t.platformSpec != nil {
if _, ok := t.platformSpec.Platforms["kubernetes"]; ok {
kubernetesSpec = t.platformSpec.Platforms["kubernetes"]
}
}

The specific feature you're adding here, TTL, doesn't apply to Vertex. So what you have in the code and IR links look good to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for clarifying @chensun !


return platformSpec


Expand Down
6 changes: 4 additions & 2 deletions sdk/python/kfp/dsl/pipeline_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@
class PipelineConfig:
"""PipelineConfig contains pipeline-level config options."""

def __init__(self):
pass
def __init__(self, ttl=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would suggest adding a test to ensure compiled IR yaml has ttl fields as expected. Could use this test as a reference.

self.ttl, ttl

# TODO add pipeline level configs
def set_pipeline_ttl(self, ttl: int):
self.ttl = ttl
Loading