Simplify champion-challenger approach

[felix-datatonic]

Description

Add Vertex AI services:

• custom training job
• model evaluation import
• model versioning

Change of logic:

• upload train script before triggering training pipeline
• always upload a new model version after training
• don't re-evaluate an existing champion, rather use historic evaluation results

How has this been tested?

• make test-all-components
• make e2e-tests

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have successfully run the E2E tests, and have included the links to the pipeline runs below
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have updated any relevant documentation to reflect my changes
• I have assigned a reviewer and messaged them

Pipeline run links:

• XGB train champion
• XGB train challenger and beat champion

[ghost]

I think we should be less prescriptive with the command-line arguments. Ideally these would be provided as a list by the user as input to the component (but I see that is tricky given that it contains artifact paths). Could we output the paths as strings from previous components, construct the args in the pipeline definition and then pass into this components as args: List[str] ?

[felix-datatonic]

unfortunately can't use op outputs (or pipeline params) in nested objects, leads to compilation error.

[felix-datatonic]

tested:

train_dataset_uri = (
    extract_bq_to_dataset(
        #...
    )
    .after(data_cleaning)
    .set_display_name("Extract train data to storage")
).outputs['dataset_gcs_uri']

train_args = [
    "--train_data", train_dataset_uri,
    # ...
]

train_model = custom_train_job(
    train_script_uri=train_script_uri,
    args=train_args,
    # ...
).set_display_name("Train model")

Error: TypeError: Object of type PipelineParam is not JSON serializable

[ghost]

another option - pass in args as a string instead of List[str]? I think this would then work

[ghost]

don't forget this one!

[ghost]

@felix-datatonic are you sure about this? Won't this create a nested directory?

[felix-datatonic]

yes, tested successfully. in the prior case only the contents of ${COMMIT_SHA} where copied to ${_PIPELINE_PUBLISH_GCS_PATH}. feel free to test it different combinations and values in Cloud Build though.