Lock in version of kfp

[MattMorgis]

My team and I hit a snag this week when trying to compile Kubeflow Pipelines:

File "/tfx/examples/chicago_taxi_pipeline/taxi_pipeline_kubeflow.py", line 187, in <module>
    pipeline = KubeflowRunner().run(_create_pipeline())

File "/tfx/orchestration/kubeflow/runner.py", line 122, in run
    _construct_pipeline)

TypeError: add_pipeline() takes 2 positional arguments but 3 were given

Thanks to commit eb63dae this week which added a test for this, we were able to debug easier.

It turns out the issue was that we had version 0.1.14 of the Kubeflow Pipelines SDK installed. The docs recommended this, and the docs and the Release Page do not install from PyPi.

This feels like it was intentional, but only version published to PyPi is 0.1.11, the last compatible version with tfx. I've tried 0.1.12 all the way to the most recent 0.1.15 and continue to get the same error as above.

Additionally, the check submitted by this PR doesn't even catch this.

When running pip show kfp from the non-PyPi install, I get:

Name: kfp
Version: 0.1
Summary: KubeFlow Pipelines SDK
Home-page: UNKNOWN
Author: google
Author-email: None
License: UNKNOWN
Location: /usr/local/lib/python3.7/site-packages
Requires: PyYAML, kubernetes, requests-toolbelt, certifi, google-cloud-storage, google-auth, cryptography, python-dateutil, six, PyJWT, urllib3
Required-by:

And when running after installing from setup.py:

Name: kfp
Version: 0.1.11
Summary: KubeFlow Pipelines SDK
Home-page: UNKNOWN
Author: google
Author-email: None
License: UNKNOWN
Location:  /usr/local/lib/python3.7/site-packages
Requires: PyJWT, google-cloud-storage, PyYAML, certifi, six, python-dateutil, kubernetes, requests-toolbelt, cryptography, urllib3, google-auth

I'm not sure if I should follow an issue in Kubeflow/pipelines and ask if they can tag with the patch, or if these will never be published to PyPi and this is a non-issue.

I'm unsure what the proper fix should be, but wanted to submit to start the conversation and document for anyone else who may run into this.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[MattMorgis]

I signed it

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[neuromage]

Thanks for the fix @MattMorgis!

I'll also send out a fix later so that later versions of KFP can be used as well.

[Ark-kun]

Thanks for the fix.

The runner is failing, because it was using undocumented hack.

The KFP documentation clearly says that the pipeline function must be decorated with @pipeline decorator. It's even mentioned in the compile function documentation: https://github.com/kubeflow/pipelines/blob/086d4763d9f163acdeb423bc2bc49ab470442a92/sdk/python/kfp/compiler/compiler.py#L636

Running compile without having the function decorated

tfx/tfx/orchestration/kubeflow/runner.py

Line 125 in d9f87b9

compiler.Compiler().compile(_construct_pipeline, pipeline_file)

will raise error: https://github.com/kubeflow/pipelines/blob/086d4763d9f163acdeb423bc2bc49ab470442a92/sdk/python/kfp/compiler/compiler.py#L578 'Please use a function with @dsl.pipeline decorator.'.

This documented behavior was circumvented by a hack that used an undocumented function that should be a part of internal compiler API. Ning has recently changed this internal function, which triggered TFX failure.

I'll send a PR fixing TFX code fixing this issue and removing the use of undocumented API.

[Ark-kun]

Added the PR to fix the TFX code: #39
It adds the @pipeline decorator to the pipeline function just like the documentation says.