Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Enable flytekit to authenticate with proxy in front of FlyteAdmin #1787

Merged
merged 24 commits into from
Sep 20, 2023

Conversation

fg91
Copy link
Member

@fg91 fg91 commented Aug 11, 2023

TL;DR

Part of an effort to integrate Flyte with GCP Identity Aware Proxy (IAP).
See tracking issue for details and motivation: flyteorg/flyte#3965

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested (Deployed Flyte with IAP in sandbox cluster according to deployment guide in the readme added in this PR. Tested that FlyteRemote can talk to admin through the IAP, e.g. to retrieve executions, start new ones etc.)
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

  • Add a new option called proxyCommand to the platform config in which users can specify an external command which, if specified, is used to generate ID tokens for a proxy in front of Flyte.
    • This idea is derived from the existing command (external command auth type) which is used to generate tokens for Flyte itself.
  • The ID tokens generated by this proxyCommand are added as "proxy-authorization" header to every request (http, gRPC) that flytekit's client makes to flyteadmin, including the initial unauthenticated requests of the auth flow with flyteadmin. These additional ID tokens in the "proxy-authorization" header allow clients to interact with a flyteadmin that is protected by GCP Identity Aware Proxy (or in theory similar services from other providers)
  • Refactor the AuthorizationClient which currently can only be used for a pkce auth flow (despite its doc string claiming it works for a general OAuth 2.0 flow) to actually work for both pkce and default OAuth 2.0.
  • Add a plugin which provides a CLI that generates ID tokens for GCP Identity Aware Proxy
    • This plugin re-uses the AuthorizationClient in flytekit to perform an OAuth 2.0 flow with accounts.google.com.
    • The command is to be used as proxyCommand in the platform config for Flyte deployments protected with IAP
  • Add a step-by-step guide on how to deploy Flyte protected with GCP IAP

Tracking Issue

https://github.com/flyteorg/flyte/issues/

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

@@ -64,6 +64,13 @@ class Credentials(object):
This command is executed to return a token using an external process.
"""

PROXY_COMMAND = ConfigEntry(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: do we need the legacy config entry or do we only support this in yaml config?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is the correct declaration. Let's keep it. But what is this? what does this do?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an explanation in this PR's description.

@@ -61,7 +61,7 @@ def intercept_unary_unary(
fut: grpc.Future = continuation(updated_call_details, request)
e = fut.exception()
if e:
if e.code() == grpc.StatusCode.UNAUTHENTICATED:
if e.code() == grpc.StatusCode.UNAUTHENTICATED or e.code() == grpc.StatusCode.UNKNOWN:
Copy link
Member Author

@fg91 fg91 Aug 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When trying to reach flyteadmin behind GCP Identity Aware Proxy and when omitting the "proxy-authorization" header, we get back:

<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Stream removed"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:<my-ip>:443 {grpc_message:"Stream removed", grpc_status:2, created_time:"2023-xxx"}"

@fg91 fg91 changed the title Fg91/feat/proxy authentication Feat: Enable flytekit to authenticate with proxy in front of FlyteAdmin Aug 11, 2023
@fg91 fg91 force-pushed the fg91/feat/proxy-authentication branch from 7987098 to c80cf60 Compare August 15, 2023 14:06
@fg91
Copy link
Member Author

fg91 commented Aug 24, 2023

@wild-endeavor I merged #1795 into this one as requested 👍

authn.refresh_credentials()
expected_scopes = static_cfg_store.get_client_config().scopes

assert authn._creds
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order of mocking vs instantiation of the authenticator matters now.
I added the more explicit check assert authn._creds.access_token == "abc" because this would fail in case somebody in the future reverses the order again (instantiate the authenticator first, then mock).

assert t
assert e

assert t == "abc"
Copy link
Member Author

@fg91 fg91 Aug 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be more specific here. When adapting the magic mocks, I noticed that here one easily gets a MagicMock object for t which then also is not None but is not "abc".

This makes sure that in the future no one accidentally breaks the mocking.

Fabio Grätz and others added 19 commits August 25, 2023 17:50
…ng external command authenticator

Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
#1795)

* Add external command plugin to generate id tokens for identity aware proxy

Signed-off-by: Fabio Grätz <[email protected]>

* Retrieve desktop app client secret from gcp secret manager

Signed-off-by: Fabio Grätz <[email protected]>

* Remove comments

Signed-off-by: Fabio Grätz <[email protected]>

* Introduce a command group that allows adding a command to generate service account id tokens later

Signed-off-by: Fabio Grätz <[email protected]>

* Document how to use plugin and deploy Flyte with IAP

Signed-off-by: Fabio Grätz <[email protected]>

* Minor corrections README.md

Signed-off-by: Fabio Grätz <[email protected]>

---------

Signed-off-by: Fabio Grätz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
@fg91 fg91 force-pushed the fg91/feat/proxy-authentication branch from 858e96b to c677ff3 Compare August 25, 2023 15:50
@fg91 fg91 marked this pull request as ready for review August 25, 2023 16:14
@fg91 fg91 requested a review from kumare3 as a code owner August 25, 2023 16:14
@fg91 fg91 force-pushed the fg91/feat/proxy-authentication branch from 4d09bf8 to 336aebc Compare September 7, 2023 18:32
@fg91
Copy link
Member Author

fg91 commented Sep 9, 2023

Thanks for taking a look @wild-endeavor

I added a second subcommand to flyte-iap to generate ID tokens for service accounts:

~ flyte-iap --help 
Usage: flyte-iap [OPTIONS] COMMAND [ARGS]...

  Generate ID tokens for GCP Identity Aware Proxy (IAP).

Options:
  --help  Show this message and exit.

Commands:
  generate-service-account-id-token                                         # <- new
                                  Generate a service account ID token...
  generate-user-id-token          Generate a user account ID token for...

The subcommand uses either

  • a service account key json file or
  • contacts the GCP metadata server to obtain a token when executed in Compute Engine etc.

To generate a token for service accounts, in contrast to generating one for user accounts, no browser needs to be opened interactively. The goal of this subcommand is that CICD pipelines can talk to flyteadmin through IAP. (@corleyma said they will require this.)

With that, flyte-iap is complete.

flytectl still doesn't understand the proxyCommand in the config but I will work on this next:

flytectl config init
Error:

1 error(s) decoding:

* '' has invalid keys: proxycommand
ERRO[0000]

1 error(s) decoding:

* '' has invalid keys: proxycommand  src="main.go:13"

@fg91
Copy link
Member Author

fg91 commented Sep 9, 2023

i think this looks good. cc @EngHabu to take a look though.

Tagging you as reviewer, @EngHabu, as suggested by @wild-endeavor 🙏

@fg91 fg91 requested a review from EngHabu September 9, 2023 07:50
EngHabu
EngHabu previously approved these changes Sep 10, 2023
Copy link
Collaborator

@EngHabu EngHabu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm less sure about reusing this command for flytectl. Requiring python dependency isn't something I would want to require in CI/CD and other automated systems

@fg91
Copy link
Member Author

fg91 commented Sep 11, 2023

Hi Haytham, I really appreciate the time you’ve invested in looking at this PR, I'm sure you have a busy schedule 🙏

I'm less sure about reusing this command for flytectl. Requiring python dependency isn't something I would want to require in CI/CD and other automated systems

In our case, we will run pyflyte register/package in CICD as well but I agree with your concern that this shouldn't be required. I documented 2 additional ways to obtain an id token for service accounts using 1) GCP's gcloud sdk and 2) a bash script from the identity aware proxy documentation. This way, users can choose to not use python.
2437b84

@wild-endeavor
Copy link
Contributor

tested this with pkce, cc, and device flow btw, all works

@fg91
Copy link
Member Author

fg91 commented Sep 20, 2023

tested this with pkce, cc, and device flow btw, all works

Thanks for testing this @wild-endeavor! 🙏
I also implemented the same logic for flytectl/the admin client in idl. Everything works, just need to polish a bit and adapt/add tests. Will ping you there as well then.

@fg91 fg91 merged commit cdcba2f into master Sep 20, 2023
132 of 134 checks passed
jeevb pushed a commit that referenced this pull request Sep 20, 2023
…dmin (#1787)

* Introduce authenticator engine and make proxy auth work

Signed-off-by: Fabio Grätz <[email protected]>

* Use proxy authed session for client credentials flow

Signed-off-by: Fabio Grätz <[email protected]>

* Don't use authenticator engine but do proxy authentication via existing external command authenticator

Signed-off-by: Fabio Grätz <[email protected]>

* Add docstring to AuthenticationHTTPAdapter

Signed-off-by: Fabio Grätz <[email protected]>

* Address todo in docstring

Signed-off-by: Fabio Grätz <[email protected]>

* Create blank session if none provided

Signed-off-by: Fabio Grätz <[email protected]>

* Create blank session if none provided in get_token

Signed-off-by: Fabio Grätz <[email protected]>

* Refresh proxy creds in session when not existing without triggering 401

Signed-off-by: Fabio Grätz <[email protected]>

* Add test for get_session

Signed-off-by: Fabio Grätz <[email protected]>

* Move auth helper test into existing module

Signed-off-by: Fabio Grätz <[email protected]>

* Move auth helper test into existing module

Signed-off-by: Fabio Grätz <[email protected]>

* Add test for upgrade_channel_to_proxy_authenticated

Signed-off-by: Fabio Grätz <[email protected]>

* Auth helper tests without use of responses package

Signed-off-by: Fabio Grätz <[email protected]>

* Feat: Add plugin for generating GCP IAP ID tokens via external command (#1795)

* Add external command plugin to generate id tokens for identity aware proxy

Signed-off-by: Fabio Grätz <[email protected]>

* Retrieve desktop app client secret from gcp secret manager

Signed-off-by: Fabio Grätz <[email protected]>

* Remove comments

Signed-off-by: Fabio Grätz <[email protected]>

* Introduce a command group that allows adding a command to generate service account id tokens later

Signed-off-by: Fabio Grätz <[email protected]>

* Document how to use plugin and deploy Flyte with IAP

Signed-off-by: Fabio Grätz <[email protected]>

* Minor corrections README.md

Signed-off-by: Fabio Grätz <[email protected]>

---------

Signed-off-by: Fabio Grätz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>

* Use proxy auth'ed session for device code auth flow

Signed-off-by: Fabio Grätz <[email protected]>

* Fix token client tests

Signed-off-by: Fabio Grätz <[email protected]>

* Make poll token endpoint test more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Make test_client_creds_authenticator test work and more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Make test_client_creds_authenticator_with_custom_scopes test work and more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Implement subcommand to generate id tokens for service accounts

Signed-off-by: Fabio Graetz <[email protected]>

* Test id token generation from service accounts

Signed-off-by: Fabio Graetz <[email protected]>

* Fix plugin requirements

Signed-off-by: Fabio Graetz <[email protected]>

* Document usage of generate-service-account-id-token subcommand

Signed-off-by: Fabio Grätz <[email protected]>

* Document alternative ways to obtain service account id tokens

Signed-off-by: Fabio Grätz <[email protected]>

---------

Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Graetz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Signed-off-by: Jeev B <[email protected]>
Future-Outlier pushed a commit to Future-Outlier/flytekit that referenced this pull request Oct 3, 2023
…dmin (flyteorg#1787)

* Introduce authenticator engine and make proxy auth work

Signed-off-by: Fabio Grätz <[email protected]>

* Use proxy authed session for client credentials flow

Signed-off-by: Fabio Grätz <[email protected]>

* Don't use authenticator engine but do proxy authentication via existing external command authenticator

Signed-off-by: Fabio Grätz <[email protected]>

* Add docstring to AuthenticationHTTPAdapter

Signed-off-by: Fabio Grätz <[email protected]>

* Address todo in docstring

Signed-off-by: Fabio Grätz <[email protected]>

* Create blank session if none provided

Signed-off-by: Fabio Grätz <[email protected]>

* Create blank session if none provided in get_token

Signed-off-by: Fabio Grätz <[email protected]>

* Refresh proxy creds in session when not existing without triggering 401

Signed-off-by: Fabio Grätz <[email protected]>

* Add test for get_session

Signed-off-by: Fabio Grätz <[email protected]>

* Move auth helper test into existing module

Signed-off-by: Fabio Grätz <[email protected]>

* Move auth helper test into existing module

Signed-off-by: Fabio Grätz <[email protected]>

* Add test for upgrade_channel_to_proxy_authenticated

Signed-off-by: Fabio Grätz <[email protected]>

* Auth helper tests without use of responses package

Signed-off-by: Fabio Grätz <[email protected]>

* Feat: Add plugin for generating GCP IAP ID tokens via external command (flyteorg#1795)

* Add external command plugin to generate id tokens for identity aware proxy

Signed-off-by: Fabio Grätz <[email protected]>

* Retrieve desktop app client secret from gcp secret manager

Signed-off-by: Fabio Grätz <[email protected]>

* Remove comments

Signed-off-by: Fabio Grätz <[email protected]>

* Introduce a command group that allows adding a command to generate service account id tokens later

Signed-off-by: Fabio Grätz <[email protected]>

* Document how to use plugin and deploy Flyte with IAP

Signed-off-by: Fabio Grätz <[email protected]>

* Minor corrections README.md

Signed-off-by: Fabio Grätz <[email protected]>

---------

Signed-off-by: Fabio Grätz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>

* Use proxy auth'ed session for device code auth flow

Signed-off-by: Fabio Grätz <[email protected]>

* Fix token client tests

Signed-off-by: Fabio Grätz <[email protected]>

* Make poll token endpoint test more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Make test_client_creds_authenticator test work and more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Make test_client_creds_authenticator_with_custom_scopes test work and more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Implement subcommand to generate id tokens for service accounts

Signed-off-by: Fabio Graetz <[email protected]>

* Test id token generation from service accounts

Signed-off-by: Fabio Graetz <[email protected]>

* Fix plugin requirements

Signed-off-by: Fabio Graetz <[email protected]>

* Document usage of generate-service-account-id-token subcommand

Signed-off-by: Fabio Grätz <[email protected]>

* Document alternative ways to obtain service account id tokens

Signed-off-by: Fabio Grätz <[email protected]>

---------

Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Graetz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Signed-off-by: Future Outlier <[email protected]>
jeevb added a commit that referenced this pull request Nov 1, 2023
* pip through to container

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* move around

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add asserts

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* delete bad line

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* switch to abc and add support for gpu unpartitioned

Signed-off-by: Jeev B <[email protected]>

* Add Azure-specific headers when uploading to blob storage (#1784)

* Add Azure-specific headers when uploading to blob storage

Signed-off-by: Victor Delépine <[email protected]>

* Add comment about HTTP 201 check

Signed-off-by: Victor Delépine <[email protected]>

---------

Signed-off-by: Victor Delépine <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add async delete function in base_agent (#1800)

Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add support for execution name prefixes (#1803)

Signed-off-by: troychiu <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Remove ref in output (#1794)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (#1801)

* Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking

As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module

Signed-off-by: Matthew Hoffman <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Async file sensor (#1790)

---------
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Eager workflows to support async workflows (#1579)

* Eager workflows to support async workflows

Signed-off-by: Niels Bantilan <[email protected]>

* move array node maptask to experimental/__init__.py

Signed-off-by: Niels Bantilan <[email protected]>

* clean up docs

Signed-off-by: Niels Bantilan <[email protected]>

* clean up

Signed-off-by: Niels Bantilan <[email protected]>

* more clean up

Signed-off-by: Niels Bantilan <[email protected]>

* docs cleanup

Signed-off-by: Niels Bantilan <[email protected]>

* Update test_eager_workflows.py

* clean up timeout handling

Signed-off-by: Niels Bantilan <[email protected]>

* fix lint

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Enable SecretsManager.get to load and return bytes (#1798)

* fix secretsmanager

Signed-off-by: Yue Shang <[email protected]>

* fix lint issue

Signed-off-by: Yue Shang <[email protected]>

* add doc

Signed-off-by: Yue Shang <[email protected]>

* fix github check

Signed-off-by: Yue Shang <[email protected]>

---------

Signed-off-by: Yue Shang <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Batch upload flyte directory (#1806)

* Batch upload flyte directory

Signed-off-by: Kevin Su <[email protected]>

* Update get method

Signed-off-by: Kevin Su <[email protected]>

* Move batch size to type engine

Signed-off-by: Kevin Su <[email protected]>

* comment

Signed-off-by: Kevin Su <[email protected]>

* update comment

Signed-off-by: Kevin Su <[email protected]>

* Update flytekit/core/type_engine.py

Co-authored-by: Eduardo Apolinario <[email protected]>

* Add test

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Better error messaging for overrides (#1807)

- using incorrect type of overrides
 - using incorrect type for resources
 - using promises in overrides

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Run remote Launchplan from `pyflyte run` (#1785)

* Beautified pyflyte run even for every task and workflow

- identify a task or a workflow
- task or workflow help menus show types and use rich to beautify

Signed-off-by: Ketan Umare <[email protected]>

* one more improvement

Signed-off-by: Ketan Umare <[email protected]>

* updated

Signed-off-by: Ketan Umare <[email protected]>

* updated command

Signed-off-by: Ketan Umare <[email protected]>

* Updated

Signed-off-by: Ketan Umare <[email protected]>

* updated formatting

Signed-off-by: Ketan Umare <[email protected]>

* updated

Signed-off-by: Ketan Umare <[email protected]>

* updated

Signed-off-by: Ketan Umare <[email protected]>

* bug fixed in types

Signed-off-by: Ketan Umare <[email protected]>

* Updated

Signed-off-by: Ketan Umare <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add is none function (#1757)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Dynamic workflow should not throw nested task warning (#1812)

Signed-off-by: oliverhu <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add a manual image building GH action (#1816)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (#1813)

Signed-off-by: Jan Fiedler <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* None doesnt work

Signed-off-by: Jeev B <[email protected]>

* unpartitioned selector

Signed-off-by: Jeev B <[email protected]>

* Fix list of annotated structured dataset (#1817)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (#1819)

* Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit

Signed-off-by: Chao-Heng Lee <[email protected]>

* remove helper of getting env var.

Signed-off-by: Chao-Heng Lee <[email protected]>

* refactor variable name.

Signed-off-by: Chao-Heng Lee <[email protected]>

---------

Signed-off-by: Chao-Heng Lee <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Async agent delete function for while loop case (#1802)

Signed-off-by: Future Outlier <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* refactor

Signed-off-by: Jeev B <[email protected]>

* fix docs warnings (#1827)

Signed-off-by: Jeev B <[email protected]>

* Fix extract_task_module (#1829)

---------

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Feat: Add type support for pydantic BaseModels (#1660)

Signed-off-by: Adrian Rumpold <[email protected]>
Signed-off-by: Arthur <[email protected]>
Signed-off-by: wirthual <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add test for unspecified mig

Signed-off-by: Jeev B <[email protected]>

* add support for overriding accelerator

Signed-off-by: Jeev B <[email protected]>

* cleanup

Signed-off-by: Jeev B <[email protected]>

* move from core to extras

Signed-off-by: Jeev B <[email protected]>

* fixes

Signed-off-by: Jeev B <[email protected]>

* fixes

Signed-off-by: Jeev B <[email protected]>

* fixes

Signed-off-by: Jeev B <[email protected]>

* cleanup

Signed-off-by: Jeev B <[email protected]>

* Make FlyteRemote slightly more copy/pastable (#1830)

Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Pyflyte meta inputs (#1823)

* Re-orgining pyflyte run

Signed-off-by: Ketan Umare <[email protected]>

* Pyflyte beautified and simplified

Signed-off-by: Ketan Umare <[email protected]>

* fixed unit test

Signed-off-by: Ketan Umare <[email protected]>

* Added Launch options

Signed-off-by: Ketan Umare <[email protected]>

* lint fix

Signed-off-by: Ketan Umare <[email protected]>

* test fix

Signed-off-by: Ketan Umare <[email protected]>

* fixing docs failure

Signed-off-by: Ketan Umare <[email protected]>

---------

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Use mashumaro to serialize/deserialize dataclass (#1735)

Signed-off-by: HH <[email protected]>
Signed-off-by: hhcs9527 <[email protected]>
Signed-off-by: Matthew Hoffman <[email protected]>
Co-authored-by: Matthew Hoffman <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Databricks Agent (#1797)

Signed-off-by: Future Outlier <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Prometheus metrics (#1815)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Pyflyte register optionally activates schedule (#1832)

* Pyflyte register auto activates schedule

Signed-off-by: Ketan Umare <[email protected]>

* comment addressed

Signed-off-by: Ketan Umare <[email protected]>

---------

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Remove versions 3.9 and 3.10 (#1831)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Snowflake agent (#1799)

Signed-off-by: hhcs9527 <[email protected]>
Signed-off-by: HH <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Update agent metric name (#1835)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* MemVerge MMCloud Agent (#1821)

Signed-off-by: Edwin Yu <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add download badges in readme (#1836)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Eager local entrypoint and support for offloaded types (#1833)

* implement eager workflow local entrypoint, support offloaded types

Signed-off-by: Niels Bantilan <[email protected]>

* wip local entrypoint

Signed-off-by: Niels Bantilan <[email protected]>

* add tests

Signed-off-by: Niels Bantilan <[email protected]>

* add local entrypoint tests

Signed-off-by: Niels Bantilan <[email protected]>

* update eager unit tests, delete test script

Signed-off-by: Niels Bantilan <[email protected]>

* clean up tests

Signed-off-by: Niels Bantilan <[email protected]>

* update ci

Signed-off-by: Niels Bantilan <[email protected]>

* update ci

Signed-off-by: Niels Bantilan <[email protected]>

* update ci

Signed-off-by: Niels Bantilan <[email protected]>

* update ci

Signed-off-by: Niels Bantilan <[email protected]>

* remove push step

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* update requirements and add snowflake agent to api reference (#1838)

* update requirements and add snowflake agent to api reference

Signed-off-by: Samhita Alla <[email protected]>

* update requirements

Signed-off-by: Samhita Alla <[email protected]>

* remove versions

Signed-off-by: Samhita Alla <[email protected]>

* remove tensorflow-macos

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* downgrade sphinxcontrib-youtube package

Signed-off-by: Samhita Alla <[email protected]>

---------

Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Fix: Make sure decks created in elastic task workers are transferred to parent process (#1837)

* Transfer decks created in the worker process to the parent process

Signed-off-by: Fabio Graetz <[email protected]>

* Add test for decks in elastic tasks

Signed-off-by: Fabio Graetz <[email protected]>

* Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py

Signed-off-by: Fabio Graetz <[email protected]>

* Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py

Signed-off-by: Fabio Graetz <[email protected]>

---------

Signed-off-by: Fabio Graetz <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add accept grpc (#1841)

* add accept grpc

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* unpin setup.py grpc

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Revert "add accept grpc"

This reverts commit 2294592.

Signed-off-by: Jeev B <[email protected]>

* default headers interceptor

Signed-off-by: Jeev B <[email protected]>

* setup.py

Signed-off-by: Jeev B <[email protected]>

* fixes

Signed-off-by: Jeev B <[email protected]>

* fmt

Signed-off-by: Jeev B <[email protected]>

* move prometheus-client import

Signed-off-by: Jeev B <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>
Co-authored-by: Jeev B <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (#1787)

* Introduce authenticator engine and make proxy auth work

Signed-off-by: Fabio Grätz <[email protected]>

* Use proxy authed session for client credentials flow

Signed-off-by: Fabio Grätz <[email protected]>

* Don't use authenticator engine but do proxy authentication via existing external command authenticator

Signed-off-by: Fabio Grätz <[email protected]>

* Add docstring to AuthenticationHTTPAdapter

Signed-off-by: Fabio Grätz <[email protected]>

* Address todo in docstring

Signed-off-by: Fabio Grätz <[email protected]>

* Create blank session if none provided

Signed-off-by: Fabio Grätz <[email protected]>

* Create blank session if none provided in get_token

Signed-off-by: Fabio Grätz <[email protected]>

* Refresh proxy creds in session when not existing without triggering 401

Signed-off-by: Fabio Grätz <[email protected]>

* Add test for get_session

Signed-off-by: Fabio Grätz <[email protected]>

* Move auth helper test into existing module

Signed-off-by: Fabio Grätz <[email protected]>

* Move auth helper test into existing module

Signed-off-by: Fabio Grätz <[email protected]>

* Add test for upgrade_channel_to_proxy_authenticated

Signed-off-by: Fabio Grätz <[email protected]>

* Auth helper tests without use of responses package

Signed-off-by: Fabio Grätz <[email protected]>

* Feat: Add plugin for generating GCP IAP ID tokens via external command (#1795)

* Add external command plugin to generate id tokens for identity aware proxy

Signed-off-by: Fabio Grätz <[email protected]>

* Retrieve desktop app client secret from gcp secret manager

Signed-off-by: Fabio Grätz <[email protected]>

* Remove comments

Signed-off-by: Fabio Grätz <[email protected]>

* Introduce a command group that allows adding a command to generate service account id tokens later

Signed-off-by: Fabio Grätz <[email protected]>

* Document how to use plugin and deploy Flyte with IAP

Signed-off-by: Fabio Grätz <[email protected]>

* Minor corrections README.md

Signed-off-by: Fabio Grätz <[email protected]>

---------

Signed-off-by: Fabio Grätz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>

* Use proxy auth'ed session for device code auth flow

Signed-off-by: Fabio Grätz <[email protected]>

* Fix token client tests

Signed-off-by: Fabio Grätz <[email protected]>

* Make poll token endpoint test more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Make test_client_creds_authenticator test work and more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Make test_client_creds_authenticator_with_custom_scopes test work and more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Implement subcommand to generate id tokens for service accounts

Signed-off-by: Fabio Graetz <[email protected]>

* Test id token generation from service accounts

Signed-off-by: Fabio Graetz <[email protected]>

* Fix plugin requirements

Signed-off-by: Fabio Graetz <[email protected]>

* Document usage of generate-service-account-id-token subcommand

Signed-off-by: Fabio Grätz <[email protected]>

* Document alternative ways to obtain service account id tokens

Signed-off-by: Fabio Grätz <[email protected]>

---------

Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Graetz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* bump flyteidl

Signed-off-by: Jeev B <[email protected]>

* make requirements

Signed-off-by: Jeev B <[email protected]>

* fix failing tests

Signed-off-by: Jeev B <[email protected]>

* move gpu accelerator to flyteidl.core.Resources

Signed-off-by: Jeev B <[email protected]>

* Use ResourceExtensions for extended resources

Signed-off-by: Jeev B <[email protected]>

* cleanup

Signed-off-by: Jeev B <[email protected]>

* Switch to using ExtendedResources in TaskTemplate

Signed-off-by: Jeev B <[email protected]>

* cleanups

Signed-off-by: Jeev B <[email protected]>

* update flyteidl

Signed-off-by: Jeev B <[email protected]>

* Replace _core_task imports with tasks_pb2

Signed-off-by: Jeev B <[email protected]>

* less verbose definitions

Signed-off-by: Jeev B <[email protected]>

* Attempt at less confusing syntax

Signed-off-by: Jeev B <[email protected]>

* Streamline UX

Signed-off-by: Jeev B <[email protected]>

* Run make fmt

Signed-off-by: Jeev B <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>
Signed-off-by: Victor Delépine <[email protected]>
Signed-off-by: Future Outlier <[email protected]>
Signed-off-by: troychiu <[email protected]>
Signed-off-by: Matthew Hoffman <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Yue Shang <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: oliverhu <[email protected]>
Signed-off-by: Jan Fiedler <[email protected]>
Signed-off-by: Chao-Heng Lee <[email protected]>
Signed-off-by: Adrian Rumpold <[email protected]>
Signed-off-by: Arthur <[email protected]>
Signed-off-by: wirthual <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: HH <[email protected]>
Signed-off-by: hhcs9527 <[email protected]>
Signed-off-by: Edwin Yu <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Fabio Graetz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
Co-authored-by: Victor Delépine <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Co-authored-by: Yi Chiu <[email protected]>
Co-authored-by: Matthew Hoffman <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Niels Bantilan <[email protected]>
Co-authored-by: Yue Shang <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Ketan Umare <[email protected]>
Co-authored-by: Keqiu Hu <[email protected]>
Co-authored-by: Jan Fiedler <[email protected]>
Co-authored-by: Chao-Heng Lee <[email protected]>
Co-authored-by: Samhita Alla <[email protected]>
Co-authored-by: Arthur Böök <[email protected]>
Co-authored-by: Katrina Rogan <[email protected]>
Co-authored-by: Po Han(Hank) Huang <[email protected]>
Co-authored-by: Edwin Yu <[email protected]>
Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
ringohoffman added a commit to ringohoffman/flytekit that referenced this pull request Nov 24, 2023
* pip through to container

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* move around

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add asserts

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* delete bad line

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* switch to abc and add support for gpu unpartitioned

Signed-off-by: Jeev B <[email protected]>

* Add Azure-specific headers when uploading to blob storage (flyteorg#1784)

* Add Azure-specific headers when uploading to blob storage

Signed-off-by: Victor Delépine <[email protected]>

* Add comment about HTTP 201 check

Signed-off-by: Victor Delépine <[email protected]>

---------

Signed-off-by: Victor Delépine <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add async delete function in base_agent (flyteorg#1800)

Signed-off-by: Future Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add support for execution name prefixes (flyteorg#1803)

Signed-off-by: troychiu <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Remove ref in output (flyteorg#1794)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (flyteorg#1801)

* Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking

As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module

Signed-off-by: Matthew Hoffman <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Async file sensor (flyteorg#1790)

---------
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Eager workflows to support async workflows (flyteorg#1579)

* Eager workflows to support async workflows

Signed-off-by: Niels Bantilan <[email protected]>

* move array node maptask to experimental/__init__.py

Signed-off-by: Niels Bantilan <[email protected]>

* clean up docs

Signed-off-by: Niels Bantilan <[email protected]>

* clean up

Signed-off-by: Niels Bantilan <[email protected]>

* more clean up

Signed-off-by: Niels Bantilan <[email protected]>

* docs cleanup

Signed-off-by: Niels Bantilan <[email protected]>

* Update test_eager_workflows.py

* clean up timeout handling

Signed-off-by: Niels Bantilan <[email protected]>

* fix lint

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Enable SecretsManager.get to load and return bytes (flyteorg#1798)

* fix secretsmanager

Signed-off-by: Yue Shang <[email protected]>

* fix lint issue

Signed-off-by: Yue Shang <[email protected]>

* add doc

Signed-off-by: Yue Shang <[email protected]>

* fix github check

Signed-off-by: Yue Shang <[email protected]>

---------

Signed-off-by: Yue Shang <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Batch upload flyte directory (flyteorg#1806)

* Batch upload flyte directory

Signed-off-by: Kevin Su <[email protected]>

* Update get method

Signed-off-by: Kevin Su <[email protected]>

* Move batch size to type engine

Signed-off-by: Kevin Su <[email protected]>

* comment

Signed-off-by: Kevin Su <[email protected]>

* update comment

Signed-off-by: Kevin Su <[email protected]>

* Update flytekit/core/type_engine.py

Co-authored-by: Eduardo Apolinario <[email protected]>

* Add test

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Better error messaging for overrides (flyteorg#1807)

- using incorrect type of overrides
 - using incorrect type for resources
 - using promises in overrides

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Run remote Launchplan from `pyflyte run` (flyteorg#1785)

* Beautified pyflyte run even for every task and workflow

- identify a task or a workflow
- task or workflow help menus show types and use rich to beautify

Signed-off-by: Ketan Umare <[email protected]>

* one more improvement

Signed-off-by: Ketan Umare <[email protected]>

* updated

Signed-off-by: Ketan Umare <[email protected]>

* updated command

Signed-off-by: Ketan Umare <[email protected]>

* Updated

Signed-off-by: Ketan Umare <[email protected]>

* updated formatting

Signed-off-by: Ketan Umare <[email protected]>

* updated

Signed-off-by: Ketan Umare <[email protected]>

* updated

Signed-off-by: Ketan Umare <[email protected]>

* bug fixed in types

Signed-off-by: Ketan Umare <[email protected]>

* Updated

Signed-off-by: Ketan Umare <[email protected]>

* lint

Signed-off-by: Kevin Su <[email protected]>

---------

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add is none function (flyteorg#1757)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Dynamic workflow should not throw nested task warning (flyteorg#1812)

Signed-off-by: oliverhu <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add a manual image building GH action (flyteorg#1816)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (flyteorg#1813)

Signed-off-by: Jan Fiedler <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* None doesnt work

Signed-off-by: Jeev B <[email protected]>

* unpartitioned selector

Signed-off-by: Jeev B <[email protected]>

* Fix list of annotated structured dataset (flyteorg#1817)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (flyteorg#1819)

* Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit

Signed-off-by: Chao-Heng Lee <[email protected]>

* remove helper of getting env var.

Signed-off-by: Chao-Heng Lee <[email protected]>

* refactor variable name.

Signed-off-by: Chao-Heng Lee <[email protected]>

---------

Signed-off-by: Chao-Heng Lee <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Async agent delete function for while loop case (flyteorg#1802)

Signed-off-by: Future Outlier <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* refactor

Signed-off-by: Jeev B <[email protected]>

* fix docs warnings (flyteorg#1827)

Signed-off-by: Jeev B <[email protected]>

* Fix extract_task_module (flyteorg#1829)

---------

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Feat: Add type support for pydantic BaseModels (flyteorg#1660)

Signed-off-by: Adrian Rumpold <[email protected]>
Signed-off-by: Arthur <[email protected]>
Signed-off-by: wirthual <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add test for unspecified mig

Signed-off-by: Jeev B <[email protected]>

* add support for overriding accelerator

Signed-off-by: Jeev B <[email protected]>

* cleanup

Signed-off-by: Jeev B <[email protected]>

* move from core to extras

Signed-off-by: Jeev B <[email protected]>

* fixes

Signed-off-by: Jeev B <[email protected]>

* fixes

Signed-off-by: Jeev B <[email protected]>

* fixes

Signed-off-by: Jeev B <[email protected]>

* cleanup

Signed-off-by: Jeev B <[email protected]>

* Make FlyteRemote slightly more copy/pastable (flyteorg#1830)

Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Pyflyte meta inputs (flyteorg#1823)

* Re-orgining pyflyte run

Signed-off-by: Ketan Umare <[email protected]>

* Pyflyte beautified and simplified

Signed-off-by: Ketan Umare <[email protected]>

* fixed unit test

Signed-off-by: Ketan Umare <[email protected]>

* Added Launch options

Signed-off-by: Ketan Umare <[email protected]>

* lint fix

Signed-off-by: Ketan Umare <[email protected]>

* test fix

Signed-off-by: Ketan Umare <[email protected]>

* fixing docs failure

Signed-off-by: Ketan Umare <[email protected]>

---------

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Use mashumaro to serialize/deserialize dataclass (flyteorg#1735)

Signed-off-by: HH <[email protected]>
Signed-off-by: hhcs9527 <[email protected]>
Signed-off-by: Matthew Hoffman <[email protected]>
Co-authored-by: Matthew Hoffman <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Databricks Agent (flyteorg#1797)

Signed-off-by: Future Outlier <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Prometheus metrics (flyteorg#1815)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Pyflyte register optionally activates schedule (flyteorg#1832)

* Pyflyte register auto activates schedule

Signed-off-by: Ketan Umare <[email protected]>

* comment addressed

Signed-off-by: Ketan Umare <[email protected]>

---------

Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Remove versions 3.9 and 3.10 (flyteorg#1831)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Snowflake agent (flyteorg#1799)

Signed-off-by: hhcs9527 <[email protected]>
Signed-off-by: HH <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Update agent metric name (flyteorg#1835)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* MemVerge MMCloud Agent (flyteorg#1821)

Signed-off-by: Edwin Yu <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Add download badges in readme (flyteorg#1836)

Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Eager local entrypoint and support for offloaded types (flyteorg#1833)

* implement eager workflow local entrypoint, support offloaded types

Signed-off-by: Niels Bantilan <[email protected]>

* wip local entrypoint

Signed-off-by: Niels Bantilan <[email protected]>

* add tests

Signed-off-by: Niels Bantilan <[email protected]>

* add local entrypoint tests

Signed-off-by: Niels Bantilan <[email protected]>

* update eager unit tests, delete test script

Signed-off-by: Niels Bantilan <[email protected]>

* clean up tests

Signed-off-by: Niels Bantilan <[email protected]>

* update ci

Signed-off-by: Niels Bantilan <[email protected]>

* update ci

Signed-off-by: Niels Bantilan <[email protected]>

* update ci

Signed-off-by: Niels Bantilan <[email protected]>

* update ci

Signed-off-by: Niels Bantilan <[email protected]>

* remove push step

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* update requirements and add snowflake agent to api reference (flyteorg#1838)

* update requirements and add snowflake agent to api reference

Signed-off-by: Samhita Alla <[email protected]>

* update requirements

Signed-off-by: Samhita Alla <[email protected]>

* remove versions

Signed-off-by: Samhita Alla <[email protected]>

* remove tensorflow-macos

Signed-off-by: Samhita Alla <[email protected]>

* lint

Signed-off-by: Samhita Alla <[email protected]>

* downgrade sphinxcontrib-youtube package

Signed-off-by: Samhita Alla <[email protected]>

---------

Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Fix: Make sure decks created in elastic task workers are transferred to parent process (flyteorg#1837)

* Transfer decks created in the worker process to the parent process

Signed-off-by: Fabio Graetz <[email protected]>

* Add test for decks in elastic tasks

Signed-off-by: Fabio Graetz <[email protected]>

* Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py

Signed-off-by: Fabio Graetz <[email protected]>

* Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py

Signed-off-by: Fabio Graetz <[email protected]>

---------

Signed-off-by: Fabio Graetz <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* add accept grpc (flyteorg#1841)

* add accept grpc

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* unpin setup.py grpc

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Revert "add accept grpc"

This reverts commit 2294592.

Signed-off-by: Jeev B <[email protected]>

* default headers interceptor

Signed-off-by: Jeev B <[email protected]>

* setup.py

Signed-off-by: Jeev B <[email protected]>

* fixes

Signed-off-by: Jeev B <[email protected]>

* fmt

Signed-off-by: Jeev B <[email protected]>

* move prometheus-client import

Signed-off-by: Jeev B <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>
Co-authored-by: Jeev B <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (flyteorg#1787)

* Introduce authenticator engine and make proxy auth work

Signed-off-by: Fabio Grätz <[email protected]>

* Use proxy authed session for client credentials flow

Signed-off-by: Fabio Grätz <[email protected]>

* Don't use authenticator engine but do proxy authentication via existing external command authenticator

Signed-off-by: Fabio Grätz <[email protected]>

* Add docstring to AuthenticationHTTPAdapter

Signed-off-by: Fabio Grätz <[email protected]>

* Address todo in docstring

Signed-off-by: Fabio Grätz <[email protected]>

* Create blank session if none provided

Signed-off-by: Fabio Grätz <[email protected]>

* Create blank session if none provided in get_token

Signed-off-by: Fabio Grätz <[email protected]>

* Refresh proxy creds in session when not existing without triggering 401

Signed-off-by: Fabio Grätz <[email protected]>

* Add test for get_session

Signed-off-by: Fabio Grätz <[email protected]>

* Move auth helper test into existing module

Signed-off-by: Fabio Grätz <[email protected]>

* Move auth helper test into existing module

Signed-off-by: Fabio Grätz <[email protected]>

* Add test for upgrade_channel_to_proxy_authenticated

Signed-off-by: Fabio Grätz <[email protected]>

* Auth helper tests without use of responses package

Signed-off-by: Fabio Grätz <[email protected]>

* Feat: Add plugin for generating GCP IAP ID tokens via external command (flyteorg#1795)

* Add external command plugin to generate id tokens for identity aware proxy

Signed-off-by: Fabio Grätz <[email protected]>

* Retrieve desktop app client secret from gcp secret manager

Signed-off-by: Fabio Grätz <[email protected]>

* Remove comments

Signed-off-by: Fabio Grätz <[email protected]>

* Introduce a command group that allows adding a command to generate service account id tokens later

Signed-off-by: Fabio Grätz <[email protected]>

* Document how to use plugin and deploy Flyte with IAP

Signed-off-by: Fabio Grätz <[email protected]>

* Minor corrections README.md

Signed-off-by: Fabio Grätz <[email protected]>

---------

Signed-off-by: Fabio Grätz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>

* Use proxy auth'ed session for device code auth flow

Signed-off-by: Fabio Grätz <[email protected]>

* Fix token client tests

Signed-off-by: Fabio Grätz <[email protected]>

* Make poll token endpoint test more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Make test_client_creds_authenticator test work and more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Make test_client_creds_authenticator_with_custom_scopes test work and more specific

Signed-off-by: Fabio Grätz <[email protected]>

* Implement subcommand to generate id tokens for service accounts

Signed-off-by: Fabio Graetz <[email protected]>

* Test id token generation from service accounts

Signed-off-by: Fabio Graetz <[email protected]>

* Fix plugin requirements

Signed-off-by: Fabio Graetz <[email protected]>

* Document usage of generate-service-account-id-token subcommand

Signed-off-by: Fabio Grätz <[email protected]>

* Document alternative ways to obtain service account id tokens

Signed-off-by: Fabio Grätz <[email protected]>

---------

Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Graetz <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
Signed-off-by: Jeev B <[email protected]>

* bump flyteidl

Signed-off-by: Jeev B <[email protected]>

* make requirements

Signed-off-by: Jeev B <[email protected]>

* fix failing tests

Signed-off-by: Jeev B <[email protected]>

* move gpu accelerator to flyteidl.core.Resources

Signed-off-by: Jeev B <[email protected]>

* Use ResourceExtensions for extended resources

Signed-off-by: Jeev B <[email protected]>

* cleanup

Signed-off-by: Jeev B <[email protected]>

* Switch to using ExtendedResources in TaskTemplate

Signed-off-by: Jeev B <[email protected]>

* cleanups

Signed-off-by: Jeev B <[email protected]>

* update flyteidl

Signed-off-by: Jeev B <[email protected]>

* Replace _core_task imports with tasks_pb2

Signed-off-by: Jeev B <[email protected]>

* less verbose definitions

Signed-off-by: Jeev B <[email protected]>

* Attempt at less confusing syntax

Signed-off-by: Jeev B <[email protected]>

* Streamline UX

Signed-off-by: Jeev B <[email protected]>

* Run make fmt

Signed-off-by: Jeev B <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Jeev B <[email protected]>
Signed-off-by: Victor Delépine <[email protected]>
Signed-off-by: Future Outlier <[email protected]>
Signed-off-by: troychiu <[email protected]>
Signed-off-by: Matthew Hoffman <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Yue Shang <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: oliverhu <[email protected]>
Signed-off-by: Jan Fiedler <[email protected]>
Signed-off-by: Chao-Heng Lee <[email protected]>
Signed-off-by: Adrian Rumpold <[email protected]>
Signed-off-by: Arthur <[email protected]>
Signed-off-by: wirthual <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: HH <[email protected]>
Signed-off-by: hhcs9527 <[email protected]>
Signed-off-by: Edwin Yu <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Fabio Graetz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
Co-authored-by: Victor Delépine <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
Co-authored-by: Future Outlier <[email protected]>
Co-authored-by: Yi Chiu <[email protected]>
Co-authored-by: Matthew Hoffman <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Niels Bantilan <[email protected]>
Co-authored-by: Yue Shang <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Ketan Umare <[email protected]>
Co-authored-by: Keqiu Hu <[email protected]>
Co-authored-by: Jan Fiedler <[email protected]>
Co-authored-by: Chao-Heng Lee <[email protected]>
Co-authored-by: Samhita Alla <[email protected]>
Co-authored-by: Arthur Böök <[email protected]>
Co-authored-by: Katrina Rogan <[email protected]>
Co-authored-by: Po Han(Hank) Huang <[email protected]>
Co-authored-by: Edwin Yu <[email protected]>
Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]>
Co-authored-by: Fabio Grätz <[email protected]>
@jtyberg
Copy link

jtyberg commented Jun 3, 2024

I'm trying to get all Flyte clients to pass the additional proxy-authorization header with each request.

I can use the flytekit-identity-aware-proxy plugin (with proxyCommand in the Flyte config) successfully from a FlyteRemote script, but I'm seeing this error when trying to use flytectl.

bin/flytectl get projects                                                 

Error: 

1 error(s) decoding:

* '' has invalid keys: proxycommand
ERRO[0000] 

1 error(s) decoding:

* '' has invalid keys: proxycommand  src="main.go:13"

It seems the issue is that flytekit and flytectl are stuck at different versions of flyteidl? (this PR never made it in).

@fg91
Copy link
Member Author

fg91 commented Jun 3, 2024

I'm trying to get all Flyte clients to pass the additional proxy-authorization header with each request.

I can use the flytekit-identity-aware-proxy plugin (with proxyCommand in the Flyte config) successfully from a FlyteRemote script, but I'm seeing this error when trying to use flytectl.


bin/flytectl get projects                                                 



Error: 



1 error(s) decoding:



* '' has invalid keys: proxycommand

ERRO[0000] 



1 error(s) decoding:



* '' has invalid keys: proxycommand  src="main.go:13"

It seems the issue is that flytekit and flytectl are stuck at different versions of flyteidl? (this PR never made it in).

The most recent versions of flytectl (released after the move to the monorepo) use the flyteidl admin client which supports proxy auth, try upgrading to e.g. >= 0.8.18 please.

@jtyberg
Copy link

jtyberg commented Jun 3, 2024

The most recent versions of flytectl (released after the move to the monorepo) use the flyteidl admin client which supports proxy auth, try upgrading to e.g. >= 0.8.18 please.

I apologize, I should have posted my flytectl version, which is the v0.8.18 that you recommend.

Installed using curl -sL https://ctl.flyte.org/install | bash.

bin/flytectl version

 A new release of flytectl is available: 0.8.18 → v0.8.22 
To upgrade, run: flytectl upgrade 
https://github.com/flyteorg/flytectl/releases/tag/v0.8.22 

{
  "App": "flytectl",
  "Build": "0a0cbce",
  "Version": "0.8.18",
  "BuildTime": "2024-06-03 10:50:36.469871 -0400 EDT m=+0.045492501"
}%                                                                                                                      

v0.8.18 still shows the error when I add proxyCommand to the flytectl config.

bin/flytectl version

Error: 

1 error(s) decoding:

* '' has invalid keys: proxycommand

@fg91
Copy link
Member Author

fg91 commented Jun 3, 2024

I apologize, I should have posted my flytectl version, which is the v0.8.18 that you recommend.

Sorry, you are right about v0.8.18 (was on my phone). Can you please try v0.8.23? The install script pointed to the archived flytectl repo until this fix a few days ago.

@jtyberg
Copy link

jtyberg commented Jun 3, 2024

I apologize, I should have posted my flytectl version, which is the v0.8.18 that you recommend.

Sorry, you are right about v0.8.18 (was on my phone). Can you please try v0.8.23? The install script pointed to the archived flytectl repo until this fix a few days ago.

Ah, v0.8.23 did it. I can now use the proxyCommand field in my Flyte config.yaml.

bin/flytectl get projects                    

 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| myproject | myproject  | myproject description |
 ----- ------ ----------------- 
1 rows

Thanks for this! (And apologies for opening an old thread).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants