-
Notifications
You must be signed in to change notification settings - Fork 300
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feat: Enable flytekit
to authenticate with proxy in front of FlyteAdmin
#1787
Conversation
@@ -64,6 +64,13 @@ class Credentials(object): | |||
This command is executed to return a token using an external process. | |||
""" | |||
|
|||
PROXY_COMMAND = ConfigEntry( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: do we need the legacy config entry or do we only support this in yaml config?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think this is the correct declaration. Let's keep it. But what is this? what does this do?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added an explanation in this PR's description.
@@ -61,7 +61,7 @@ def intercept_unary_unary( | |||
fut: grpc.Future = continuation(updated_call_details, request) | |||
e = fut.exception() | |||
if e: | |||
if e.code() == grpc.StatusCode.UNAUTHENTICATED: | |||
if e.code() == grpc.StatusCode.UNAUTHENTICATED or e.code() == grpc.StatusCode.UNKNOWN: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When trying to reach flyteadmin behind GCP Identity Aware Proxy and when omitting the "proxy-authorization"
header, we get back:
<_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Stream removed"
debug_error_string = "UNKNOWN:Error received from peer ipv4:<my-ip>:443 {grpc_message:"Stream removed", grpc_status:2, created_time:"2023-xxx"}"
flytekit
to authenticate with proxy in front of FlyteAdmin
7987098
to
c80cf60
Compare
@wild-endeavor I merged #1795 into this one as requested 👍 |
authn.refresh_credentials() | ||
expected_scopes = static_cfg_store.get_client_config().scopes | ||
|
||
assert authn._creds |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The order of mocking vs instantiation of the authenticator matters now.
I added the more explicit check assert authn._creds.access_token == "abc"
because this would fail in case somebody in the future reverses the order again (instantiate the authenticator first, then mock).
assert t | ||
assert e | ||
|
||
assert t == "abc" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's be more specific here. When adapting the magic mocks, I noticed that here one easily gets a MagicMock
object for t
which then also is not None
but is not "abc"
.
This makes sure that in the future no one accidentally breaks the mocking.
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
…ng external command authenticator Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <[email protected]> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <[email protected]> * Remove comments Signed-off-by: Fabio Grätz <[email protected]> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <[email protected]> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <[email protected]> * Minor corrections README.md Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
Signed-off-by: Fabio Grätz <[email protected]>
… more specific Signed-off-by: Fabio Grätz <[email protected]>
858e96b
to
c677ff3
Compare
4d09bf8
to
336aebc
Compare
Signed-off-by: Fabio Graetz <[email protected]>
Signed-off-by: Fabio Graetz <[email protected]>
Thanks for taking a look @wild-endeavor I added a second subcommand to ~ flyte-iap --help
Usage: flyte-iap [OPTIONS] COMMAND [ARGS]...
Generate ID tokens for GCP Identity Aware Proxy (IAP).
Options:
--help Show this message and exit.
Commands:
generate-service-account-id-token # <- new
Generate a service account ID token...
generate-user-id-token Generate a user account ID token for... The subcommand uses either
To generate a token for service accounts, in contrast to generating one for user accounts, no browser needs to be opened interactively. The goal of this subcommand is that CICD pipelines can talk to flyteadmin through IAP. (@corleyma said they will require this.) With that,
❯ flytectl config init
Error:
1 error(s) decoding:
* '' has invalid keys: proxycommand
ERRO[0000]
1 error(s) decoding:
* '' has invalid keys: proxycommand src="main.go:13" |
Tagging you as reviewer, @EngHabu, as suggested by @wild-endeavor 🙏 |
Signed-off-by: Fabio Grätz <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm less sure about reusing this command for flytectl. Requiring python dependency isn't something I would want to require in CI/CD and other automated systems
Signed-off-by: Fabio Grätz <[email protected]>
Hi Haytham, I really appreciate the time you’ve invested in looking at this PR, I'm sure you have a busy schedule 🙏
In our case, we will run |
tested this with pkce, cc, and device flow btw, all works |
Thanks for testing this @wild-endeavor! 🙏 |
…dmin (#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <[email protected]> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <[email protected]> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <[email protected]> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <[email protected]> * Address todo in docstring Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <[email protected]> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <[email protected]> * Add test for get_session Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <[email protected]> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <[email protected]> * Feat: Add plugin for generating GCP IAP ID tokens via external command (#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <[email protected]> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <[email protected]> * Remove comments Signed-off-by: Fabio Grätz <[email protected]> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <[email protected]> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <[email protected]> * Minor corrections README.md Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <[email protected]> * Fix token client tests Signed-off-by: Fabio Grätz <[email protected]> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <[email protected]> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <[email protected]> * Fix plugin requirements Signed-off-by: Fabio Graetz <[email protected]> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <[email protected]> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Jeev B <[email protected]>
…dmin (flyteorg#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <[email protected]> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <[email protected]> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <[email protected]> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <[email protected]> * Address todo in docstring Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <[email protected]> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <[email protected]> * Add test for get_session Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <[email protected]> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <[email protected]> * Feat: Add plugin for generating GCP IAP ID tokens via external command (flyteorg#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <[email protected]> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <[email protected]> * Remove comments Signed-off-by: Fabio Grätz <[email protected]> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <[email protected]> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <[email protected]> * Minor corrections README.md Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <[email protected]> * Fix token client tests Signed-off-by: Fabio Grätz <[email protected]> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <[email protected]> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <[email protected]> * Fix plugin requirements Signed-off-by: Fabio Graetz <[email protected]> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <[email protected]> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Future Outlier <[email protected]>
* pip through to container Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * move around Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * add asserts Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * delete bad line Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * switch to abc and add support for gpu unpartitioned Signed-off-by: Jeev B <[email protected]> * Add Azure-specific headers when uploading to blob storage (#1784) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <[email protected]> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <[email protected]> --------- Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add async delete function in base_agent (#1800) Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add support for execution name prefixes (#1803) Signed-off-by: troychiu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove ref in output (#1794) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (#1801) * Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async file sensor (#1790) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager workflows to support async workflows (#1579) * Eager workflows to support async workflows Signed-off-by: Niels Bantilan <[email protected]> * move array node maptask to experimental/__init__.py Signed-off-by: Niels Bantilan <[email protected]> * clean up docs Signed-off-by: Niels Bantilan <[email protected]> * clean up Signed-off-by: Niels Bantilan <[email protected]> * more clean up Signed-off-by: Niels Bantilan <[email protected]> * docs cleanup Signed-off-by: Niels Bantilan <[email protected]> * Update test_eager_workflows.py * clean up timeout handling Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Enable SecretsManager.get to load and return bytes (#1798) * fix secretsmanager Signed-off-by: Yue Shang <[email protected]> * fix lint issue Signed-off-by: Yue Shang <[email protected]> * add doc Signed-off-by: Yue Shang <[email protected]> * fix github check Signed-off-by: Yue Shang <[email protected]> --------- Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Jeev B <[email protected]> * Batch upload flyte directory (#1806) * Batch upload flyte directory Signed-off-by: Kevin Su <[email protected]> * Update get method Signed-off-by: Kevin Su <[email protected]> * Move batch size to type engine Signed-off-by: Kevin Su <[email protected]> * comment Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * Update flytekit/core/type_engine.py Co-authored-by: Eduardo Apolinario <[email protected]> * Add test Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * Better error messaging for overrides (#1807) - using incorrect type of overrides - using incorrect type for resources - using promises in overrides Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Run remote Launchplan from `pyflyte run` (#1785) * Beautified pyflyte run even for every task and workflow - identify a task or a workflow - task or workflow help menus show types and use rich to beautify Signed-off-by: Ketan Umare <[email protected]> * one more improvement Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated command Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * updated formatting Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * bug fixed in types Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add is none function (#1757) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Dynamic workflow should not throw nested task warning (#1812) Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add a manual image building GH action (#1816) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (#1813) Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Jeev B <[email protected]> * None doesnt work Signed-off-by: Jeev B <[email protected]> * unpartitioned selector Signed-off-by: Jeev B <[email protected]> * Fix list of annotated structured dataset (#1817) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (#1819) * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit Signed-off-by: Chao-Heng Lee <[email protected]> * remove helper of getting env var. Signed-off-by: Chao-Heng Lee <[email protected]> * refactor variable name. Signed-off-by: Chao-Heng Lee <[email protected]> --------- Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async agent delete function for while loop case (#1802) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * refactor Signed-off-by: Jeev B <[email protected]> * fix docs warnings (#1827) Signed-off-by: Jeev B <[email protected]> * Fix extract_task_module (#1829) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Add type support for pydantic BaseModels (#1660) Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * add test for unspecified mig Signed-off-by: Jeev B <[email protected]> * add support for overriding accelerator Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * move from core to extras Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Make FlyteRemote slightly more copy/pastable (#1830) Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte meta inputs (#1823) * Re-orgining pyflyte run Signed-off-by: Ketan Umare <[email protected]> * Pyflyte beautified and simplified Signed-off-by: Ketan Umare <[email protected]> * fixed unit test Signed-off-by: Ketan Umare <[email protected]> * Added Launch options Signed-off-by: Ketan Umare <[email protected]> * lint fix Signed-off-by: Ketan Umare <[email protected]> * test fix Signed-off-by: Ketan Umare <[email protected]> * fixing docs failure Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Use mashumaro to serialize/deserialize dataclass (#1735) Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Databricks Agent (#1797) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Prometheus metrics (#1815) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte register optionally activates schedule (#1832) * Pyflyte register auto activates schedule Signed-off-by: Ketan Umare <[email protected]> * comment addressed Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove versions 3.9 and 3.10 (#1831) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Snowflake agent (#1799) Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: Jeev B <[email protected]> * Update agent metric name (#1835) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * MemVerge MMCloud Agent (#1821) Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add download badges in readme (#1836) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager local entrypoint and support for offloaded types (#1833) * implement eager workflow local entrypoint, support offloaded types Signed-off-by: Niels Bantilan <[email protected]> * wip local entrypoint Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add local entrypoint tests Signed-off-by: Niels Bantilan <[email protected]> * update eager unit tests, delete test script Signed-off-by: Niels Bantilan <[email protected]> * clean up tests Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * remove push step Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * update requirements and add snowflake agent to api reference (#1838) * update requirements and add snowflake agent to api reference Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> * remove versions Signed-off-by: Samhita Alla <[email protected]> * remove tensorflow-macos Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * downgrade sphinxcontrib-youtube package Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Jeev B <[email protected]> * Fix: Make sure decks created in elastic task workers are transferred to parent process (#1837) * Transfer decks created in the worker process to the parent process Signed-off-by: Fabio Graetz <[email protected]> * Add test for decks in elastic tasks Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> --------- Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Jeev B <[email protected]> * add accept grpc (#1841) * add accept grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * unpin setup.py grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Revert "add accept grpc" This reverts commit 2294592. Signed-off-by: Jeev B <[email protected]> * default headers interceptor Signed-off-by: Jeev B <[email protected]> * setup.py Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fmt Signed-off-by: Jeev B <[email protected]> * move prometheus-client import Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Co-authored-by: Jeev B <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <[email protected]> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <[email protected]> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <[email protected]> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <[email protected]> * Address todo in docstring Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <[email protected]> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <[email protected]> * Add test for get_session Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <[email protected]> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <[email protected]> * Feat: Add plugin for generating GCP IAP ID tokens via external command (#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <[email protected]> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <[email protected]> * Remove comments Signed-off-by: Fabio Grätz <[email protected]> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <[email protected]> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <[email protected]> * Minor corrections README.md Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <[email protected]> * Fix token client tests Signed-off-by: Fabio Grätz <[email protected]> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <[email protected]> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <[email protected]> * Fix plugin requirements Signed-off-by: Fabio Graetz <[email protected]> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <[email protected]> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Jeev B <[email protected]> * bump flyteidl Signed-off-by: Jeev B <[email protected]> * make requirements Signed-off-by: Jeev B <[email protected]> * fix failing tests Signed-off-by: Jeev B <[email protected]> * move gpu accelerator to flyteidl.core.Resources Signed-off-by: Jeev B <[email protected]> * Use ResourceExtensions for extended resources Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Switch to using ExtendedResources in TaskTemplate Signed-off-by: Jeev B <[email protected]> * cleanups Signed-off-by: Jeev B <[email protected]> * update flyteidl Signed-off-by: Jeev B <[email protected]> * Replace _core_task imports with tasks_pb2 Signed-off-by: Jeev B <[email protected]> * less verbose definitions Signed-off-by: Jeev B <[email protected]> * Attempt at less confusing syntax Signed-off-by: Jeev B <[email protected]> * Streamline UX Signed-off-by: Jeev B <[email protected]> * Run make fmt Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Future Outlier <[email protected]> Signed-off-by: troychiu <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Victor Delépine <[email protected]> Co-authored-by: Future-Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Yi Chiu <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Yue Shang <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Keqiu Hu <[email protected]> Co-authored-by: Jan Fiedler <[email protected]> Co-authored-by: Chao-Heng Lee <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Arthur Böök <[email protected]> Co-authored-by: Katrina Rogan <[email protected]> Co-authored-by: Po Han(Hank) Huang <[email protected]> Co-authored-by: Edwin Yu <[email protected]> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Fabio Grätz <[email protected]>
* pip through to container Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * move around Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * add asserts Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * delete bad line Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * switch to abc and add support for gpu unpartitioned Signed-off-by: Jeev B <[email protected]> * Add Azure-specific headers when uploading to blob storage (flyteorg#1784) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <[email protected]> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <[email protected]> --------- Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add async delete function in base_agent (flyteorg#1800) Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add support for execution name prefixes (flyteorg#1803) Signed-off-by: troychiu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove ref in output (flyteorg#1794) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (flyteorg#1801) * Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async file sensor (flyteorg#1790) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager workflows to support async workflows (flyteorg#1579) * Eager workflows to support async workflows Signed-off-by: Niels Bantilan <[email protected]> * move array node maptask to experimental/__init__.py Signed-off-by: Niels Bantilan <[email protected]> * clean up docs Signed-off-by: Niels Bantilan <[email protected]> * clean up Signed-off-by: Niels Bantilan <[email protected]> * more clean up Signed-off-by: Niels Bantilan <[email protected]> * docs cleanup Signed-off-by: Niels Bantilan <[email protected]> * Update test_eager_workflows.py * clean up timeout handling Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Enable SecretsManager.get to load and return bytes (flyteorg#1798) * fix secretsmanager Signed-off-by: Yue Shang <[email protected]> * fix lint issue Signed-off-by: Yue Shang <[email protected]> * add doc Signed-off-by: Yue Shang <[email protected]> * fix github check Signed-off-by: Yue Shang <[email protected]> --------- Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Jeev B <[email protected]> * Batch upload flyte directory (flyteorg#1806) * Batch upload flyte directory Signed-off-by: Kevin Su <[email protected]> * Update get method Signed-off-by: Kevin Su <[email protected]> * Move batch size to type engine Signed-off-by: Kevin Su <[email protected]> * comment Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * Update flytekit/core/type_engine.py Co-authored-by: Eduardo Apolinario <[email protected]> * Add test Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * Better error messaging for overrides (flyteorg#1807) - using incorrect type of overrides - using incorrect type for resources - using promises in overrides Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Run remote Launchplan from `pyflyte run` (flyteorg#1785) * Beautified pyflyte run even for every task and workflow - identify a task or a workflow - task or workflow help menus show types and use rich to beautify Signed-off-by: Ketan Umare <[email protected]> * one more improvement Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated command Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * updated formatting Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * bug fixed in types Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add is none function (flyteorg#1757) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Dynamic workflow should not throw nested task warning (flyteorg#1812) Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add a manual image building GH action (flyteorg#1816) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (flyteorg#1813) Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Jeev B <[email protected]> * None doesnt work Signed-off-by: Jeev B <[email protected]> * unpartitioned selector Signed-off-by: Jeev B <[email protected]> * Fix list of annotated structured dataset (flyteorg#1817) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (flyteorg#1819) * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit Signed-off-by: Chao-Heng Lee <[email protected]> * remove helper of getting env var. Signed-off-by: Chao-Heng Lee <[email protected]> * refactor variable name. Signed-off-by: Chao-Heng Lee <[email protected]> --------- Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async agent delete function for while loop case (flyteorg#1802) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * refactor Signed-off-by: Jeev B <[email protected]> * fix docs warnings (flyteorg#1827) Signed-off-by: Jeev B <[email protected]> * Fix extract_task_module (flyteorg#1829) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Add type support for pydantic BaseModels (flyteorg#1660) Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * add test for unspecified mig Signed-off-by: Jeev B <[email protected]> * add support for overriding accelerator Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * move from core to extras Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Make FlyteRemote slightly more copy/pastable (flyteorg#1830) Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte meta inputs (flyteorg#1823) * Re-orgining pyflyte run Signed-off-by: Ketan Umare <[email protected]> * Pyflyte beautified and simplified Signed-off-by: Ketan Umare <[email protected]> * fixed unit test Signed-off-by: Ketan Umare <[email protected]> * Added Launch options Signed-off-by: Ketan Umare <[email protected]> * lint fix Signed-off-by: Ketan Umare <[email protected]> * test fix Signed-off-by: Ketan Umare <[email protected]> * fixing docs failure Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Use mashumaro to serialize/deserialize dataclass (flyteorg#1735) Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Databricks Agent (flyteorg#1797) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Prometheus metrics (flyteorg#1815) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte register optionally activates schedule (flyteorg#1832) * Pyflyte register auto activates schedule Signed-off-by: Ketan Umare <[email protected]> * comment addressed Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove versions 3.9 and 3.10 (flyteorg#1831) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Snowflake agent (flyteorg#1799) Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: Jeev B <[email protected]> * Update agent metric name (flyteorg#1835) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * MemVerge MMCloud Agent (flyteorg#1821) Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add download badges in readme (flyteorg#1836) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager local entrypoint and support for offloaded types (flyteorg#1833) * implement eager workflow local entrypoint, support offloaded types Signed-off-by: Niels Bantilan <[email protected]> * wip local entrypoint Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add local entrypoint tests Signed-off-by: Niels Bantilan <[email protected]> * update eager unit tests, delete test script Signed-off-by: Niels Bantilan <[email protected]> * clean up tests Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * remove push step Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * update requirements and add snowflake agent to api reference (flyteorg#1838) * update requirements and add snowflake agent to api reference Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> * remove versions Signed-off-by: Samhita Alla <[email protected]> * remove tensorflow-macos Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * downgrade sphinxcontrib-youtube package Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Jeev B <[email protected]> * Fix: Make sure decks created in elastic task workers are transferred to parent process (flyteorg#1837) * Transfer decks created in the worker process to the parent process Signed-off-by: Fabio Graetz <[email protected]> * Add test for decks in elastic tasks Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> --------- Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Jeev B <[email protected]> * add accept grpc (flyteorg#1841) * add accept grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * unpin setup.py grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Revert "add accept grpc" This reverts commit 2294592. Signed-off-by: Jeev B <[email protected]> * default headers interceptor Signed-off-by: Jeev B <[email protected]> * setup.py Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fmt Signed-off-by: Jeev B <[email protected]> * move prometheus-client import Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Co-authored-by: Jeev B <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (flyteorg#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <[email protected]> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <[email protected]> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <[email protected]> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <[email protected]> * Address todo in docstring Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <[email protected]> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <[email protected]> * Add test for get_session Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <[email protected]> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <[email protected]> * Feat: Add plugin for generating GCP IAP ID tokens via external command (flyteorg#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <[email protected]> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <[email protected]> * Remove comments Signed-off-by: Fabio Grätz <[email protected]> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <[email protected]> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <[email protected]> * Minor corrections README.md Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <[email protected]> * Fix token client tests Signed-off-by: Fabio Grätz <[email protected]> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <[email protected]> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <[email protected]> * Fix plugin requirements Signed-off-by: Fabio Graetz <[email protected]> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <[email protected]> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Jeev B <[email protected]> * bump flyteidl Signed-off-by: Jeev B <[email protected]> * make requirements Signed-off-by: Jeev B <[email protected]> * fix failing tests Signed-off-by: Jeev B <[email protected]> * move gpu accelerator to flyteidl.core.Resources Signed-off-by: Jeev B <[email protected]> * Use ResourceExtensions for extended resources Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Switch to using ExtendedResources in TaskTemplate Signed-off-by: Jeev B <[email protected]> * cleanups Signed-off-by: Jeev B <[email protected]> * update flyteidl Signed-off-by: Jeev B <[email protected]> * Replace _core_task imports with tasks_pb2 Signed-off-by: Jeev B <[email protected]> * less verbose definitions Signed-off-by: Jeev B <[email protected]> * Attempt at less confusing syntax Signed-off-by: Jeev B <[email protected]> * Streamline UX Signed-off-by: Jeev B <[email protected]> * Run make fmt Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Future Outlier <[email protected]> Signed-off-by: troychiu <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Victor Delépine <[email protected]> Co-authored-by: Future-Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Yi Chiu <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Yue Shang <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Keqiu Hu <[email protected]> Co-authored-by: Jan Fiedler <[email protected]> Co-authored-by: Chao-Heng Lee <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Arthur Böök <[email protected]> Co-authored-by: Katrina Rogan <[email protected]> Co-authored-by: Po Han(Hank) Huang <[email protected]> Co-authored-by: Edwin Yu <[email protected]> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Fabio Grätz <[email protected]>
I'm trying to get all Flyte clients to pass the additional I can use the flytekit-identity-aware-proxy plugin (with
It seems the issue is that |
The most recent versions of flytectl (released after the move to the monorepo) use the flyteidl admin client which supports proxy auth, try upgrading to e.g. >= 0.8.18 please. |
I apologize, I should have posted my Installed using
|
Ah, v0.8.23 did it. I can now use the
Thanks for this! (And apologies for opening an old thread). |
TL;DR
Part of an effort to integrate Flyte with GCP Identity Aware Proxy (IAP).
See tracking issue for details and motivation: flyteorg/flyte#3965
Type
Are all requirements met?
FlyteRemote
can talk to admin through the IAP, e.g. to retrieve executions, start new ones etc.)Complete description
proxyCommand
to the platform config in which users can specify an external command which, if specified, is used to generate ID tokens for a proxy in front of Flyte.command
(external command auth type) which is used to generate tokens for Flyte itself.proxyCommand
are added as"proxy-authorization"
header to every request (http, gRPC) that flytekit's client makes to flyteadmin, including the initial unauthenticated requests of the auth flow with flyteadmin. These additional ID tokens in the"proxy-authorization"
header allow clients to interact with a flyteadmin that is protected by GCP Identity Aware Proxy (or in theory similar services from other providers)AuthorizationClient
which currently can only be used for a pkce auth flow (despite its doc string claiming it works for a general OAuth 2.0 flow) to actually work for both pkce and default OAuth 2.0.AuthorizationClient
in flytekit to perform an OAuth 2.0 flow withaccounts.google.com
.proxyCommand
in the platform config for Flyte deployments protected with IAPTracking Issue
https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/