Add Azure-specific headers when uploading to blob storage #1784

devictr · 2023-08-10T13:10:36Z

TL;DR

This PR fixes an issue causing pyflyte run --remote to fail when using Azure.

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Code completed
Smoke tested
Unit tests added
Code documentation added
[] Any pending items have an associated Issue

Complete description

The PUT call to Azure Blob Storage returned an HTTP 400 Error:

<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>MissingRequiredHeader</Code>
    <Message>
        An HTTP header that's mandatory for this request is not specified.
        RequestId:ebd5dfbc-f01e-001d-5074-cb4f26000000
        Time:2023-08-10T10:24:58.6553767Z
    </Message>
    <HeaderName>x-ms-blob-type</HeaderName>
</Error>

This is a starting point that unblocks me, but there's a discussion to be had regarding where you would like to handle such cases. Some open questions:

Should this be done on the client? And if yes, is that the right place?
Or should the response that contains the signed URL also specify headers based on configuration options?

I have tested this locally by tweaking the class and trying to run a workflow

Tracking Issue

flyteorg/flyte#3942

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

welcome · 2023-08-10T13:10:38Z

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
Sign off your commits (Reference: DCO Guide).

Signed-off-by: Victor Delépine <[email protected]>

eapolinario · 2023-08-10T22:06:25Z

flytekit/remote/remote.py

                verify=False
                if self._config.platform.insecure_skip_verify is True
                else self._config.platform.ca_cert_file_path,
            )

-            if rsp.status_code != requests.codes["OK"]:
+            if rsp.status_code not in (requests.codes["OK"], requests.codes["created"]):


why do we have to check this value as well?

Ah sorry, forgot to specify in my PR description.
Azure returns an HTTP 201. So this was throwing an exception even though the request was successful

can you add this as a comment please?

devictr · 2023-08-10T23:08:40Z

Maybe I should add a test on the entire upload_file function that also mocks the put call and checks that the headers are set. Might do that tomorrow

pingsutw · 2023-08-11T14:05:15Z

flytekit/remote/remote.py

+    @staticmethod
+    def get_extra_headers_for_protocol(native_url):
+        if native_url.startswith("abfs://"):
+            return {"x-ms-blob-type": "BlockBlob"}


In the case of blockblob type, the return code will become 200, won't it?
https://stackoverflow.com/questions/67015459/why-does-azure-blob-storage-return-201-but-not-upload

Seems like it returns a 201 https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob?tabs=azure-ad#status-code

That matches what happened when I tested it

Signed-off-by: Victor Delépine <[email protected]>

pingsutw

Thank you @devictr

welcome · 2023-08-17T23:13:44Z

Congrats on merging your first pull request! 🎉

* Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <[email protected]> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <[email protected]> --------- Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Jeev B <[email protected]>

) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <[email protected]> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <[email protected]> --------- Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Future Outlier <[email protected]>

* pip through to container Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * move around Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * add asserts Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * delete bad line Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * switch to abc and add support for gpu unpartitioned Signed-off-by: Jeev B <[email protected]> * Add Azure-specific headers when uploading to blob storage (#1784) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <[email protected]> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <[email protected]> --------- Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add async delete function in base_agent (#1800) Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add support for execution name prefixes (#1803) Signed-off-by: troychiu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove ref in output (#1794) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (#1801) * Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async file sensor (#1790) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager workflows to support async workflows (#1579) * Eager workflows to support async workflows Signed-off-by: Niels Bantilan <[email protected]> * move array node maptask to experimental/__init__.py Signed-off-by: Niels Bantilan <[email protected]> * clean up docs Signed-off-by: Niels Bantilan <[email protected]> * clean up Signed-off-by: Niels Bantilan <[email protected]> * more clean up Signed-off-by: Niels Bantilan <[email protected]> * docs cleanup Signed-off-by: Niels Bantilan <[email protected]> * Update test_eager_workflows.py * clean up timeout handling Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Enable SecretsManager.get to load and return bytes (#1798) * fix secretsmanager Signed-off-by: Yue Shang <[email protected]> * fix lint issue Signed-off-by: Yue Shang <[email protected]> * add doc Signed-off-by: Yue Shang <[email protected]> * fix github check Signed-off-by: Yue Shang <[email protected]> --------- Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Jeev B <[email protected]> * Batch upload flyte directory (#1806) * Batch upload flyte directory Signed-off-by: Kevin Su <[email protected]> * Update get method Signed-off-by: Kevin Su <[email protected]> * Move batch size to type engine Signed-off-by: Kevin Su <[email protected]> * comment Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * Update flytekit/core/type_engine.py Co-authored-by: Eduardo Apolinario <[email protected]> * Add test Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * Better error messaging for overrides (#1807) - using incorrect type of overrides - using incorrect type for resources - using promises in overrides Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Run remote Launchplan from `pyflyte run` (#1785) * Beautified pyflyte run even for every task and workflow - identify a task or a workflow - task or workflow help menus show types and use rich to beautify Signed-off-by: Ketan Umare <[email protected]> * one more improvement Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated command Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * updated formatting Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * bug fixed in types Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add is none function (#1757) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Dynamic workflow should not throw nested task warning (#1812) Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add a manual image building GH action (#1816) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (#1813) Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Jeev B <[email protected]> * None doesnt work Signed-off-by: Jeev B <[email protected]> * unpartitioned selector Signed-off-by: Jeev B <[email protected]> * Fix list of annotated structured dataset (#1817) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (#1819) * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit Signed-off-by: Chao-Heng Lee <[email protected]> * remove helper of getting env var. Signed-off-by: Chao-Heng Lee <[email protected]> * refactor variable name. Signed-off-by: Chao-Heng Lee <[email protected]> --------- Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async agent delete function for while loop case (#1802) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * refactor Signed-off-by: Jeev B <[email protected]> * fix docs warnings (#1827) Signed-off-by: Jeev B <[email protected]> * Fix extract_task_module (#1829) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Add type support for pydantic BaseModels (#1660) Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * add test for unspecified mig Signed-off-by: Jeev B <[email protected]> * add support for overriding accelerator Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * move from core to extras Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Make FlyteRemote slightly more copy/pastable (#1830) Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte meta inputs (#1823) * Re-orgining pyflyte run Signed-off-by: Ketan Umare <[email protected]> * Pyflyte beautified and simplified Signed-off-by: Ketan Umare <[email protected]> * fixed unit test Signed-off-by: Ketan Umare <[email protected]> * Added Launch options Signed-off-by: Ketan Umare <[email protected]> * lint fix Signed-off-by: Ketan Umare <[email protected]> * test fix Signed-off-by: Ketan Umare <[email protected]> * fixing docs failure Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Use mashumaro to serialize/deserialize dataclass (#1735) Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Databricks Agent (#1797) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Prometheus metrics (#1815) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte register optionally activates schedule (#1832) * Pyflyte register auto activates schedule Signed-off-by: Ketan Umare <[email protected]> * comment addressed Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove versions 3.9 and 3.10 (#1831) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Snowflake agent (#1799) Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: Jeev B <[email protected]> * Update agent metric name (#1835) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * MemVerge MMCloud Agent (#1821) Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add download badges in readme (#1836) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager local entrypoint and support for offloaded types (#1833) * implement eager workflow local entrypoint, support offloaded types Signed-off-by: Niels Bantilan <[email protected]> * wip local entrypoint Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add local entrypoint tests Signed-off-by: Niels Bantilan <[email protected]> * update eager unit tests, delete test script Signed-off-by: Niels Bantilan <[email protected]> * clean up tests Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * remove push step Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * update requirements and add snowflake agent to api reference (#1838) * update requirements and add snowflake agent to api reference Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> * remove versions Signed-off-by: Samhita Alla <[email protected]> * remove tensorflow-macos Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * downgrade sphinxcontrib-youtube package Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Jeev B <[email protected]> * Fix: Make sure decks created in elastic task workers are transferred to parent process (#1837) * Transfer decks created in the worker process to the parent process Signed-off-by: Fabio Graetz <[email protected]> * Add test for decks in elastic tasks Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> --------- Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Jeev B <[email protected]> * add accept grpc (#1841) * add accept grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * unpin setup.py grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Revert "add accept grpc" This reverts commit 2294592. Signed-off-by: Jeev B <[email protected]> * default headers interceptor Signed-off-by: Jeev B <[email protected]> * setup.py Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fmt Signed-off-by: Jeev B <[email protected]> * move prometheus-client import Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Co-authored-by: Jeev B <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <[email protected]> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <[email protected]> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <[email protected]> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <[email protected]> * Address todo in docstring Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <[email protected]> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <[email protected]> * Add test for get_session Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <[email protected]> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <[email protected]> * Feat: Add plugin for generating GCP IAP ID tokens via external command (#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <[email protected]> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <[email protected]> * Remove comments Signed-off-by: Fabio Grätz <[email protected]> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <[email protected]> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <[email protected]> * Minor corrections README.md Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <[email protected]> * Fix token client tests Signed-off-by: Fabio Grätz <[email protected]> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <[email protected]> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <[email protected]> * Fix plugin requirements Signed-off-by: Fabio Graetz <[email protected]> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <[email protected]> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Jeev B <[email protected]> * bump flyteidl Signed-off-by: Jeev B <[email protected]> * make requirements Signed-off-by: Jeev B <[email protected]> * fix failing tests Signed-off-by: Jeev B <[email protected]> * move gpu accelerator to flyteidl.core.Resources Signed-off-by: Jeev B <[email protected]> * Use ResourceExtensions for extended resources Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Switch to using ExtendedResources in TaskTemplate Signed-off-by: Jeev B <[email protected]> * cleanups Signed-off-by: Jeev B <[email protected]> * update flyteidl Signed-off-by: Jeev B <[email protected]> * Replace _core_task imports with tasks_pb2 Signed-off-by: Jeev B <[email protected]> * less verbose definitions Signed-off-by: Jeev B <[email protected]> * Attempt at less confusing syntax Signed-off-by: Jeev B <[email protected]> * Streamline UX Signed-off-by: Jeev B <[email protected]> * Run make fmt Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Future Outlier <[email protected]> Signed-off-by: troychiu <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Victor Delépine <[email protected]> Co-authored-by: Future-Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Yi Chiu <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Yue Shang <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Keqiu Hu <[email protected]> Co-authored-by: Jan Fiedler <[email protected]> Co-authored-by: Chao-Heng Lee <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Arthur Böök <[email protected]> Co-authored-by: Katrina Rogan <[email protected]> Co-authored-by: Po Han(Hank) Huang <[email protected]> Co-authored-by: Edwin Yu <[email protected]> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Fabio Grätz <[email protected]>

* pip through to container Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * move around Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * add asserts Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * delete bad line Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * switch to abc and add support for gpu unpartitioned Signed-off-by: Jeev B <[email protected]> * Add Azure-specific headers when uploading to blob storage (flyteorg#1784) * Add Azure-specific headers when uploading to blob storage Signed-off-by: Victor Delépine <[email protected]> * Add comment about HTTP 201 check Signed-off-by: Victor Delépine <[email protected]> --------- Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add async delete function in base_agent (flyteorg#1800) Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add support for execution name prefixes (flyteorg#1803) Signed-off-by: troychiu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove ref in output (flyteorg#1794) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Inherit directly from DataClassJsonMixin instead of using @dataclass_json for improved static type checking (flyteorg#1801) * Inherit directly from DataClassJsonMixin instead of @dataclass_json for improved static type checking As it says in the dataclasses-json README: https://github.com/lidatong/dataclasses-json/blob/89578cb9ebed290e70dba8946bfdb68ff6746755/README.md?plain=1#L111-L129, we can use inheritance for improved static type checking; this one change eliminates something like 467 pyright errors from the flytekit module Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async file sensor (flyteorg#1790) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager workflows to support async workflows (flyteorg#1579) * Eager workflows to support async workflows Signed-off-by: Niels Bantilan <[email protected]> * move array node maptask to experimental/__init__.py Signed-off-by: Niels Bantilan <[email protected]> * clean up docs Signed-off-by: Niels Bantilan <[email protected]> * clean up Signed-off-by: Niels Bantilan <[email protected]> * more clean up Signed-off-by: Niels Bantilan <[email protected]> * docs cleanup Signed-off-by: Niels Bantilan <[email protected]> * Update test_eager_workflows.py * clean up timeout handling Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Enable SecretsManager.get to load and return bytes (flyteorg#1798) * fix secretsmanager Signed-off-by: Yue Shang <[email protected]> * fix lint issue Signed-off-by: Yue Shang <[email protected]> * add doc Signed-off-by: Yue Shang <[email protected]> * fix github check Signed-off-by: Yue Shang <[email protected]> --------- Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Jeev B <[email protected]> * Batch upload flyte directory (flyteorg#1806) * Batch upload flyte directory Signed-off-by: Kevin Su <[email protected]> * Update get method Signed-off-by: Kevin Su <[email protected]> * Move batch size to type engine Signed-off-by: Kevin Su <[email protected]> * comment Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * Update flytekit/core/type_engine.py Co-authored-by: Eduardo Apolinario <[email protected]> * Add test Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * Better error messaging for overrides (flyteorg#1807) - using incorrect type of overrides - using incorrect type for resources - using promises in overrides Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Run remote Launchplan from `pyflyte run` (flyteorg#1785) * Beautified pyflyte run even for every task and workflow - identify a task or a workflow - task or workflow help menus show types and use rich to beautify Signed-off-by: Ketan Umare <[email protected]> * one more improvement Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated command Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * updated formatting Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * bug fixed in types Signed-off-by: Ketan Umare <[email protected]> * Updated Signed-off-by: Ketan Umare <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add is none function (flyteorg#1757) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Dynamic workflow should not throw nested task warning (flyteorg#1812) Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add a manual image building GH action (flyteorg#1816) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * catch abfs protocol in data_persistence.py/get_filesystem and set anon to False (flyteorg#1813) Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Jeev B <[email protected]> * None doesnt work Signed-off-by: Jeev B <[email protected]> * unpartitioned selector Signed-off-by: Jeev B <[email protected]> * Fix list of annotated structured dataset (flyteorg#1817) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit (flyteorg#1819) * Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit Signed-off-by: Chao-Heng Lee <[email protected]> * remove helper of getting env var. Signed-off-by: Chao-Heng Lee <[email protected]> * refactor variable name. Signed-off-by: Chao-Heng Lee <[email protected]> --------- Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Jeev B <[email protected]> * Async agent delete function for while loop case (flyteorg#1802) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * refactor Signed-off-by: Jeev B <[email protected]> * fix docs warnings (flyteorg#1827) Signed-off-by: Jeev B <[email protected]> * Fix extract_task_module (flyteorg#1829) --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Add type support for pydantic BaseModels (flyteorg#1660) Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Jeev B <[email protected]> * add test for unspecified mig Signed-off-by: Jeev B <[email protected]> * add support for overriding accelerator Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * move from core to extras Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Make FlyteRemote slightly more copy/pastable (flyteorg#1830) Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte meta inputs (flyteorg#1823) * Re-orgining pyflyte run Signed-off-by: Ketan Umare <[email protected]> * Pyflyte beautified and simplified Signed-off-by: Ketan Umare <[email protected]> * fixed unit test Signed-off-by: Ketan Umare <[email protected]> * Added Launch options Signed-off-by: Ketan Umare <[email protected]> * lint fix Signed-off-by: Ketan Umare <[email protected]> * test fix Signed-off-by: Ketan Umare <[email protected]> * fixing docs failure Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Use mashumaro to serialize/deserialize dataclass (flyteorg#1735) Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Signed-off-by: Jeev B <[email protected]> * Databricks Agent (flyteorg#1797) Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Prometheus metrics (flyteorg#1815) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Pyflyte register optionally activates schedule (flyteorg#1832) * Pyflyte register auto activates schedule Signed-off-by: Ketan Umare <[email protected]> * comment addressed Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Jeev B <[email protected]> * Remove versions 3.9 and 3.10 (flyteorg#1831) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Snowflake agent (flyteorg#1799) Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: Jeev B <[email protected]> * Update agent metric name (flyteorg#1835) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * MemVerge MMCloud Agent (flyteorg#1821) Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Jeev B <[email protected]> * Add download badges in readme (flyteorg#1836) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Jeev B <[email protected]> * Eager local entrypoint and support for offloaded types (flyteorg#1833) * implement eager workflow local entrypoint, support offloaded types Signed-off-by: Niels Bantilan <[email protected]> * wip local entrypoint Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add local entrypoint tests Signed-off-by: Niels Bantilan <[email protected]> * update eager unit tests, delete test script Signed-off-by: Niels Bantilan <[email protected]> * clean up tests Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * update ci Signed-off-by: Niels Bantilan <[email protected]> * remove push step Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> * update requirements and add snowflake agent to api reference (flyteorg#1838) * update requirements and add snowflake agent to api reference Signed-off-by: Samhita Alla <[email protected]> * update requirements Signed-off-by: Samhita Alla <[email protected]> * remove versions Signed-off-by: Samhita Alla <[email protected]> * remove tensorflow-macos Signed-off-by: Samhita Alla <[email protected]> * lint Signed-off-by: Samhita Alla <[email protected]> * downgrade sphinxcontrib-youtube package Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Jeev B <[email protected]> * Fix: Make sure decks created in elastic task workers are transferred to parent process (flyteorg#1837) * Transfer decks created in the worker process to the parent process Signed-off-by: Fabio Graetz <[email protected]> * Add test for decks in elastic tasks Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> * Update plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/task.py Signed-off-by: Fabio Graetz <[email protected]> --------- Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Jeev B <[email protected]> * add accept grpc (flyteorg#1841) * add accept grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * unpin setup.py grpc Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> * Revert "add accept grpc" This reverts commit 2294592. Signed-off-by: Jeev B <[email protected]> * default headers interceptor Signed-off-by: Jeev B <[email protected]> * setup.py Signed-off-by: Jeev B <[email protected]> * fixes Signed-off-by: Jeev B <[email protected]> * fmt Signed-off-by: Jeev B <[email protected]> * move prometheus-client import Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Co-authored-by: Jeev B <[email protected]> Signed-off-by: Jeev B <[email protected]> * Feat: Enable `flytekit` to authenticate with proxy in front of FlyteAdmin (flyteorg#1787) * Introduce authenticator engine and make proxy auth work Signed-off-by: Fabio Grätz <[email protected]> * Use proxy authed session for client credentials flow Signed-off-by: Fabio Grätz <[email protected]> * Don't use authenticator engine but do proxy authentication via existing external command authenticator Signed-off-by: Fabio Grätz <[email protected]> * Add docstring to AuthenticationHTTPAdapter Signed-off-by: Fabio Grätz <[email protected]> * Address todo in docstring Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided Signed-off-by: Fabio Grätz <[email protected]> * Create blank session if none provided in get_token Signed-off-by: Fabio Grätz <[email protected]> * Refresh proxy creds in session when not existing without triggering 401 Signed-off-by: Fabio Grätz <[email protected]> * Add test for get_session Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Move auth helper test into existing module Signed-off-by: Fabio Grätz <[email protected]> * Add test for upgrade_channel_to_proxy_authenticated Signed-off-by: Fabio Grätz <[email protected]> * Auth helper tests without use of responses package Signed-off-by: Fabio Grätz <[email protected]> * Feat: Add plugin for generating GCP IAP ID tokens via external command (flyteorg#1795) * Add external command plugin to generate id tokens for identity aware proxy Signed-off-by: Fabio Grätz <[email protected]> * Retrieve desktop app client secret from gcp secret manager Signed-off-by: Fabio Grätz <[email protected]> * Remove comments Signed-off-by: Fabio Grätz <[email protected]> * Introduce a command group that allows adding a command to generate service account id tokens later Signed-off-by: Fabio Grätz <[email protected]> * Document how to use plugin and deploy Flyte with IAP Signed-off-by: Fabio Grätz <[email protected]> * Minor corrections README.md Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> * Use proxy auth'ed session for device code auth flow Signed-off-by: Fabio Grätz <[email protected]> * Fix token client tests Signed-off-by: Fabio Grätz <[email protected]> * Make poll token endpoint test more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Make test_client_creds_authenticator_with_custom_scopes test work and more specific Signed-off-by: Fabio Grätz <[email protected]> * Implement subcommand to generate id tokens for service accounts Signed-off-by: Fabio Graetz <[email protected]> * Test id token generation from service accounts Signed-off-by: Fabio Graetz <[email protected]> * Fix plugin requirements Signed-off-by: Fabio Graetz <[email protected]> * Document usage of generate-service-account-id-token subcommand Signed-off-by: Fabio Grätz <[email protected]> * Document alternative ways to obtain service account id tokens Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Jeev B <[email protected]> * bump flyteidl Signed-off-by: Jeev B <[email protected]> * make requirements Signed-off-by: Jeev B <[email protected]> * fix failing tests Signed-off-by: Jeev B <[email protected]> * move gpu accelerator to flyteidl.core.Resources Signed-off-by: Jeev B <[email protected]> * Use ResourceExtensions for extended resources Signed-off-by: Jeev B <[email protected]> * cleanup Signed-off-by: Jeev B <[email protected]> * Switch to using ExtendedResources in TaskTemplate Signed-off-by: Jeev B <[email protected]> * cleanups Signed-off-by: Jeev B <[email protected]> * update flyteidl Signed-off-by: Jeev B <[email protected]> * Replace _core_task imports with tasks_pb2 Signed-off-by: Jeev B <[email protected]> * less verbose definitions Signed-off-by: Jeev B <[email protected]> * Attempt at less confusing syntax Signed-off-by: Jeev B <[email protected]> * Streamline UX Signed-off-by: Jeev B <[email protected]> * Run make fmt Signed-off-by: Jeev B <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Jeev B <[email protected]> Signed-off-by: Victor Delépine <[email protected]> Signed-off-by: Future Outlier <[email protected]> Signed-off-by: troychiu <[email protected]> Signed-off-by: Matthew Hoffman <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yue Shang <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: oliverhu <[email protected]> Signed-off-by: Jan Fiedler <[email protected]> Signed-off-by: Chao-Heng Lee <[email protected]> Signed-off-by: Adrian Rumpold <[email protected]> Signed-off-by: Arthur <[email protected]> Signed-off-by: wirthual <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Katrina Rogan <[email protected]> Signed-off-by: HH <[email protected]> Signed-off-by: hhcs9527 <[email protected]> Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Victor Delépine <[email protected]> Co-authored-by: Future-Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]> Co-authored-by: Yi Chiu <[email protected]> Co-authored-by: Matthew Hoffman <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Yue Shang <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Keqiu Hu <[email protected]> Co-authored-by: Jan Fiedler <[email protected]> Co-authored-by: Chao-Heng Lee <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Arthur Böök <[email protected]> Co-authored-by: Katrina Rogan <[email protected]> Co-authored-by: Po Han(Hank) Huang <[email protected]> Co-authored-by: Edwin Yu <[email protected]> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Fabio Grätz <[email protected]>

devictr requested review from wild-endeavor, kumare3, eapolinario, pingsutw and cosmicBboy as code owners August 10, 2023 13:10

Add Azure-specific headers when uploading to blob storage

91dafce

Signed-off-by: Victor Delépine <[email protected]>

devictr force-pushed the azure-headers branch from 3ff5c10 to 91dafce Compare August 10, 2023 13:15

eapolinario reviewed Aug 10, 2023

View reviewed changes

pingsutw reviewed Aug 11, 2023

View reviewed changes

Add comment about HTTP 201 check

14a19a5

Signed-off-by: Victor Delépine <[email protected]>

devictr force-pushed the azure-headers branch from 6438828 to 14a19a5 Compare August 16, 2023 16:04

wild-endeavor approved these changes Aug 17, 2023

View reviewed changes

pingsutw approved these changes Aug 17, 2023

View reviewed changes

wild-endeavor merged commit 6acc97b into flyteorg:master Aug 17, 2023

Tom-Newton mentioned this pull request Jan 22, 2024

Flyte flyte:// file system and improve remote file handling #1674

Merged

8 tasks

olandr mentioned this pull request Sep 19, 2024

[BUG] [flytekit] pyflyte register fails for Azure Blob Storage (Missing Headers) flyteorg/flyte#5762

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Azure-specific headers when uploading to blob storage #1784

Add Azure-specific headers when uploading to blob storage #1784

devictr commented Aug 10, 2023 •

edited

Loading

welcome bot commented Aug 10, 2023

eapolinario Aug 10, 2023

devictr Aug 10, 2023

wild-endeavor Aug 14, 2023

devictr Aug 15, 2023

devictr commented Aug 10, 2023

pingsutw Aug 11, 2023

devictr Aug 11, 2023

devictr Aug 11, 2023

pingsutw left a comment

welcome bot commented Aug 17, 2023

Add Azure-specific headers when uploading to blob storage #1784

Add Azure-specific headers when uploading to blob storage #1784

Conversation

devictr commented Aug 10, 2023 • edited Loading

TL;DR

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

welcome bot commented Aug 10, 2023

eapolinario Aug 10, 2023

Choose a reason for hiding this comment

devictr Aug 10, 2023

Choose a reason for hiding this comment

wild-endeavor Aug 14, 2023

Choose a reason for hiding this comment

devictr Aug 15, 2023

Choose a reason for hiding this comment

devictr commented Aug 10, 2023

pingsutw Aug 11, 2023

Choose a reason for hiding this comment

devictr Aug 11, 2023

Choose a reason for hiding this comment

devictr Aug 11, 2023

Choose a reason for hiding this comment

pingsutw left a comment

Choose a reason for hiding this comment

welcome bot commented Aug 17, 2023

devictr commented Aug 10, 2023 •

edited

Loading